How accurate are health headlines in your social media feed?

{***Pause/Music***}
{***Noah***}

Coming up on Harvard Chan: This Week in Health…

How accurate are those health headlines you see in your social media feeds?

{***Noah Haber Soundbite***}
( think the big takeaway is assume that if it’s on your Facebook feed or your Twitter feed, and it’s health study says X linked to Y, you should assume that’s not a causal relationship.)

New research shows that the studies and related articles shared in social media linking something with a health result are likely to be overstated or inaccurate. In this week’s episode, we speak with the author of that study about how that happens—and what can be done to improve health reporting.

{***Pause/Music***}
{**Noah**}

Hello and welcome to Harvard Chan: This Week in Health, it’s Friday, November 2, 2018. I’m Noah Leavitt.

{***Amie***}

And I’m Amie Montemurro.

{***Noah***}

Amie, every day we are bombarded with health news in our social media feeds: from studies touting the benefits—or harms—of a particular food to research on a new treatment for a disease.

{***Amie***}

But how accurate are those headlines?

We’re all familiar with the saying “correlation doesn’t equal causation”—but do the stories you’re reading accurately report that distinction?

{***Noah***}

That’s the question that a multidisciplinary research team tried to answer.

{***Noah Haber Soundbite***}
(I’m Noah Haber. I recently completed my SD from the Harvard T.H. Chan School of Public Health in the Department of Global Health and Population. I am now a postdoctoral researcher at the Carolina Population Center at the University of North Carolina-Chapel Hill.)

{***Noah***}

That’s Noah Haber, who led the research team. And we recently spoke to him via Skype about that study.

{***Amie***}

He and his colleagues looked at 50 academic journal articles published in 2015, linking any exposure with a health outcome and media stories covered those articles. The articles were all among the most widely shared on social media that year.

{***Noah***}

The team assessed the studies’ strength of causal inference, or whether the study could determine that the exposure itself changed the health outcome, using a novel systematic review tool. They then compared them with the strength of causal language used to describe results in both academic journal articles and media articles.

{***Amie***}

The study found that 34% of the academic studies reviewed used language that reviewers considered too strong given their strength of causal inference, and 48% of media articles used stronger language than their associated academic articles.

In addition, 58% of media articles inaccurately reported the question, results, intervention, or population of the academic study.

{***Noah***}

I spoke to Haber about the key findings of the study—and also what it tells us about the intersection of science, the media, and social media.

I began our conversation by asking him to explain what inspired this research in the first place.

NOAH HABER: So the inspiration of this study I think started from something that is a relatively common experience among health researchers. So we were all looking at our Facebook feeds and our Twitter feeds and what our friends were sharing.

And what we had found was if we bothered to click on the links, the media articles about them were slightly overstated and inaccurate. And if we bothered to follow a little bit more, we could actually see the studies itself. And what we as statisticians, epidemiologists, econometricians had tended to find was a lot of these articles made a pretty simple mistake, which is– maybe not a mistake necessarily, but a bit of an overstatement where a lot of these things were really just associative studies where chocolate might be associated or linked to Alzheimer’s or something like that, but they could not find causation.

And based on that experience we developed the claim study, which in some sense is sort of a systematic review version of that informal process. So instead of just one person clicking on what our friends are sharing on Facebook, we decided to find the most popular and most shared articles of 2015 by partnering with an organization called NewsWhip that keeps track of social media sharing.

And instead of just one person reading all of these and giving personal opinions about it, we gathered a team of 21 researchers across six institutions that were all trained in epidemiology and causal inference and econometrics and other related fields, and all had the credentials to sort of show it.

And instead of just sort of reading them and giving our opinions, we developed a systematic review tool to help guide this discussion. And we used a fairly common two plus one style of review where each article has two reviewers, plus one arbitrating reviewer to give sort of the final rank on all of these. So we turned our informal, being just made at our Facebook feeds into research-grade material on which we can build some more research.

NOAH LEAVITT: And so you touched on this idea of causal inference. So for people who aren’t familiar with that, can you explain what causal inference is?

NOAH HABER: Causal inference is one of those things that I think a lot of people assume must be true when they see these things, but don’t necessarily see the difference between the association versus causation. So for example, if I were to say the phrase, chocolate is linked to Alzheimer’s, what that generally will mean is conditional on a bunch of other stuff. People who eat more chocolate might have less Alzheimer’s. I’ve made that up. I have no idea, so don’t quote me. You know, don’t necessarily come back to me and say that I have said that that is true.

But the difference here is that may very well be true, that people who eat chocolate may have less Alzheimer’s. But that doesn’t mean that eating more chocolate will give you less probability of having Alzheimer’s later. That is a causal statement, and that’s a much more difficult problem to try to tackle, particularly in observational studies, which is a lot of what pops up on people’s news feeds.

That’s not to say that it is impossible to do in an observational study. It’s just quite difficult and requires fairly specific circumstances to occur, as well as often some fairly sophisticated analysis.

NOAH LEAVITT: And so I was looking through some of the tables, and kind of the system you set up– and hopefully, I’m explaining this correctly– is that you basically would look at the original research article, kind of look at the strength of causal inference in that research, and then kind of on the flip side, look at the media coverage of it and see whether that coverage maybe overstated, understated, accurately stated the causal inference. I mean, is that accurate? And then so what did you find by doing that analysis?

NOAH HABER: So in some sense, we sort of started from the reverse, right? So in order to figure out what was most popular, we had to start with the media article side effectively, right? Very few people actually study– I’m sorry, actually share the original articles themselves. They typically share a news article about some study or a clip or a tweet, whatever it might be.

So we started from the most popular news articles about a study. You know, we went down sort of the list of the most shared on Facebook and Twitter, and then extracted the study out of that.

We then took the academic article from the media articles, and then applied both the media article we picked up and the academic article to a systematic review process. So the academic article was reviewed for what we called strength of causal inference. So all of these studies were of form X versus Y. Some sort of associational link between an exposure and an outcome.

And we evaluated whether or not we believed through our systematic review process that the causal argument was strong. And then we looked at the language that was used in the academic article and said, you know, did they say or imply through some means that they had a strong causal connection between the two? Or did they sort of couch it in terms that do not suggest causation?

And it’s worth noting that not all studies need to be causal by any means. Plenty of studies in our sample have value just within the association and do not necessarily intend to imply causation.

And once we evaluated the language, we then went to the media article about which that academic article was written, and we looked to see if the language that was used was strongly causal, whether it matched what the original academic article said, and whether or not it was accurate by a few specific questions.

NOAH LEAVITT: And so what were some of the trends you saw when doing that analysis?

NOAH HABER: We saw all kinds of things. So for the most part–

[LAUGH]

–for the most part, what we expected to be true was true, which is sort of an unexciting result in a lot of ways. You know, it’s a lot more fun to be wrong, I think. So what we expected to find was that most of what people were sharing across the internet were relatively low-strength, fairly overstated articles. So on the academic side, there was a slight tendency to overstate the strength of causal inference, relative to what our reviewers believed was the sort of– subjectively, but in a guided fashion believed was the proper strength of causal inference.

A slightly higher tendency among the media articles that people were reading to overstate just a little bit further, or in some cases, quite a bit further. And then, you know, inaccuracies sort of across the board.

It’s worth noting that our study also tended to give benefit of the doubt to basically all parties, right? So our subjective causal inference review tool can only do so much. We can’t review for all aspects of the study. We didn’t do any kind of replication or anything like that. And we’re sort of only subject to the things that our reviewers through our process noticed, effectively. So anything they didn’t notice that was an error would reduce the strength even further.

And we also took the authors basically at their word, that these were the sum total of all the tests that they did, and so on. So in general, we sort of think that what we have is a fairly conservative rating that probably, if we were to investigate even further, we’d find that a lot of these things were a little bit worse.

NOAH LEAVITT: And I know one of the points you make is that this is much more of a what– the question you really can’t answer is the why. And so, I mean, I guess, is that kind of one of your next steps now? Is now that you’ve kind of established what these trends are to kind of maybe dig in deeper and see if you can unpack sort of the why of why media articles may be overstating causal inference and things like that.

NOAH HABER: Absolutely. So the way that we like to think about this is that there’s sort of a pathway from research generation to research consumption, right? So we start off with people like us that are generating all of these studies and writing them, publishing them– or hopefully publishing them, and so on. So once we write the studies– well, so that’s an important point, which I’ll get back to you in a second.

So once we write the studies, we send it off to journals to hopefully get published. The information from the journals, once it is or is not published, then goes to media. And then in theory, social media or however that information is spread through various means, gets to people who can make decisions.

But that’s a long pathway, and each of those steps has a number of potential points at which error can happen, selection can happen, right? So not everything gets published, not everything gets reported on, and not everything gets read. And at each of those steps, things can change, right? So you know, we have sort of a chain of overstatement potentially as you go further down the chain.

But at the same time, there’s also what we like to call feedback, right? So me as a researcher, I wouldn’t bother doing a formal research project that won’t get published. So I will probably tailor what I do towards what is likely to be published.

And it’s potentially possible that academic journals might have some incentives to be more likely to publish things that are going to get reported about that might have some sort of impact elsewhere. And then obviously, we have sort of the incentives for media, which are generating clicks, which is sort of a feedback form by itself.

And what we cannot– we can only really look at the end of this chain in the claim study. We can’t really examine each of the individual parts and how much each of the individual parts contributed to our result. We can only sort of see the state of the world at the end.

Now what we want to do later is we want to break down that pathway quite a bit more. So we can do a version of the study where we start from the top end of academia, maybe at the journal publication phase, and look at the state of causal inference at the journal point, the end of the academic phase at the point of publication in journals, and see what those all look like.

And then we can sort of infer downward, OK. So this is what it looks like at the beginning of the academic chain. How does that sort of move down? And we can also examine how things get more popular, right? So we only picked amongst the most popularly shared news articles. What if we picked studies that were less popularly shared, and see what the differences between the language and the content of those types of studies. And we can start to break down that chain a little bit.

NOAH LEAVITT: And so you kind of touched on it there, but for kind of the people who are more interested in this, or even just, I mean, people who want to think more critically about the science or health news they’re consuming, I guess, what with some of your takeaways be for those people? And I think this goes for whether you’re reading an academic article or you’re just reading that article that a friend shared on Facebook. I mean, what are some takeaways for just kind of the people on the consumption end?

NOAH HABER: I think the big takeaway is assume that if it’s on your Facebook feed or your Twitter feed, and it’s health study says X linked to Y, you should assume that that’s not a causal relationship. If it is meaningful that it would be a causal relationship, then you should probably just ignore the article all together.

So you know, on of the examples I like to bring up is if you see a study that is about the association between butter and cardiovascular disease, there really is no purpose in having just an association between those two things. That doesn’t really serve any practical purpose. You really want to know the causal effect. Do I eat more or less butter to cause or prevent cardiovascular disease?

And in those cases, if the causality is sort of intuitive and assumed, then you should probably ignore the results of that particular article. At least the version of it that you see on Facebook and Twitter.

[LAUGH]

NOAH LEAVITT: One of the things, you know, you mentioned earlier is that just because a study doesn’t find a causal link doesn’t mean that it doesn’t add value. There’s a lot of studies that have kind of association. So I mean, can you kind of, I guess, maybe talk a little bit about the balance there of the value of those association studies? You know, but also the value of studies that do show a causal link.

So how do we strike that balance both on the kind of academic research side, but then on the media side, and then as news consumers? So kind of, how do we find that balance between those different kinds of findings?

NOAH HABER: So that’s sort of a Pandora’s box of a question. Let me give you two ways to think about this. So one way to think about this is there do exist studies for which the association is useful by itself.

So a couple of examples here are if you’re looking at targeting, right? So if we want to know, how do we find the people that an intervention is best applied to, then we might be able to find by association ways that we can predict who is going to get some outcome maybe based on some idea of an exposure. And then that lets us better target interventions towards people. And that, the association is useful by itself, because we are only trying to predict who the most useful sort of targets are.

And there are all sorts of other reasons that association by itself is very useful. So another one is disparities, right? So if we want to know why different types of people are having different outcomes, then association is often useful and enough for those sorts of things.

There’s another idea, which is that individual studies really are not the best or not the best way to go about a lot of these ideas. And it’s really about scientific consensus.

So an individual study might be able to say, OK, we have this hypothesis. Our results show this association, so it’s still plausible that this causal relationship exists. And that helps us develop hypotheses for better tests and more tests until we have enough evidence to be able to say that this particular thing is true, that there is a causal relationship, or there is a reasonably sized causal relationship between these two things.

And that can often take years and many studies and much more difficult studies to perform, you know, that what you’re clicking on in the end of a study says X versus Y is really not the sort of information maybe that you should be seeing. We should really be seeing sort of the end of the scientific consensus process, but we tend to see the sort of overhyped eureka study says types of things. So science is often much slower than maybe we would like.

NOAH LEAVITT: Well, I mean, I feel like that’s such an important point, that I think people maybe want science to move faster or perceive that it moves faster. So I mean, is that– I guess, is that part of that better communicating from those of us at research institutions, kind of communicating the pace of science, and that oftentimes these things happen incrementally or might take several studies to kind of build up a base of evidence in a particular area?

NOAH HABER: Yeah, so that’s absolutely true. I think that we need to be happier in general with a slower pace in science and less immediate discovery. At the same time, though, we as economics tend to make a lot of mistakes in this field. And it’s very tempting to try to make these sort of big headline sort of associative studies– associational studies, excuse me. And there’s a lot we can do to help do better causal inference where it is possible.

In cases where we’re doing a lot of weak causal inference studies, we could be doing something that maybe a little bit stronger elsewhere. And it’s pretty remarkable, particularly in certain subfields of health science, the sorts of methods and ideas that are used as the standard when much better– not universally better, but more useful ideas exist and are often totally ignored.

NOAH LEAVITT: As you’ve been doing this work, you and your co-authors, has it changed at all how you view past work you’ve done? How you’ll approach future work that maybe doesn’t kind of fall under this umbrella? Has this kind of made an impact just kind of on your personal path as a researcher?

NOAH HABER: Absolutely.

[LAUGH]

For me personally, early in my career, I used to work on cost-effectiveness modeling based on clinical trial results and a bunch of other sorts of things. And before, I was sort of a causal inference specialist. And some of those– I’m now realizing a lot of mistakes that I personally made.

And actually, I plan on writing some of that up and sort of digging into the failures of assumptions of some of my own work back in the day, and digging into that a little bit more. And also how I made those mistakes and why I sort of believed the things I did, and why I was wrong, and maybe what could have happened then that I could have avoided– that could have helped me avoid those mistakes. But definitely.

It also has helped us understand a lot of the subtleties in the development process, which I think that we did not realize and a lot of people didn’t realize as well in how these things are produced, how a lot of this information moves around and changes throughout sort of this pathway idea. That has made us very cognizant of how we can better communicate our own research.

One of the things we’re working on now– so we have a website. We started a website that is sort of the companion to this study called metacausal.com. M-E-T-A, causal, C-A-U-S-A-L .com.

And what that website does is it contains a public explainer for the whole study so that instead of having to go through and reading the abstract on the study itself, we put the whole study in relatively, we hope, plain language for basically anybody to be able to read.

But we also– we went a couple of steps further. We’re really trying to experiment while we’re doing this. So one of the things we did is we sort of expect that there are a lot of ways that people can misinterpret our study. So we gave people ways that they could frame talking about our work that we felt were accurate and easy to understand, as well as a list of ways that are tempting but inaccurate ways of talking about our work, and pointing those out beforehand.

So we are in a weird world right now with very complicated public relationships with science, to put it mildly. And there is a sentiment that everything– you know, that science is doing it all wrong, and so on in certain political corners at the moment. And so it’s a very tempting thing to believe that, oh. This just shows that science is terrible.

And we’ve actually gotten some interesting– I don’t know. Fan mail might not be the best phrase, but some interesting mail in that regard from some folks who are maybe in that line of thought. But you know, that’s an inaccurate description of our work. We don’t really show that. We see how people could interpret it that way, and we also see how people could feel that way, right?

So if the majority of what people are seeing is relatively weak and overstated– and that’s probably true, assuming that our study holds up well. It’s replicated. What people are seeing is not necessarily representative of the whole of academic literature or the whole of media even, for example. It’s really sort of a very distilled version of both of those things, but distilled in a potentially bad way.

{***Pause/Music***}

{***Noah***}

That was my conversation with Noah Haber about his research looking at how health studies are reported by the media and on social media.

{***Amie***}

One interesting note from Haber—and something that’s obviously relevant to us: Harvard was the most represented institution in studies. And top journals were also well-represented.

{***Noah***}

Haber says this shouldn’t suggest anything about the quality of research at Harvard—or the quality of research published by these journals.

Rather he says that the prestige of a journal and institution can send strong signals which may influence how research is covered and increase overstatement.

{***Amie***}

Haber also added that in this study they didn’t look at the press releases sent out by each institution, so it’s hard to say how that may have influenced coverage.

{***Noah***}

And you heard Haber mention that they have a website: metacausal.com. That’s m-e-t-a-c-a-u-s-a-l-dot-com. And it’s focused on exploring the intersection of media/social media/science as it relates to this research.

One cool thing they’ve done is analyzed the coverage of this study itself—basically seeing if the media got it right.

{***Amie***}

We’ll have a link to that website, as well as the full study, on our website: hsph.me/thisweekinhealth.

{***Noah***}

That’s all for this week’s episode. A reminder that you can always find us on iTunes, Soundcloud, Stitcher, and Spotify.