Insights on the Burden of Proof

Key Takeaways:

• We designed the Burden of Proof study to help consumers makes sense of confusing health guidance by assigning a star-rating to pairs of risks and outcomes.
• We reviewed thousands of studies on risks like smoking and eating red meat to determine how strong the evidence was that those risks lead to health impacts, and whether it’s worth changing your behavior.
• Five-star ratings – like smoking & lung cancer and high blood pressure & heart attacks – indicate that there is strong evidence of association.
• One-star ratings indicate that there may be no association at all, or that more evidence is needed on the topic.

Compare the ratings

This transcript has been lightly edited for clarity

This is a project that started nearly five years ago where we were looking at all the controversy in the literature, both in the media and in the scientific literature about certain risk-outcome associations. It started with diet, but there’s also a lot of controversy around air pollution and controversy about other associations. And we wanted to find a way to help the public, to help research funders and to help decision-makers in government navigate this very confusing field.

Remember back to the Time magazine cover about Don’t Eat Bacon a few years back. And, you know, this is happening all the time. You see one study swinging you in one direction, another study swinging you back. And so we really wanted to find a way to help everybody navigate that complexity. So the burden of proof study is our attempt to look at all the evidence that’s out there on a risk-outcome relationship.

We’ve done this for nearly 200 risk-outcome pairs and try and figure out how strong is the evidence and to rate the strength of that relationship on a five-star scale from one star where the average interpretation of the evidence is that it’s either harmful or protective. But perhaps if you take a conservative view of the evidence, there’s no relationship at all. Right through to the five-star association where there is overwhelmingly convincing evidence of the risk-outcome relationship. 

The key findings is that we see a really wide range of strength of evidence around these relationships from smoking and lung cancer or high blood pressure and heart attacks, which are five-star relationships, down to quite a large number of relationships ­­– many of them are between diet or components of diet and outcome – where the relationships are not very strong at all.

In fact, in the case of the one-star associations, they may simply be something that will change in the future as a new study comes along. And so the conservative view of those is there may be no relationship there at all.

I think the most surprising aspect of the findings is that there are some associations, think about red meat and heart attacks or physical activity and diabetes, that many of us took as grounded in really strong evidence. But at the end of this process of looking at the thousands of articles that we review and then coming up with this overall assessment of how strong is the evidence and risk and outcome relationships, some of those things that we were very convinced about in the past turn out to not have such compelling evidence behind them. 

If you start with the associations at the other end of the spectrum, the five-star or the four-star or the three-star associations, these are things that everybody who’s concerned about their health will want to take into account and to avoid them if they’re harmful and to embrace them if they’re good for you or protective.

When you get into the zone of the two-star and the one-star associations, then it’s going to be more about your personal risk tolerance. If you are somebody that really is risk-averse and wants to avoid any possible risk to their health, you’ll want to avoid the harmful one-star and two-star risks and embrace the protective one- and two-star associations.

But if you are somebody that’s willing to accept some risk or that there might be risk, then you can take a more nuanced approach to how you view a one-star association. Personally, for me as an individual, I’m not going to change my behavior for a one-star association. But I’m probably going to act on two stars and above.

So we see this burden of proof, risk, function, work, and the star rating is something that we are going to keep up to date so that the public has access to this sort of assessment on a regular basis and can feel confident that it reflects our current knowledge. And there will be or are associated online tools for those who are motivated to get into the details and see what studies are out there and how it is we’ve come up with our assessment around the star rating for risk-outcome relationship. So we hope this is a service to the public and will continue into the future.

Read the research