Promoting Healthy Skepticism in the News: Helping Journalists Get It Right
Authors: Steven Woloshin, Lisa M. Schwartz, Barnett S. Kramer
Journal: JNCI Journal of the National Cancer Institute (2009)

The constant barrage of fear and hope in the health news can make your head spin. Nearly every day we hear that almost everything we do (or fail to do) leads to cancer, suffering, and death. Just as frequently (and often on the very same day) we hear about new breakthroughs, tests, and miracle drugs that may save us. Fortunately, we know the fears are usually wildly exaggerated. Unfortunately, we know the hopes usually are, too.

Where does this exaggerated fear and hope come from? Consider the recent media coverage (Table 1) of a New England Journal of Medicine article about a new cancer treatment—olaparib [a poly (ADP-ribose) polymerase—or PARP inhibitor] (1). In this phase I uncontrolled study of 60 patients with a variety of treatment refractory solid tumors, the drug appeared to have an effect in one subgroup: 12 of the 19 patients with BRCA1 or BRCA2 mutations and breast, ovarian, or prostate cancer experienced either improvement or no tumor progression (according to radiological or tumor markers) sustained for at least 4 months.



Table 1

A recent medical journal article and excerpts of subsequent media coverage



The study received prominent television coverage—ABC, CBS, and NBC national news all ran stories. The coverage was full of hope: NBC began … “some are calling this the most important cancer breakthrough of the decade” (2). But the effect of treatment was not quantified, so viewers had no way to know how well it worked. All they heard were two dramatic anecdotes about patients who did well. NBC's chief medical correspondent concluded that “these drugs look like they will eventually save thousands of lives.”

The enthusiasm in the NBC story was only tempered by a single caution: the study was small. Two fundamental cautions were missing. First, without a control group, there is no way to know if the drug accounted for the findings. Second, there is no way to know if “stable or improved radiological and tumor markers” translate into a clinically meaningful outcome such as longer life.

Of course the news is also full of fear. Reporting on a study published in the Journal (4), CNN's Sanjay Gupta told American women to worry more about cancer: “there is no level of alcohol consumption that can be considered safe when it comes to cancer” (4). That is a lot of worrying: Almost three-quarters of US adult women have consumed alcohol (5). CNN was not alone. The Today Show also covered the story (6). And The Washington Post ran a front-page article “A drink a day raises women's risk of cancer.” Unfortunately, the coverage did not provide the magnitude of the risk. Comparing the highest level of drinking (≥15 drinks a week) to the lowest (one to two drinks per week), the investigators observed a 0.6% absolute increase in the risk breast cancer diagnosis: from 2% to 2.6% over more than 7 years.

Perhaps, most importantly, none of these news stories highlighted the most fundamental limitation of this observational study: confounding. There may have been something else about the women who happened to drink alcohol—compared with those who did not—that explains the findings. If more health conscious women drank alcohol regularly (eg, one drink a day because of its purported heart benefit), it is possible that these women were more likely to adhere to regular screening mammography. So it could be that more screening—not more alcohol—explains the higher risk of breast cancer diagnosis. Journalists (and journals) should remind readers that confounding is always a threat to the validity of observational studies.

It would be easy to pin all the blame for exaggeration on journalists. After all, they have to grab their reader's (or listener’s) attention. Screaming headlines and breathless reporting come in handy. And many health journalists lack the medical or statistical training needed to appraise research critically. Curiously, many fail to approach medical research with the same skepticism they routinely apply to political reporting. Nonetheless, blaming journalists for all exaggeration would be unfair. Many health journalists (and their editors) do a great job. For example, only one major newspaper covered the New England Journal of Medicine PARP study (3). The rest passed (in our opinion the right decision). And while not perfect, this story was at least somewhat tentative (“may help … , early stage work … , preliminary”) and quantified the result (Table 1).

When it comes to exaggeration of health hazards and medical breakthroughs, there is plenty of blame to go around. Researchers contribute to the problem. Research takes years of dedication and sacrifice—investigators would not last long if they did not believe passionately in what they do. And good press can help advance careers. The combination of strong beliefs and self-interest can be an irresistible recipe for exaggeration. Although we know of no systematic examination of how investigators talk about their own research to the media, a study of press releases issued by academic medical centers found exaggeration to be common: Almost all press releases included investigator quotes—one-quarter of which overstated the importance of the findings (7).

Exaggeration, however, starts with medical journal articles themselves. This is ironic because the journals work hard to fairly represent study findings, provide context, and acknowledge important limitations.

But journals sometimes drop the ball. Important elements that journalists (and, really, all readers) need are sometimes missing or hard to find in the published articles. For example, in six high-profile journals, two-thirds of articles reporting ratio measures failed to provide the underlying absolute risks in the abstract (8) [and one-third failed to provide them anywhere in the article—despite the recommendation to do so whenever possible by the International Committee of Medical Journal Editors (9)]. Effects expressed in relative terms alone have been repeatedly shown to seem more impressive than the same effects expressed in absolute terms in studies of physicians (10,11), policy makers (12), and patients (13,14). Nor are study limitations routinely highlighted in journal abstracts (the exception is the Annals of Internal Medicine)—and sometimes they too are missing from articles altogether. All studies have limitations, which need to be highlighted to ensure that readers are aware of them and take them into account when interpreting findings.

Journal press releases—the most direct way that journals communicate with journalists—can also be a problem. Press releases issued by nine of the most prominent journals (according to the Institute for Scientific Information impact factor listings) were also missing fundamental information (15): Only half of the press releases reporting on differences between study groups provided absolute risks; less than one-quarter noted any study limitation.

Can we really expect journalists to do a better job than the medical journals, researchers, or their university public relations offices?