Health News Review is a website published by Gary Schwitzer that is dedicated to improving the accuracy of news stories about medical treatments, tests, products and procedures and helping consumers evaluate the evidence for and against new ideas in health care. To this end, the foundation established a set of ten criteria to apply to medical stories reported in the popular media in order to evaluate their accuracy, balance, and completeness. Editor-at-Large for MedPage Today, George Lundberg MD, refers to these criteria as “10 Rules to Expose Medical Junk News” and argues that, “While [Schwitzer's] approach cannot prevent fraud, liars, and fabricators, a careful use of his criteria can help the reader filter out what is likely to be real junk, or even worse, harmful.” In an environment of medical hype, I’d have to agree.
Check out “The Rules” and consider how they may be used to evaluate a news story about one of the most sensationalized and divisive topics in the breast cancer world: screening mammograms.
Rules for Medical Reporting:
- How available is the treatment/test/product/procedure to the likely reader/viewer/listener at the time of the report?
- What is the cost or charge for the test/treatment/product or procedure mentioned in the story? To the patient? The insurance company? The government?
- Is there evidence of disease mongering in the story? Does it oversell or exaggerate a condition or create unwarranted fear?
- Does the story seem to grasp and convey the quality of the evidence supporting the basis for the study?
- Does the article provide appropriate balance about harms that might be caused by the treatment/test/product/procedure that constitutes the basis for the story?
- Does the story establish the true novelty of the approach? Much that is purported to be new, really is not.
- How does the story frame the relative quantitative value of a new treatment, test, product, or procedure and place the benefits in context with others, especially dealing with absolute and relative values?
- Did the author and editor of the medical news story rely solely or largely on a press release or did they also seek and quote other sources?
- Was there an independent source and were any possible conflicts of interests of sources disclosed in the article?
- Does the story provide the context of treatment/test/product/procedure other than those that are being reported?
Health News Review goes into detail about how to use each of these criteria to evaluate health news. For instance, a story that uses disease mongering (#3) may present spurious statistics, exaggerate human consequences, or present ”worst-case” scenarios as if they represented everyone with the condition. A story that is unsatisfactory in providing balanced coverage of harms (#5) may fail to mention, quantify, or describe the severity of potential harms or side effects, or it may rely on an investigator’s comment or anecdotal evidence of safety.
How do the criteria work with an actual news story?
Consider a story from Reuters (June 28, 2011), “Benefit of Mammograms Even Greater Than Thought.” The story reported on the results of a long-term Swedish study that compared women who were invited for routine screening and compared their outcomes to those who were not invited for routine screening but still received usual care. The study found a significant reduction in mortality 29 years after women received their first mammogram. At first glance it appears to be confirmation that routine screening significantly reduces death from breast cancer. However, Health News Review considered the story to be confusing, biased, and incomplete. When applying the ten criteria for medical reporting the short-comings are clear.
Health News Review rated the story as satisfactory on only three of the criteria: (#3) Avoiding disease-mongering, (#6) establishing the novelty of approach, and (#8) not relying solely on a press release for story content. However, the story was unsatisfactory on each of the remaining criteria: (#2) giving no cost information, (#4) omitting a discussion of the study limitations and quality of evidence, (#5) failing to provide a balanced review of the benefits and harms, (#7) failing to contextualize the quantitative data, (#9) lacking an independent perspective, and (#10) failing to provide a greater context for the approach, particularly in light of the findings from the U.S. Preventive Services Task Force. Health News Review stated further that,
“The available evidence on the utility of screening mammograms is conflicting and complex and this story muddies the waters even more by largely ignoring the discussion of harms associated with mammography and introducing a suggested screening schedule, which this study was not even designed to address. In addition, the story line about the mammography debate is not, “Does screening save lives or not?” Rather, it is, “Is it worth it for an individual woman to be screened, when for some (e.g. 40-50 year olds), the benefit is quite small?” The U.S. Preventive Services Task Force did not say that screening did not reduce mortality. It said that for some women, the trade-off between that benefit and the potential harms is a close call and should be weighed by the woman and her physician.”
Taking the story at face value would give a false impression of the study results and their contribution to the body of research on screening. Given the ongoing debate about the role of screening for all women and the ongoing concern about how women might make definitive decisions about when, and how often, to get screened in the face of it, the story has the potential to do more harm than good.
Apply “The Rules” & Avoid Junk News.