Fair tests of treatments: a quick guide for journalists

Format: Texts

,

Language/s: English

,

Target Audience: Journalists and science writers

|

Short Description:

When deadlines are pressing, how can journalists tell whether to believe claims about the effects of a new treatment or breakthrough?

Key Concepts addressed:

• Claims: are they justified?

Details

When deadlines are pressing, how can you tell whether to believe the claims about a new medical treatment or breakthrough?

As journalists you are immensely influential on public understanding of new treatments.  So it’s essential that you can tell fact from fiction.

Researchers rarely say that their research was non-significant or biased, and even more rarely do manufacturers that their product doesn’t work.  This in spite of the fact that about half of the time, this is the truth behind their claims.

Can you tell when this is happening?  This TTi webpage for journalists aims to help you.

How can I get fact-checks of health news?

If you don’t have the time or skills to do a full fact-check yourself, there is a range of trustworthy sources of evidence-based analysis of current health news stories.

We recommend these sources because they formally critically appraise the research behind news stories and they are free from conflicts of interest.

You can use them as sources for stories or cross checking stories, and for learning the pitfalls of medical research and how to spot problems.

The aim of critical appraisal is to determine whether:

1. The researchers did all they could to eliminate bias from their study
2. The observed results could have occurred if there was no difference between the treatments (in other words, could they have occurred by chance)?
3. The reported benefit is important in clinical practice.

Critical appraisal should adopt explicit criteria for evaluating these questions.  The best-known criteria are the ones by the Critical Appraisal Skills Programme (www.casp-uk.net).

Where can I find trustworthy information?

Finding reliable, user-friendly sources of information about clinical conditions and treatments can be difficult.  We recommend regular doses of the following:

How can I check stories myself?

Use Ray Moynihan’s tipsheet checklist to assess the importance and reliability of a claim:

1. What is the size of the potential benefit offered by the therapy, and for what types of patients is it beneficial?
2. What are the potential harms associated with the therapy?
3. What are the links between your sources of information about the therapy and those promoting it?
4. How strong is the evidence to support the claims being made about the therapy, and how does it relate to other available evidence?
5. What is the natural history of the condition for which the therapy is being offered, and is there potential for what some have called “disease mongering”?
6. What are the alternatives to the therapy being offered (e.g., no action or watchful waiting, generic drugs, non-drug options, complementary therapies)?
7. What are the costs of the therapy and are the potential benefits worth the cost?

“Getting answers to these questions can help reporters craft stories that are more likely to serve the needs of their readers, viewers and listeners.  They can also help reporters identify warning signs that the enthusiasm of researchers or the promotional interests of companies and institutions may be getting ahead of the science.” Commonwealth Fund.  Download the tipsheet as a PDF.

Some real life examples

This section documents historical cases that illustrate the need for good reporting of health claims regarding treatments.

MMR

In 1998, Dr Andrew Wakefield claimed that the combined vaccine for Measles, Mumps and Rubella (MMR) might cause autism in young children.

In spite of the vast swathes of evidence contradicting this claim, and the complete absence of evidence to support it, the claim received enormous media attention.

This led to a substantial drop in the uptake of the combined vaccine, a reduction in levels of immunity and subsequent increases in the incidence of these harmful diseases.

This illustrates:

• Conflict of interest:  Dr Wakefield had commercial interest in selling single vaccinations against each disease
• Uncritical acceptance of medical claims by the media, leading to the creation of controversy where none existed
• Measurable harm from inaccurate reporting:  The incidence of measles has increased as our population’s immunity decreased.  The outbreak of measles in South Wales was one notable example.

Read more:

• Sense About Science briefing paper about the MMR scandal (PDF, 9 pages)
• Cartoon:  The Facts in the Case of Dr Andrew Wakefield

Other examples

Please see Why do we need fair tests of treatments? which contains numerous examples of the myriad ways in which patients have been harmed by our failure to tests treatments properly.