Little Ogg

Format: Cartoons

,

Language/s: English

,

Target Audience: Self-directed learning

|

Short Description:

Lack of evidence of effects is not the same as evidence of no effects.

Key Concepts addressed:

• 2-18 Don’t confuse “no evidence” with “no effect”

Details

The mammoth has a good point. Ogg’s father is making a classic error of logic. Not having found proof that something really happens, is not the same as having definitive proof that this thing cannot possibly happen.

Ogg’s family doesn’t have the benefit of Aristotle’s explanation of deductive reasoning. But even two thousand years after Aristotle got started, we still often fall into this trap.

In evidence-based medicine, a part of this problem is touched on by the saying, “absence of evidence is not evidence of absence.” A study says “there’s no evidence” of a positive effect, and people jump to the conclusion – “it doesn’t work.” Baby Ogg gets thrown out with the bathwater.

The same thing is happening when there are no statistically significant serious adverse effects reported, and people infer from that, “it’s safe.”

This situation is the opposite of the problem of reading too much into a finding of statistical significance (covered here in Statistically Funny). Only in this case, people areover-interpreting non-significance. Maybe the researchers simply didn’t study enough of the right people, or they weren’t looking at the outcomes that later turn out to be critical

Researchers themselves can over-interpret negative results. Or they might phrase their conclusions carelessly. Even if they avoid the language pitfalls here, journalists could miss the nuance (or think the researchers are just being wishy-washy) and spread the wrong message. And even if everyone else phrased it carefully, the reader might jump to that conclusion anyway.

When researchers say “there is no evidence that…”, they generally mean they didn’t find any, or enough of, a particular type of evidence that they would find convincing. Obviously, no one can ever be sure they have even seen all the evidence. And it doesn’t mean everyone would agree with their conclusion, either. To be reasonably sure of a negative, you might need quite a lot of evidence.

On the other hand, the probability of something being extremely unlikely to be real based on quite a lot of knowledge – that there’s a community of giant blue swans with orange and pink polka dots on the Nile, say – increases the confidence you might have in even a small study exploring that hypothesis.

Which brings us to the other side of this coin. Proving that something doesn’t exist to the satisfaction of people who perhaps need to believe it most earnestly, can be quite impossible. People trying to disprove the claim that vaccination causes autism, for example, are finding that despite the Enlightenment, our rational side can be vulnerable to highjacking. Voltaire hit that nail on the head in the 18th century: “The interest I have to believe a thing is no proof that such a thing exists.”

Cartoons are available for use, with credit to Hilda Bastian.