18.9: Effect Sizes

Last updated
Save as PDF

Page ID: 95189

$ \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } $ $ \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} $$\newcommand{\id}{\mathrm{id}}$ $ \newcommand{\Span}{\mathrm{span}}$ $ \newcommand{\kernel}{\mathrm{null}\,}$ $ \newcommand{\range}{\mathrm{range}\,}$ $ \newcommand{\RealPart}{\mathrm{Re}}$ $ \newcommand{\ImaginaryPart}{\mathrm{Im}}$ $ \newcommand{\Argument}{\mathrm{Arg}}$ $ \newcommand{\norm}[1]{\| #1 \|}$ $ \newcommand{\inner}[2]{\langle #1, #2 \rangle}$ $ \newcommand{\Span}{\mathrm{span}}$ $\newcommand{\id}{\mathrm{id}}$ $ \newcommand{\Span}{\mathrm{span}}$ $ \newcommand{\kernel}{\mathrm{null}\,}$ $ \newcommand{\range}{\mathrm{range}\,}$ $ \newcommand{\RealPart}{\mathrm{Re}}$ $ \newcommand{\ImaginaryPart}{\mathrm{Im}}$ $ \newcommand{\Argument}{\mathrm{Arg}}$ $ \newcommand{\norm}[1]{\| #1 \|}$ $ \newcommand{\inner}[2]{\langle #1, #2 \rangle}$ $ \newcommand{\Span}{\mathrm{span}}$$\newcommand{\AA}{\unicode[.8,0]{x212B}}$

You hear on the news that a famous and trustworthy medical journal has recently published a study showing that a large daily dose of a moderately expensive dietary supplement, vitamin Q, will cut your risks of developing XYZ syndrome in half. In other words, people who do not take the vitamin will be twice as likely to develop this syndrome. The syndrome is painful but not life threatening, and the vitamin costs $20 a day. Should you start taking the vitamin?

The answer depends on a variety of things (e.g., whether you can afford the vitamin). But the first question you should always ask about such reports is:

What is the base rate of XYZ syndrome?

Suppose that only 1% of the population (who don’t take vitamin Q) ever develops XYZ syndrome. Then if you take the vitamin, you cut your chances in half, down to 0.5%. In other words, you go from a 1 in 100 chance of developing the syndrome to a 1 in 200 chance. These numbersare so small that taking the vitamin may not be worth your time. By contrast, suppose that the base rate was 20%. If you could cut that in half, down to 10%, you would go from a 1 in 5 risk to a 1 in 10. The numbers here are big enough that you might want to give the vitamin some serious thought.

This is a fictitious example, but there are many real-life cases that illustrate the same point. Suppose that you learn that people who don’t get enough vitamin C are ten times more likely to get botulism (which results from a deadly poison) or rabies. What are the first questions you should ask? What is the base rate for botulism? What is the base rate for rabies? It turns out that no more than three or four Americans die of either botulism or rabies each year. So, even if some factor made rabies ten times as likely to kill you, your chances would still be about 30 out of 250 million.

But what if you learned of some precaution that could decrease your chances of having a heart attack by 20%? Again, the relevant question is: what is the base rate? It turns out that about 1 in 3 Americans die from a heart attack, so if you could decrease your chances of heart disease by 20% it would be worth doing (we will return to this issue in more detail in the chapter on risks).

In these examples, we are at least given percentages that tell us something about the impact of various drugs and the like. But media reports of experimental results often don’t tell us about the magnitude of effects. The anchorwoman tells us that the manipulation of a certain experimental variable (e.g., taking vitamin Q) reduced cancer, and that this result is statistically significant. But statistical significance does not mean the same thing as practical significance.

To say that a result is statistically significant simply means that it is unlikely that it was due to chance (to sampling variability). But with large samples, small and trivial differences are often statistically significant. For example, a study might find that vitamin R reduces the risks for XYZ syndrome by 0.20% (i.e., it reduces it by 15 of 1%). If our sample is large enough, this result may well be statistically significant. But the effect is so small that it won’t be of much practical significance to anyone.