14.7: Assessing Alternative Explanations

Suppose your TV-headache correlation has a proper temporal relation, regularity across several studies, strength, and coherence with your background knowledge. Despite this, a factor lurking in the background could be causing both the headaches and the TV to be on. For example, maybe family quarrels cause the headaches and also cause you to turn on the TV to escape the quarrels.

Unless you had thought of this possibility, or alternative hypothesis, and unless you had also thought of ways to test it and had then ruled it out successfully, your confidently asserting that sleeping next to the TV caused your headaches would be committing the post hoc fallacy.

Here is one test that would help confirm some of the hypotheses and rule out others. Suppose you were to hire several people to sleep in a laboratory near TV sets like yours. For half the subjects in this test, the TV would be on; for the other half, the set would be off. What would you predict about the results, given each of the alternative hypotheses? The prediction from your favored hypothesis is that there would be a lot more headaches among those people whose set is on and a lot fewer among those whose set is off. However, from the hypothesis that family quarrels are to blame, you would predict that the two groups of subjects would have the same frequency of headaches. Therefore, your test can produce results that will be consistent with one hypothesis while refuting the other. That ability to discriminate between two competing hypotheses is a sign of a good scientific test.

Another point illustrated here is that the best way to infer from a correlation between C and E to the claim that C causes E is to do a controlled experiment. In a controlled experiment, you divide your subjects into two groups that are essentially equivalent with one major exception: the suspected cause C is present in one group but not the other. The latter group is called the control group. The group with the suspected cause is called the experimental group, or, in the case of a drug experiment, the treatment group. Run the test and see whether the effect E occurs much more in the experimental group than in the control group. If you notice a big difference, then it's likely that C really does cause E.

Let's suppose that you decide it is too expensive to do this sort of testing—you don't have the lab or the TV sets or the time it would take to run the experiment. Instead you decide to put an advertisement in a magazine asking readers to write if they have noticed headaches after sleeping near a TV set while it is on. You subsequently receive forty letters from people who have had experiences similar to yours. If you were to take these letters to be sufficient evidence that sleeping near the TV does cause headaches, you'd be jumping to conclusions. This second test isn't very good. All you’ve found are data consistent with your favorite hypothesis. You haven't actively tried to refute your hypothesis. So, you haven't really followed the scientific method. Your results don't qualify as a scientific proof.

This same point about experimental design can be made by considering whether it's true that every emerald is green. Let's say that, although you do suspect this hypothesis to be true, you want to test it to find out. Which would be the best test? (1) Try to find positive evidence by putting an ad in the paper asking people to let you know whether they own a green emerald. (2) Try to find negative evidence by putting an ad in the paper asking people to let you know whether they own an emerald that is not green. One response to the second ad would be much more informative than many responses to the first, wouldn't it? So strategy 2 is the best. This is the strategy of searching for inconsistent data rather than confirming data.

Probably the single greatest source of error in science is the natural human failing of trying to confirm one's own guess about the correct explanation of something while failing to pay enough attention to trying to disconfirm that explanation. That is, we all have a tendency to latch onto a hypothesis and try to find data consistent with it without also actively searching for data that would show us to be wrong. We have this tendency because we really don't want to be shown wrong, because we are lazy about using our imagination to produce alternative explanations, and because it takes a lot of physical effort and expense to create tests that would show our suspicions to be wrong. In short, we have a natural tendency to look for a shortcut to the truth. Unfortunately, there is no effective substitute for the long, difficult path of conjectures and refutations. This is the path of guessing the reasonable alternative suggestions and systematically trying to refute them. The scientific method is this method of conjectures and refutations.