15.6.1: Designing a Scientific Test

Last updated
Save as PDF

Page ID: 36304

Bradley H. Dowden
California State University Sacramento

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

It is easy to agree that scientific generalizations should be tested before they are proclaimed as true, and it is easy to agree that the explanations based on those generalizations also should be tested. However, how do you actually go about testing them? The answer is not as straightforward as one might imagine. The way to properly test a generalization differs dramatically depending on whether the generalization is universal (all A are B) or non-universal (some but not all A are B). When attempting to confirm a universal generalization, it is always better to focus on refuting the claim than on finding more examples consistent with it. That is, look for negative evidence, not positive evidence. For example, if you are interested in whether all cases of malaria can be cured by drinking quinine, it would be a waste of research money to seek confirming examples. Even 20,000 such examples would be immediately shot down by finding just one person who drank quinine but was not cured. On the other hand, suppose the generalization were non-universal instead of universal, that is, that most cases of malaria can be cured by drinking quinine. Then the one case in which someone drinks quinine and is not cured would not destroy the generalization. With a non-universal generalization the name of the game would be the ratio of cures to failures. In this case, 20,000 examples would go a long way toward improving the ratio.

There are other difficulties with testing. For example, today's astronomers say that all other galaxies on average are speeding away from our Milky Way galaxy because of the Big Bang explosion. This explosion occurred 13.7 billion years ago, when the universe was smaller than the size of a pea. Can this explanation be tested to see whether it is correct? You cannot test it by rerunning the birth of the universe. But you can test its predictions. One prediction that follows from the Big Bang hypothesis is that microwave radiation of a certain frequency will be bombarding Earth from all directions. This test has been run successfully, which is one important reason why today's astronomers generally accept the Big Bang as the explanation for their observations that all the galaxies on average are speeding away from us. There are several other reasons for the Big Bang theory having to do with other predictions it makes of phenomena that do not have good explanations by competing theories.

We say a hypothesis is confirmed or proved if several diverse predictions are tested and all are found to agree with the data while none disagree. Similarly, a hypothesis gets refuted if any of the actual test results do not agree with the prediction. However, this summary is superficial—let's see why.