15.6.3: Three Conditions for a Well-Designed Test

Last updated
Save as PDF

Page ID: 36306

Bradley H. Dowden
California State University Sacramento

\( \newcommand{\vecs}[1]{\overset { \scriptstyle \rightharpoonup} {\mathbf{#1}} } \) \( \newcommand{\vecd}[1]{\overset{-\!-\!\rightharpoonup}{\vphantom{a}\smash {#1}}} \)\(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\) \(\newcommand{\id}{\mathrm{id}}\) \( \newcommand{\Span}{\mathrm{span}}\) \( \newcommand{\kernel}{\mathrm{null}\,}\) \( \newcommand{\range}{\mathrm{range}\,}\) \( \newcommand{\RealPart}{\mathrm{Re}}\) \( \newcommand{\ImaginaryPart}{\mathrm{Im}}\) \( \newcommand{\Argument}{\mathrm{Arg}}\) \( \newcommand{\norm}[1]{\| #1 \|}\) \( \newcommand{\inner}[2]{\langle #1, #2 \rangle}\) \( \newcommand{\Span}{\mathrm{span}}\)\(\newcommand{\AA}{\unicode[.8,0]{x212B}}\)

As a good rule of thumb, three definite conditions should hold in any well-designed test. First, if you use an experiment or observation to test some claim, you should be able to deduce the predicted test result from the combination of the claim plus a description of the relevant aspects of the test's initial conditions. That is, if the claim is really true, the predicted test result should follow. Second, the predicted test result should not be expected no matter what; instead, the predicted result should be unlikely if the claim is false. For example, a test that predicts water will flow downhill is a useless test because water is expected to do so no matter what. Third, it should be practical to check on whether the test did or did not come out as predicted, and this checking should not need to presume the truth of the claim being tested. It does no good to predict something that nobody can check.¹

To summarize, ideally a good test requires a prediction that meets these three conditions; it is

(1) deducible or at least probable, given that the claim is true,

(2) improbable, given that the claim is false, and

(3) verifiable.

Exercise \(\PageIndex{1}\)

Which of the three conditions above for a good scientific test are satisfied or violated, as far as you can tell, in the test mentioned in the following report?

A researcher claims that an invisible evil hand, a dark force, is at work disrupting transportation in the Bermuda Triangle. This triangle is the area of the Atlantic Ocean defined by Bermuda, Florida, and Cuba. The researcher predicted that if the hypothesis about an invisible hand were true, then there should be mysterious disappearances of planes and ships into thin air and definitely an unusual number of transportation accidents in this area. The researcher gathered data about many such cases, and then published his results claiming his testing confirmed the existence of the evil hand.

Answer: The major fault is with condition 2 (improbability). Condition 1 (deducibility) is satisfied; the researcher correctly deduced that if there were an evil hand, it would perform evil deeds. Condition 3 (verifiability) is satisfied because it is not too hard to check on whether planes and boats actually have had accidents or disappeared, and this checking doesn't depend on any assumption about whether there is or isn't an evil hand at work. Condition 2, on the other hand, hasn't been established, as far as we can tell from the information given. Condition 2 (improbability) requires that these disappearances be shown to be improbable, yet for all we know, given the traffic flow, there is no higher percentage of disappearances in the Bermuda Triangle than in any other part of the world. (In fact, there have not been an unusual number of disappearances, given that this area is one of the busiest of any ocean.)

1 These criteria for a good test are well described by Ronald Giere in Understanding Scientific Reasoning (New York: Holt, Rinehart and Winston, 1979), pp. 101-105.