15.2.2: Reliability of Scientific Reporting

Almost every piece of scientific knowledge we have, we justify on the authority of what some scientist has said or is reported to have said. Because scientists are authorities on science, we usually take their word for things scientific. But chemists are not authorities on geology, and chemists who are experts in inorganic chemistry usually are not authorities on organic chemistry. Thus, when we are told that something is so because scientists believe it to be so, we should try to determine whether the proper authorities are being appealed to. Also, we know that scientists disagree on some issues but not on others, and we know that sometimes only the experts know which issues the experts disagree about. Is the reporter reporting the view of just one scientist, unaware that other scientists disagree? Scientists have the same moral failings as the rest of us, so we should also worry about whether a scientist might be biased on some issue or other. If a newspaper reporter tells us that the scientist's research on cloth diapers versus disposable diapers was not financed by the manufacturer of either diaper, we can place more confidence in the report.

Scientific journals are under greater pressure than daily newspapers to report the truth. A scientific journal will lose its reputation and its readers faster when there is a slipup than will the daily newspaper. So the stakes in reporting the truth are higher for journals. That is one reason the editors of scientific journals demand that authors provide such good evidence in their articles. If we read a report of a scientific result in a mainstream scientific journal, we can assume that the journal editor and the reviewers demanded good evidence. But if we read the report in a less reputable source, we have to worry that sloppy operational definitions, careless data collection, inaccurate instruments, or misunderstandings by the reporter may have colored the result.

When the stakes are high and we are asked to take an authority's word for something, we want independent verification. That means doing something more than merely buying a second copy of the newspaper to check whether what our first copy says is true. In medicine, it means asking for a second opinion from a different doctor. When the doctor says he wants to cut off your leg, you want some other doctor who is independent of the first doctor to verify that your leg really needs to be amputated. The term independent rules out your going to a partner in the first doctor's practice.

Ordinarily, though, we can't be bothered to take such pains to find good evidence. When we nonscientists read in the newspaper that some scientist has discovered something or other, we don't have enough time to check out the details for ourselves; we barely have enough time to read the reporter's account, let alone read his or her sources. So, we have to absorb what we can. In doing so, though, we who are critical thinkers are not blank slates willing to accept anything told to us. We are sensitive enough to ask ourselves: Does the report sound silly? Are any scientists protesting the result? What is the source of the report? We know that a reputable scientific journal article about some topic is more reliable than a reporter's firsthand interview with the author; we trust the science reporters for the national news magazines over those for a small, daily newspaper; and we know that daily newspapers are more reliable than independent bloggers and grocery store tabloids. But except for this, we nonscientists have severe difficulties in discriminating among the sources of information.

Suppose you were to read the following passage in a magazine: "To ensure the safety of raw fish, it should be frozen for at least five days at minus 4 degrees Fahrenheit (-20°C). That temperature kills all relevant parasitic worms so far tested." Should you believe what you read? It depends. First, ask yourself, "Where was it published and who said it?" In fact, the passage appeared in Science News, a well-respected, popular scientific publication. The magazine in turn was reporting on an article in an authoritative scientific publication, the New England Journal of Medicine. The journal in turn attributed the comment to Peter M. Schantz of the Centers for Disease Control in Atlanta, Georgia, a well-respected U.S. federal research laboratory. The magazine merely reported that Schantz said this. If you learned all this about the source of the passage in Science News, then you should probably accept what is said and add it to your knowledge.

You should accept it, but to what degree? You should still have some doubts based on the following concerns. The magazine did not say whether any other scientists disagreed with what Schantz said or even whether Schantz made this comment speculatively rather than as the result of a systematic study of the question. The occurrence of the word tested in the quote would suggest the latter, but you can't be sure. Nevertheless, you can reasonably suppose that the comment by Schantz was backed up by good science or the magazine wouldn't have published it the way it did—that is, with no warning that the claims by Schantz were not well supported. So, you can give Schantz's claims a high degree of belief, but you could be surer of what Schantz said if you had gotten direct answers to your concerns. Hearing from another scientific expert that Schantz's claims about fish are correct should considerably increase your degree of belief in his claims.