3.5: Improper Operationalization
- Page ID
- 36056
Suppose a newspaper headline reported “Number of Christians in the World Increases.” The article does not tell you how the researchers decided whom to count as a Christian. You have to guess. Did the researchers mail a questionnaire to all the Christian churches in the world asking for membership totals? This way of counting would miss everyone who considers himself or herself a Christian but who isn't affiliated with a specific church, and it would over-count those persons who belong to more than one church. Did the researchers instead do a small poll of New Orleans and then assume that the rest of the world is like New Orleans as far as religion is concerned? There are many other operations they could have used to count Christians. The operation or method the researchers used could make a difference to their count. That is, the operationalization of the term Christian would make a difference in what the study says is the total number of Christians.
The operationalization of a term is the operation or method used to tell whether the term applies.
When you question the operationalization of a term, you are essentially asking, "How did they measure that?” The word Christian and other imprecise terms need good operationalizations if accurate claims are going to be made using them. Without a good operationalization, a claim using the term will suffer from inaccuracy, pseudo- precision, or both.
What operationalization problem occurs here?
Lying by politicians is up 10 percent in New York this year.
- Answer
-
The main problem is how to measure lies. One sub-problem is that liars will cover up their lying. A second sub-problem is that even if you can uncover the lies, you must still find a way to count them because there is a fuzzy border between lying and stretching the truth.
Does South America have more political instability than Central America? It is more difficult to establish claims that use imprecise terms such as political instability than it is to establish claims that use precise terms such as voltage or mile. That's one reason why social scientists have a tougher job than physical scientists in making scientific advances.
Let's consider the special problem of the social scientist. Suppose the newspaper reports on a scientist's finding that rich females are more intelligent on average than poor males. The logical reasoner will ask, "How did the researcher figure out who is rich and who is intelligent?" Let's focus on the term rich. If the researcher merely asked each person whether he or she was rich, the answers are going to be very unreliable. But perhaps the researcher operationalized the concept of rich in some other way. Did she check each individual's finances and decide that anybody making over $100,000 per year was rich and anybody under that was poor? Even one cent under? If so, then the researcher does not mean what you and I do by the term rich. Unfortunately, most of us get our information indirectly and not by reading scientific reports that specify the operationalization procedures of the key terms under study. Instead, we have to hope that the other scientists who refereed the report checked on the procedures and approved them.
Operationalizing the vague term intelligent is as difficult as the term rich. And some ways are better than others. Which among the following is the best?
a. Intelligent people are smarter than unintelligent people.
b. See whether the word intelligent has positive connotations.
c. Use an IQ test.
d. Ask the people involved whether they are intelligent.
e. Ask a large random sample of people whether they believe rich females are more intelligent than poor males.
Only (c) and (d) are operationalizations, so (a), (b), and (e) can be ruled out. Method (e) is a way to answer the question of whether rich women are more intelligent, but it's not a way to measure intelligence. Method (d) can be ruled out because of the subjectivity; people can be expected not to give honest assessments of themselves. That leaves the IQ test as the best of the lot.
Typically, an IQ test measures the ability to answer written questions. So, a typical IQ test is not an ideal measure of human intelligence. For example, if a person were to do poorly on an IQ test yet have been the primary inventor of an automobile engine that is 20 percent more energy efficient than all other existing automobile engines, that person should still be called "intelligent." In short, intelligence can show itself in many ways that aren't measured on an IQ test. But the other way works; if you do well on an IQ test you’ve got to be intelligent.
When you do read a headline such as "Biologists Report That Dolphins Are More Intelligent than Polar Bears" or "Political Scientists Report That South America Is More Stable Politically Than Central America," you probably should give the reporter the benefit of the doubt that the scientists used some sort of decent operationalization for the key terms. If you can actually discover what operationalization was used, and you have no problem with it, you can believe the scientists' report more firmly. And if you learn that the scientists published their results in a mainstream scientific journal and not in the Proceedings of the Conference on "Star Trek" Film Reruns, you can be even more confident in the report. It is safe to assume that the editors of the scientific journal checked to see that there were no significant problems with the operationalization of the key terms, nor any other problems with the scientific study. That wouldn't be as safe an assumption to make about the editor of the Proceedings of the Conference on "Star Trek" Film Reruns, nor even a safe assumption to make about editors of daily newspapers or TV news programs. And especially not for writers of individual blogs on the Internet.
When a Gallup poll reports that Americans are 5 percent happier with America now that the U.S. has completed negotiations on an international fishing treaty designed to say who can fish for which fish, critical thinkers know that the most difficult term for the pollster to operationalize was
a. reports
b. that
c. 5 percent
d. poll
e. happier
- Answer
-
Answer (e). How do you quantify (put a number on) happiness? Notice that happier is the evaluative term. The term 5 percent is not hard to operationalize. What is hard to operationalize is "5 percent happier," but that is not one of the answer choices.
Here is a more difficult question that covers several of the topics in this chapter at once.
"Social scientists from the University of Michigan report that over a quarter of all Canadian citizens they tested who had driven into Detroit from Canada on a Saturday night had blood alcohol levels of.05 percent or greater." This sentence probably
a. contains pseudoprecision.
b. is exaggerating.
c. needs operationalization of the evaluative term "Saturday night."
d. is relying too heavily on loaded language.
e. none of the above.
- Answer
-
Answer (e). The .05 percent level can be accurately measured, so answers (a) and (b) are not correct. The word Saturday is not evaluative. There is no loaded language here. Thus, by elimination, the answer is (e).