18.3: Magic Numbers

We often judge outcomes and performances relative to some special number or target. Such numbers were originally used to measure something else, but sometimes they acquire a life of their own. They take on a significance that is disproportionately large compared to their actual value. Such variables are sometimes called magic numbers.

In the case of the economy, the cost-of-living index or the GNP may be treated as the measure of economic strength. When evaluating a business with an eye to buying or selling, stock price/earning ratios are sometimes taken as the key. F. DeGeorge and co-workers found three variables of this sort in businesses: positive profit, previous year’s earnings, and the agreement of financial analysts’ earnings estimates. Closer to home, for many of you, ACT or SAT scores and GPA easily become magic numbers for measuring academic promise and success. Furthermore, we often set a somewhat arbitrary threshold for this number, with anything below the threshold counting as failure and anything above it counting as success. In public policy, business and other arenas where people want to measure progress or success, precise target numbers, quotas, acceptable level of risk, grade point average, and the like often come to define success and failure.

The Local Charity will count it a failure unless they raise $35,000, the new police chief fails unless she reduces crime by the targeted 2.5%.

Lawyers often speak of certain thresholds as bright lines. These are clear boundaries that make it easier to set policies people can understand. You must be over a very clearly defined age to buy beer; a precise level of alcohol in a driver’s bloodstream counts as driving under the influence. Bright lines are easy to judge (you aren’t 21 until midnight tomorrow), and they create clear expectations. We can generalize this notion to apply to threshold and boundaries in other settings; in the charity example, raising $35,000 is a bright target that tells us whether our fund-raising efforts were a success, or a failure. Target numbers are often ones that are salient and easy to remember; the local charity set a goal this year of raising $35,000, not $33,776. Furthermore, the numbers aren’t completely arbitrary; no one would think raising $3.50 was a worthwhile target. Still, within some vague range of sensible targets, the selection of a specific number typically is arbitrary.

Like many of the other points we have studied (e.g., heuristics), magic numbers and bright targets help us simplify very complex situations involving things like the economy and the environment. And frequently the numbers do tell us something about how the economy or a firm or a school is doing. Bright targets can also provide good motivation; if I set a goal of losing twenty pounds over the summer, it gives me something definite to shoot for.

In policy settings, bright lines are sometimes useful because they provide a line that isn’t renegotiated by each person on every occasion. In policy matters this is useful, because there is often a very real risk that the people making the decisions will do so in a way that isn’t fair, or at least that won’t appear fair. If the judge can simply decide if someone drank too much before getting behind the wheel, there would be a great deal of potential for abuse. Having a definite cutoff percentage of blood-alcohol level makes it more likely that everyone will be treated the same, and expectations are clear to everyone.

Problems with Magic Numbers

Although magic numbers and bright targets are frequently useful, even unavoidable, they often take on a life of their own.

Originally, they were a means to an end, e.g., an aid to seeing if the economy is improving, but eventually they become ends in themselves. This can lead to several problems.

1. Simply getting across the threshold is often seen as the measure of success, even when progress on either side of the line (getting closer to the goal by this much, exceeding it by that much) is equally important. Often the difference between almost making it to the target, on the one hand, and exceeding it by just a little, on the other, is insignificant. So, a bright target can promote all-or-none thinking.
2. Target numbers can also be treated as the only relevant measures of success and failure, even though the precise target numbers are somewhat arbitrary and other, perhaps more important variables, are ignored.
3. In the worst case, magic numbers represent variables that are not very important, or their specific target values are not set in any sensible way.

Magic Numbers and Suboptimal Policies

Magic numbers are often introduced in an honest effort to assess performance or progress, but they can have unintended consequences, including counterproductive policies and behavior.

For example, policy makers sometimes establish bright lines when dealing with environmental issues. For example, suppose that an agency sets a precise target for what counts as an acceptable level of arsenic in your community’s drinking water. Clearly the arsenic level come in degrees, with less arsenic being better, whatever the target value is. But hard and definite numbers can foster a feeling that either risk is present or else it’s been eliminated, when it’s more accurate to think in terms of more and less risk, rather than risk or no risk.

It is often felt that anything short of a designated target is failure, and anything over it, even if it only limps over the line by just a bit, is a success. F. DeGeorge and co-workers found that people in charge of large businesses would often manipulate their earnings to get beyond a target value. Often it didn’t matter how far from the target they ended up, provided they passed it. Worse, it often didn’t matter if the way of getting to the target would lead to problems in the longer run; for example, they would sometimes sell items at a large discount (or even a loss) late in the year, just to meet a target for yearly earnings.

C. Camerer and his colleagues found that New York City cabdrivers would set a target income for a day’s work. They would drive until they reached that target, then knock off for the day. They would make more money with less driving if they drove fewer hours on days when business was slow and more hours on days when it was brisk. But the daily-earnings target seemed to be a bright threshold that took on an intrinsic importance.

As a third example of how magic numbers can lead to less than optimal policies, consider recent programs to make schools more accountable. The basic idea is a good one; schools have a very important obligation to their students and to those who pay the bills (in many cases taxpayers like us) to do a good job. Many recent efforts to judge how well schools are doing rely on standardized tests that are administered every few years.

In some cases, this has led teachers to spend a great deal of time teaching the students things that will help them do well on such tests, while shortchanging other things. To the extent that the tests measure the things students should be learning, this might be acceptable, but it is by no means clear that the tests do that. Indeed, we will see in the final chapter that if one goal is to foster critical reasoning, then training people to do well on standardized tests is not the best thing to focus on.

Magic Numbers Can Become Ends in Themselves

Instead of being treated as indicators or indices of how well (or badly) things are going, magic numbers often displace what we originally cared about and become ends in themselves. We slip from thinking that GPA tells us something about how much a student is learning to thinking that GPA really is how well a student is doing. Furthermore, such numbers can distort our picture of the situation. For example, Wilbur may have a higher GPA than Wilma because Wilber’s courses are much easier. There is also a political dimension to the selection of many magic numbers and targets. Someone in charge of a business or governmental agency will find it tempting to propose those measures of success that will show that they are in fact succeeding, while those who are unhappy with the way things are going may propose rather different magic variables to measure performance.

The allure of bright lines and magic numbers is one of several things that can lead us to focus on factors that are easy to measure or quantify, while paying less attention to things that are harder to measure, even when they are more important. This in turn can lead to the view that what can’t be quantified or measured is unimportant, or even not real (we will return to this in Chapter 20). Magic numbers and bright lines are often useful and even unavoidable, but they can lead to genuine problems. The key questions to ask when you hear some number cited as a sign of success or failure are:

1. Is it really a good measure, an accurate indicator, of the thing that it’s supposed to be measuring, and
2. Does the specific target number associated with it really have any special significance?

Exercises

1. One of the middle schools in your city has been doing very badly by almost everyone’s standards; students, parents, teachers, and neutral observers are all upset. Last week the school district allocated almost two million dollars to improve the school, and you are put in charge of a task force to assess how much (if any) the school really does improve. You will consult experts, but the experts you consult may disagree, and then your task force will have to draw a conclusion of its own. Think about the ways you would try to assess whether the tax dollars really translated into helping the students. Focus on some of the reasons why this would be difficult. How might people with different vested interests propose different measures of success? What might they be?
2. Management and workers at a large grocery chain are locked in a bitter negotiation over salary increases. What sorts of measures of how well the business is doing and how well the workers are doing might the workers focus on? What sorts of measures of how well the business is doing and how well the workers are doing might management focus on?
3. You are appointed as a student representative to a group asked to ascertain whether classes that feature a lot of group work provide a better educational experience than classes that don’t. Your committee will consult experts, but the experts you consult may disagree, and then the committee will have to reach a conclusion of its own. How could you assess whether, on average, classes with a lot of group work did a better job of teaching people the things they should be learning than classes without group work do. Focus on the reasons why this would probably be difficult. How might people with different vested interests (e.g., those who spent a lot of time developing group projects vs. those who spent a lot of time developing lectures and individual projects) propose different measures of success? (We will return to some of the issues about group learning in Chapter 24.)
4. Give a real-life example of a bright line. Why is it supposed to be important? Who decided what it would be? How arbitrary do you think it is?
5. What connections might there be between magic numbers, on the one hand, and issues involving framing and psychological accounting, on the other?
6. Think of some variable that is important but hard to quantify. How might it be overlooked if we focus too much on how it can be measured?
7. In arguing for the need for a bright line for what counts are a reduced risk pesticide, the Consumers Union and its Consumer Policy Institute (CPI), a nonprofit product testing organization in Yonkers, New York, argued:

The purpose of these bright lines is to limit the pool of candidates for reduced risk status, and to provide EPA [Environmental Protection Agency] some easily applied criteria for saying “no” to requests for reduced risk status. Bright lines are also needed to reduce the time and agency resources required to evaluate requests for consideration for reduced risk status. If EPA opens itself up to a significant number of such requests, the time and resources entailed in reviewing and deciding upon such requests will end up diverting significant agency resources, and would hence defeat the purpose of the policy. Restate the basic points in clear, non-technical English in a more general way that could apply to other issues in addition to pesticide labeling. How strong are such arguments for having bright lines in cases like this? What might be said in defense of the other side