by Judith Curry
Robert Ellison sent me a link to a review of a book entitled “Future Babble: Why Expert Predictions Fail and Why We Believe Them“, by Dan Gardner, which describes the research of UC Berkeley Professor Philip Tetlock.
A good summary is provided by a reviewer at Amazon.com:
This is an extremely readable and thought provoking book. Gardner’s exhaustive research builds an extremely persuasive case for the book’s sub-title. He also explains why we keep coming back for more useless forecasting babble. Although some of his examples could be more succinctly summarized, most are very entertaining and enlightening. Gardner illustrates the book’s core message around the dismal failure of expert predictions with examples of both overly rosy predictions and darkly apocalyptic forecasts missing the mark by miles. He’s especially effective at pillorying the many bestselling prophets of doom. These include the authors of such pessimistically dire works as, The Population Bomb, How to Prosper in the Coming Bad Years, The Limits to Growth, The End of Affluence, An Inquiry into the Human Prospect, and Blood in the Streets.
Future Babble cites numerous studies showing the repeated and colossal failure of expert predictions in every field (except for short term weather forecasts). He quotes Scott Armstrong “an expert on forecasting at the Wharton School of the University of Pennsylvania” on his “seer-sucker theory: No matter how much evidence exists that seers do not exist, suckers will pay for the existence of seers.” Here’s another of Gardner’s examples: “The now-defunct magazine Brill’s Content, for one, compared the predictions of famous American pundits with a chimpanzee named Chippy, who made guesses by choosing among flashcards. Chippy consistently matched or beat the best in the business.”
Future Babble draws heavily on the comprehensive research of Philip Tetlock, professor of psychology, business, and political science at University of California Berkeley. His authoritative study encompassed 284 experts “giving 27,450 judgments of the future.” Tetlock concluded that the experts would have been beaten by “a dart-throwing chimpanzee.” Gardner observes that “the simple and disturbing truth is that the experts’ predictions were no more accurate than random guesses.” An especially interesting finding in these days of media sound bites, blogging, and viral videos is Tetlock’s use of Google hits to determine the fame of each of the 284 experts. His findings: “the more famous the expert, the worse he did.”
What caught my interest about this is the discussion of foxes and hedgehogs.
Hedgehogs and Foxes
“The fox knows many things but the hedgehog knows one big thing.” – Archilochus
While in general predictions are usually no better than chance guesses, Tetlock’s research revealed that some people are better than others. Tetlock identified two types of predictors: Foxes and Hedgehogs.
Hedgehogs look for a single grand design and once they have found it, they look no further. They feel that they can predict with confidence, and they do.
Foxes, on the other hand, are sceptical of grand theories. They continually incorporate new information into their understanding, and as a result, are less confident about their predictions.
Tetlock found that Foxes, although much less confident of their predictions, were more likely to be correct in their predictions.
Prediction is Hard
“It’s tough to make predictions, especially about the future.” – Yogi Berra
Gardner points out that one reason that pundit’s prediction are wrong, is that prediction is often hard to do. In fields like economics and politics, forecasters face very complex situations, where random events can radically change the outcome.
Other barriers to successful prediction lie within us. As Gardner observes: The mind is an impressive organ of thought, but it has its limitations. We are handicapped by cognitive biases, logical fallacies and other fallacies that undermine our ability to see clearly. Pundits who predict are just as vulnerable to these as anyone else.
An example the author discusses is Confirmation Bias , where, once a person has made a choice, they only accept evidence that supports their decision and reject evidence that does not. When evidence that contradicts a prediction is found, a typical pundit will find an excuse to disregard it.
Why We Believe Pundits, Even When They are Wrong
“He is often wrong but he’s never in doubt” – Norman Lamont, a British politician, on why he trusted a particular pundit.
Why, if pundit’s predictions are often so wrong, do we still turn to them for advice?
Gardner points out that the future is uncertain, and few people are comfortable with uncertainty. People are risk adverse and fear uncertainty. When someone comes along and promises that can put an end to uncertainty, most people will jump at the chance. Certainly, in my own experience, people are far more willing to take the advice of some one who sounds confident than someone who seems unsure. This is despite the fact that the less confident person is more likely to be right.
People are vulnerable to the same cognitive biases, logical fallacies and other fallacies that bedevil the pundits and predictors. These flaws in thinking undermine our ability to evaluate the predictions of pundits.
The Answer: A Better Way
“When the facts change, I change my mind. What do you do, sir?” – John Maynard Keynes
Toward the end of his book, Gardner considers how we can get better predictions. He makes four suggestions.
- Accept that the world is complex and uncertain.
- Look at a wide variety of information and combine that information to gain deeper understanding.
- Think about thinking: be aware of the biases and fallacies of thought.
- Strive for humility.
Gardner goes on to suggest that we should look for courses of action that are good, what ever happens. Although this is easier said than done, I do believe this is the wisest approach to take. What I would add, is we should continue to seek out new information and revisit our decisions in light of what we have learned.
A final bit of advice, be doubtful of people who are confident of the correctness of their own predictions.
Louis Menard also reviewed this book in the New Yorker, some excerpts that caught my eye:
The expert also suffers from knowing too much: the more facts an expert has, the more information is available to be enlisted in support of his or her pet theories, and the more chains of causation he or she can find beguiling. This helps explain why specialists fail to outguess non-specialists. The odds tend to be with the obvious.
Tetlock’s experts were also no different from the rest of us when it came to learning from their mistakes. Most people tend to dismiss new information that doesn’t fit with what they already believe. Tetlock found that his experts used a double standard: they were much tougher in assessing the validity of information that undercut their theory than they were in crediting information that supported it. The same deficiency leads liberals to read only The Nation and conservatives to read only National Review. We are not natural falsificationists: we would rather find more reasons for believing what we already believe than look for reasons that we might be wrong. In the terms of Karl Popper’s famous example, to verify our intuition that all swans are white we look for lots more white swans, when what we should really be looking for is one black swan.
Also, people tend to see the future as indeterminate and the past as inevitable. If you look backward, the dots that lead up to Hitler or the fall of the Soviet Union or the attacks on September 11th all connect. If you look forward, it’s just a random scatter of dots, many potential chains of causation leading to many possible outcomes. We have no idea today how tomorrow’s invasion of a foreign land is going to go; after the invasion, we can actually persuade ourselves that we knew all along. The result seems inevitable, and therefore predictable. Tetlock found that, consistent with this asymmetry, experts routinely misremembered the degree of probability they had assigned to an event after it came to pass. They claimed to have predicted what happened with a higher degree of certainty than, according to the record, they really did. When this was pointed out to them, by Tetlock’s researchers, they sometimes became defensive.
And, like most of us, experts violate a fundamental rule of probabilities by tending to find scenarios with more variables more likely. If a prediction needs two independent things to happen in order for it to be true, its probability is the product of the probability of each of the things it depends on. If there is a one-in-three chance of x and a one-in-four chance of y, the probability of both x and y occurring is one in twelve. But we often feel instinctively that if the two events “fit together” in some scenario the chance of both is greater, not less. The classic “Linda problem” is an analogous case. In this experiment, subjects are told, “Linda is thirty-one years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice and also participated in antinuclear demonstrations.” They are then asked to rank the probability of several possible descriptions of Linda today. Two of them are “bank teller” and “bank teller and active in the feminist movement.” People rank the second description higher than the first, even though, logically, its likelihood is smaller, because it requires two things to be true—that Linda is a bank teller and that Linda is an active feminist—rather than one.
Plausible detail makes us believers. When subjects were given a choice between an insurance policy that covered hospitalization for any reason and a policy that covered hospitalization for all accidents and diseases, they were willing to pay a higher premium for the second policy, because the added detail gave them a more vivid picture of the circumstances in which it might be needed. In 1982, an experiment was done with professional forecasters and planners. One group was asked to assess the probability of “a complete suspension of diplomatic relations between the U.S. and the Soviet Union, sometime in 1983,” and another group was asked to assess the probability of “a Russian invasion of Poland, and a complete suspension of diplomatic relations between the U.S. and the Soviet Union, sometime in 1983.” The experts judged the second scenario more likely than the first, even though it required two separate events to occur. They were seduced by the detail.
With more insights on hedgehogs and foxes:
Tetlock uses Isaiah Berlin’s metaphor from Archilochus, from his essay on Tolstoy, “The Hedgehog and the Fox,” to illustrate the difference. He says:
Low scorers look like hedgehogs: thinkers who “know one big thing,” aggressively extend the explanatory reach of that one big thing into new domains, display bristly impatience with those who “do not get it,” and express considerable confidence that they are already pretty proficient forecasters, at least in the long term. High scorers look like foxes: thinkers who know many small things (tricks of their trade), are skeptical of grand schemes, see explanation and prediction not as deductive exercises but rather as exercises in flexible “ad hocery” that require stitching together diverse sources of information, and are rather diffident about their own forecasting prowess.
A hedgehog is a person who sees international affairs to be ultimately determined by a single bottom-line force: balance-of-power considerations, or the clash of civilizations, or globalization and the spread of free markets. A hedgehog is the kind of person who holds a great-man theory of history, according to which the Cold War does not end if there is no Ronald Reagan. Or he or she might adhere to the “actor-dispensability thesis,” according to which Soviet Communism was doomed no matter what. Whatever it is, the big idea, and that idea alone, dictates the probable outcome of events. For the hedgehog, therefore, predictions that fail are only “off on timing,” or are “almost right,” derailed by an unforeseeable accident. There are always little swerves in the short run, but the long run irons them out.
Foxes, on the other hand, don’t see a single determining explanation in history. They tend, Tetlock says, “to see the world as a shifting mixture of self-fulfilling and self-negating prophecies: self-fulfilling ones in which success breeds success, and failure, failure but only up to a point, and then self-negating prophecies kick in as people recognize that things have gone too far.”
Further reflections on this are provided by T. Greer: