Let’s face it, we all read forecasts. Forecasts of the economy, political developments, social trends, etc in the media capture our attention. These forecasts are usually prepared by experts who are very knowledgeable in their field of study and we therefore give the forecasts credence. Have you ever wondered how well forecasters are at their profession?
From what I have read, the most comprehensive study of forecasting accuracy was performed by Philip Tetlock, a professor of business and political science at the University of California at Berkeley. From 1984 to 2003, Tetlock recorded and analyzed over 82,000 forecasts from 284 political and economic forecasters. Tetlock has written a book on his findings: Expert Political Judgment: How Good Is It? How Can We Know? Tetlock’s analysis showed that experts barely, if at all, produced forecasts which were better than non-experts.
Adrian E. Tschoegl and Scott Armstrong of the University of Pennsylvania have written a short review of Tetlock’s findings. Quoting from their review:
In analyzing his respondents’ methods of arriving at their forecasts, Tetlock found it insightful to employ the metaphor that Isaiah Berlin (1953) took from the Greek poet Archilochus and used in his essay on Tolstoy: “The fox knows many things, but the hedgehog knows one big thing.” Foxes draw on many ideas and sources of information; hedgehogs interpret the world using their favorite theory or dogma. Foxes are more tolerant of ambiguity and uncertainty than hedgehogs, who tend to be confident in the rightness of their view of the world.
Tetlock found that foxes usually outperform hedgehogs; foxes produced more accurate forecasts than
hedgehogs, though again neither beat simple rules. However, Tetlock reports that he was able to push
more foxes than hedgehogs into forecasts that violated a fundamental axiom of probability: that the sum of a forecaster’s forecast probabilities not exceed one.
Tetlock argues that there is an inverse relationship between what works best in forecasting and what works best in the media. The fox may make the more accurate forecasts, but it is the dramatic, single-minded, combative hedgehog ideologue who makes the best TV, especially when matched against an opposite number in a point-counterpoint debate. A debate between foxes will dissolve into a miasma of nuance. It is not surprising then that Batchelor (2007) finds that “Private sector forecasters also have incentives to bias their forecasts towards optimism or pessimism,” so as to stand out from the crowd, though we believe this is not the key explanation for the existence of hedgehogs.
A more comprehensive review by Louis Menand can be found at the New Yorker. Quoting from Menand’s article:
It is the somewhat gratifying lesson of Philip Tetlock’s new book, “Expert Political Judgment: How Good Is It? How Can We Know?” (Princeton; $35), that people who make prediction their business—people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables—are no better than the rest of us. When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons. They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake. No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote. Our system of expertise is completely inside out: it rewards bad judgments over good ones.
The next time you find yourself watching an expert on television making a forecast, ask yourself two questions: 1. Is there a selection bias in the process of determining which experts are interviewed? and 2. Why should I believe this expert’s forecast is better than my own?
{ 1 trackback }