Logo IMG
HOME > PAST ISSUE > Article Detail


Freakonomics: What Went Wrong?

Examination of a very popular popular-statistics series reveals avoidable errors

Andrew Gelman, Kaiser Fung

Problems—and Solutions

2012-01MacroGelmanFB.jpgClick to Enlarge ImageHow could an experienced journalist and a widely respected researcher slip up in so many ways? Some possible answers to this question offer insights for the would-be pop-statistics writer.

Leave friendship at the door: We attribute many of these errors to the structure of the authors’ collaboration, which, from what we can tell, relies on an informal social network that has many potential failure points. In the original Freakonomics, much of whose content appeared originally in columns for the New York Times Magazine, the network seems to have been more straightforward: Levitt did the research, Dubner trusted Levitt, the Times trusted Dubner, and we the readers trusted the Times’s endorsement. In SuperFreakonomics and the authors’ blog, it becomes less clear: Levitt trusts brilliant stars such as Myhrvold or Oster, Dubner trusts Levitt, and we the readers trust the Freakonomics brand. A more ideal process for science writing (as shown in the illustration above) will likely look much messier—but it offers the promise of better results.

Don’t sell yourself short: Perhaps Levitt’s admirable modesty—he has repeatedly attributed his success to luck and hard work rather than genius—has led him astray. If he feels he is surrounded by economists more exceptional and brilliant than he is, he may let their assertions stand without challenge. Here it might be good to remember the outsider’s perspective so prized by Levitt: If you find yourself hesitant to ask questions that seem “stupid,” or if you feel intimidated, think of yourself as a “rogue.” Just don’t take it so far that you value your own rogueness over empirical evidence.

Maintain checks and balances: A solid collaboration requires each side to check and balance the other side. Although there’s no way we can be sure, perhaps, in some of the cases described above, there was a breakdown in the division of labor when it came to investigating technical points. The most controversial statements are the most likely to be mistaken; if such assertions go unchallenged, you will have little more than a series of press releases linked by gung-ho commentary and eye-popping headlines. Hiring a meticulous editor who can evaluate the technical arguments is another way to avoid embarrassing mistakes.

Take your time: Success comes at a cost: The constraints of producing continuous content for a blog or website and meeting publisher’s deadlines may have adverse effects on accuracy. The strongest parts of the original Freakonomics book revolved around Levitt’s own peer-reviewed research. In contrast, the Freakonomics blog features the work of Levitt’s friends, and SuperFreakonomics relies heavily on anecdotes, gee-whiz technology reporting and work by Levitt’s friends and colleagues. Just like good science, good writing takes time. Remembering this can help hedge against the temptation to streamline arguments or narrow the pool of sources, even in the face of deadlines.

Be clear about where you’re coming from: Levitt’s publishers, along with Dubner, characterize him as a “rogue economist.” We find this odd: He received his Ph.D. from the Massachusetts Institue of Technology, holds the title of Alvin H. Baum Professor of Economics at the University of Chicago and has served as editor of the mainstream Journal of Political Economy. He is a research fellow with the American Bar Foundation and a member of the Harvard Society of Fellows, and has worked as a consultant for Corporate Decisions, Inc. One can be an outsider within such institutions, of course. But much of his economics is mainstream. And his statistical methods are conventional (which, we hasten to add, is not a bad thing at all!). One of the pleasures of reading Freakonomics is Levitt’s knack for finding interesting quantitative questions in obscure corners, such as the traveling bagel salesman and cheating sumo wrestlers. Often such problems have not been extensively studied or even been noticed by others, and in these cases one is hard-pressed to identify any consensus or conventional wisdom. Often, in the authors’ writing, the “conventional” and the “rogue” live side by side. Chapter one of SuperFreakonomics, for instance, can be viewed either as a clear-eyed quantitative examination of the economics of prostitution, or as an unquestioning acceptance of conventional wisdom about gender roles. In exploring new territory, it’s especially important to be plainspoken about where your assumptions come from and what your primary ideas are.

Use latitude responsibly: When a statistician criticizes a claim on technical grounds, he or she is declaring not that the original finding is wrong but that it has not been convincingly proven. Researchers—even economists endorsed by Steven Levitt—can make mistakes. It may be okay to overlook the occasional mistake in the pursuit of the larger goal of understanding the world. But once one accepts this lower standard—science as plausible stories or data-supported reasoning, rather than the more carefully tested demonstrations that are characteristic of Levitt’s peer-reviewed research articles—one really has to take extra care, consider all sides of an issue, and look out for false positive results.

The landscape of pop-statistics books grows more varied by the year, and Levitt and Dubner’s bestsellers have introduced several new ingredients to the genre. One of the delights of the books and the blog is the authors’ willingness to play with ideas and consider alternative explanations. But unquestioning trust in friends and colleagues combined with the desire to be counterintuitive appear in several cases to have undermined their work. They—and anyone who wishes to convey economics and statistics to a popular audience—just need to take the next step and avoid, in any given example, privileging one story over all other possibilities. This may require Levitt to be more skeptical of the research of his friends and colleagues, and Dubner to be more skeptical of Levitt. “Easy read” should not mean “easy write.”

And it doesn’t even always mean “easy read”: Readers should apply the same skepticism to the claims of Freakonomics as they would to the much-derided conventional wisdom. We encourage them to revisit these modern-day classics with a skeptical and inquiring mind. And we hope that future works in the pop-statistics genre will continue to impart a sense of the fun and importance of statistical reasoning, while more clearly recognizing the uncertainty and complexity inherent in scientific study of the world.


comments powered by Disqus


Subscribe to American Scientist