The Survival of the Fittists
Understanding the role of replication in research is crucial for the interpretation of scientific advances
What Have We Learned?
Science is designed to be self-correcting. Attempts to replicate provide evidence of when it has gone astray. Or at least that’s the theory. The real world, filled with fallible people and institutions, practically guarantees that the path toward progress meanders, sometimes massively. But this is not all bad. As physicist David Deutsch has emphasized, the evolution of a scientific idea is different than the Darwinian evolution of an organism in at least two important ways. First, ideas evolve in ways that are directed by the intelligence of the investigators. In contrast, biological evolution has no goal other than maximizing the likelihood that a particular mixture of genes will spread. Second, if a particular phenotype emerges that cannot survive, it becomes an evolutionary dead end, but an idea that is a failure can still have parts that can be retrieved and used subsequently.
The story told here provides some compelling examples, if any were needed, that the road to improvement is fraught with potholes of misinformation and twists of political intrigue. But as long as we maintain a healthy skepticism and remain free to publicly question the status quo, we will continue to advance—at least for those disciplines for which the direction of an advance is known. In a 1966 American Scientist article, Princeton University psychologist Julian Jaynes offered an evocative metaphor to delineate what he saw as the differences between psychology and physics. I would apply Jaynes’s words more broadly to the differences between the hard sciences and the humanities—and, to some extent, the social sciences:
Physics is like climbing a mountain: roped together by a common asceticism of mathematical method, the upward direction, through blizzard, mist, or searing sun, is always certain, though the paths are not. . . . The disorder is on the ledges, never in the direction. . . .
[Psychology] is less like a mountain than a huge entangled forest in full shining summer, so easy to walk through on certain levels, that anyone can and everyone does. The student’s problem is a frantic one: he must shift for himself. It is directions he is looking for, not height. . . . Multitudes cross each other’s paths in opposite directions with generous confidence and happy chaos. The bright past and the dark present ring with diverging cries and discrepant echoes of “here is the way!” from one vale to another.
The pitons and cleats so critical for ascending a mountain, Jaynes continues, are replaced with blinders and earplugs as people wander the forest. This passage may help explain why the scientific method, so powerful in the hard sciences, fails when applied to subjects in which there is no broad consensus of what constitutes an advance.
In the world we inhabit, the rules of science interact with the foibles of scientists. What we see should not be taken at face value—even “objective science.” Every scientific study carries along with its results some sense of its own credibility. Studies with larger sample sizes are more credible than those that are smaller, ceteris paribus. If those doing the study have a great deal riding on the result, credibility suffers. We must always be vigilant. But the current, flawed system, in which independent studies are used to test results obtained by someone else, is the best available. As flaws are detected, we can institute reforms. Such reforms should always move in the direction of greater openness and greater accessibility to the raw data from which the conclusions are drawn.
The success of the scientific method relies on the continued existence and prosperity of researchers who relentlessly fit experimental data to theory. The validity of science depends on the survival of the fittists.
I am thankful to David Donoho who, over dinner one evening, told me about the results of a meta-analysis of Chinese medical research, thus instigating the writing of this essay.
- Deutsch, D. 2011. The Beginning of Infinity: Explanations that Transform the World. New York: Viking.
- Grégoire, G., F. Derderian and J. Le Lorier. 1995. Selecting the language of the publications included in a meta-analysis: Is there a Tower of Babel bias? Journal of Clinical Epidemiology 48:159–163.
- Gulliksen, H. O. 1938. Extrasensory perception: What is it? American Journal of Sociology 43:623–634.
- Ioannidis, J. P. 2005. Why most published research findings are false. PLoS Medicine 2:e124.
- Jaynes, J. 1966. The routes of science. American Scientist 54:94–102.
- Kahneman, D. 2011. Thinking, Fast and Slow. New York: Farrar, Straus & Giroux.
- Pan, Z., et al. 2010. Local literature bias in genetic epidemiology: An empirical evaluation of the Chinese literature. PLoS Medicine 7:e334.
- Rhine, J. B. 1934. Extra-Sensory Perception. Boston, MA: Bruce Humphries.