A Troubled Tradition
It’s time to rebuild trust among authors, editors and peer reviewers
Alternative Forms of Peer Review
Some journals and conferences have adopted or tested alternative forms of peer review. One common alternative, double-blind review, could serve to reduce reviewer bias because neither authors nor reviewers know each other’s identities or affiliations. Another alternative is unblinded (or open) review, in which both authors and reviewers do know each other’s identities—a situation that might encourage ethical behavior among reviewers who cannot hide behind a cloak of anonymity.
Studies of these two forms of review have, however, yielded mixed results. Logistics are an important hurdle: In trials of double-blind review, several medical journals found that about one-quarter to one-half of reviewers were able to correctly guess authors’ identities despite blinding. And in trials of open review, referees who were asked to reveal their names to authors often refused to participate.
There is nevertheless evidence that blinding does reduce bias. Joseph Ross of the Robert Wood Johnson Clinical Scholars Program led a five-year study, published in 2006, which showed that authors’ nationality and institutional affiliation affected acceptance of abstracts for the American Heart Association’s annual Scientific Sessions. Among thousands of abstracts submitted per year, blinded reviewers accepted about 12 percent fewer abstracts from prestigious institutions than did reviewers who were aware of authors’ affiliations. Blinded reviewers also accepted fewer abstracts from within the United States and more from outside the United States than did unblinded reviewers. Blinding must have reduced bias resulting from reviewers’ assumptions about authors’ countries and institutions.
Whether blinding also improves the quality of reviews is unclear. In 1990, Robert McNutt and colleagues found that blinded reviewers provided more accurate, well-supported and courteous reviews than did unblinded reviewers of articles submitted to the Journal of General Internal Medicine. But several years later, Amy Justice led a similar study with five different medical journals, and found no effect of blinding on review quality.
Results for open review have been similarly mixed. One study at the British Journal of Psychiatry, led by Elizabeth Walsh, found that when referees revealed their identities to authors, they provided better reviews, were more courteous, took longer to complete reviews, and were more likely to recommend publication than were anonymous reviewers. But a pair of studies led by Susan Van Rooyen of the British Medical Journal found that revealing reviewers’ identities—either to authors or to co-reviewers—did not impact review quality or reviewers’ recommendations.
The discrepancies among these studies of double-blind and open review could arise from their differing methodologies and sample populations. It is also worth mentioning that none of these studies examined the most serious ethical issues, such as respect for intellectual property. Future studies should take these factors into account, and they may eventually tip the balance in favor of one form of review or another.