A Troubled Tradition
It’s time to rebuild trust among authors, editors and peer reviewers
Peer review is a key feature of scientific publication, but it is susceptible to bias, inefficacy and other ethical transgressions. Alternative forms of review have produced only equivocal improvements in fairness and efficacy and have not been tested with respect to other problems. What are the next steps we should take to improve peer review?
First, researchers should receive more education on how to review scientific articles—a skill that is not typically emphasized during research training. Some scientists do show students and postdocs how to review papers, and some research institutions cover peer review in seminars and workshops on research ethics. These practices must become more widespread. In particular, investigators should teach their trainees how to evaluate articles for scientific merit and to follow ethical standards of peer review. Asking young scientists to help review papers is a good way to educate them, provided journal editors give their permission and the process remains confidential.
Journals should also develop and publicize instructions for new reviewers and policies for reviewers and editors, just as they have done for authors. Rules should address confidentiality, fairness, conflict of interest, respect for intellectual property, and professionalism.
Editors should carefully manage the peer review process to prevent or address problems and concerns. They should explicitly inform reviewers about their journals’ peer-review policies, remind reviewers to disclose conflicts of interest and return their reports on time, and delete any personal attacks from reviewers’ comments. If editors have evidence that a reviewer provided a poor review or abused the process, they should not invite that person to do other reviews. These ideals may be complicated when editors have difficulty finding experts willing to review a manuscript and when referees submit their reviews late, overlook errors, or disagree about the quality of a submission. Workshops and conferences on the subject could help editors to cope with these challenges.
Finally, scholars should conduct additional research on the ethics of peer review. Our exploratory study of the experiences of NIEHS researchers suggests that some problems are common, but the results should be confirmed in other settings. Future work should determine how often ethical problems occur and how they affect scientists’ attitudes and behaviors. Studies should also address the causes of unethical behavior in journal peer review and the effectiveness of alternatives, such as double-blind or open review, at preventing various types of transgressions.
There is certainly no perfect solution to the problem of quality control in the scientific record. Despite its flaws, the system adopted by the editors of Philosophical Transactions two and a half centuries ago seems to work as well as any method that has been tried—but its age and pervasiveness must not foster complacency. While journals, editors, and scholars work to understand and regulate peer review, it’s up to every individual scientist to maintain a thoughtful awareness of his or her participation in the process. Such vigilance and professionalism can only improve the quality of reviews, and might even spark new insights into how the review system could eventually be improved.
I am grateful to Bruce Androphy, Zubin Master, Christine Flowers and Adil Shamoo for helpful comments. This essay does not represent the views of the NIH, the NIEHS or the U.S. government.
- Benos, D. J., K. L. Kirk and J. E. Hall. 2003. How to review a paper. Advances in Physics Education 27:47–52.
- Committee on Publication Ethics. 2010. Code of Conduct. http://publicationethics.org/files/u2/New_Code.pdf Accessed September 4, 2010.
- Dalton, R. 2001. Peers under pressure. Nature, 413:102–104.
- Fisher, M., S. B. Friedman and B. Strauss. 1994. The effects of blinding on acceptance of research papers by peer review. Journal of the American Medical Association 272:143–46.
- Garfunkle, J. M., et al. 1994. Effect of institutional prestige on reviewers’ recommendations and editorial decisions. Journal of the American Medical Association 272:137–38.
- Godlee, F., C. Gale and C. Martyn. Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: A randomized controlled trial. Journal of the American Medical Association 280:237–40.
- International Committee of Medical Journal Editors. 2010. Uniform Requirements for Manuscripts Submitted to Biomedical Journals. http://www.icmje.org/urm_main.html Accessed September 6, 2010.
- Justice, A. C., et al. 1998. Does masking author identity improve peer review quality? A randomised controlled trial. Journal of the American Medical Association 280:240–242.
- Kravitz, R., et al. 2010. Editorial peer reviewers’ recommendations at a general medical journal: Are they reliable and do editors care? PLoS One 5:e10072.
- McNutt, R. A., et al. 1990. The effects of blinding on the quality of peer review: A randomized trial. Journal of the American Medical Association 263:1371–76.
- Mulligan, A. 2005. Is peer review in crisis? Oral Oncology 41:135–141.
- Nature Editors. 2001. Editorial: Bad peer reviewers. Nature 413:93.
- Nature Editors. 2006. Editorial: Peer review and fraud. Nature 444:971–72.
- Nylenna, M., P. Riis and Y. Karlsson. 1994. Multiple blinded reviews of the same two manuscripts: Effects of referee characteristics and publication language. Journal of the American Medical Association 272:149–51.
- Peters, D., and Ceci, S. 1982. Peer-review practices of psychological journals: the fate of submitted articles, submitted again. Behavioral and Brain Science 5:187–255.
- Rennie, D. 2003. Misconduct and journal peer review. In Godlee, F., and Jefferson, T., eds. Peer Review in Health Sciences. 2nd edition. London: BMJ Books, 2003:118–29.
- Resnik D. B., C. Gutierrez-Ford and S. Peddada. 2008. Perceptions of ethical problems with scientific journal peer review: An exploratory study. Science and Engineering Ethics 14:305–10.
- Ross, J. S. et al. 2006. Effect of blinded peer review on abstract acceptance. Journal of the American Medical Association 295:1675–80.
- Rowe, B. H., et al. 2006. Reviewer agreement trends from four years of electronic submissions of conference abstract. BMC Medical Research Methodology 6:14.
- Schroter, S., et al. 2004. Effects of training on quality of peer review: Randomised controlled trial. British Medical Journal 328:673-75.
- Schroter, S., et al. 2006. Differences in review quality and recommendations for publication between reviewers suggested by authors or editors. Journal of the American Medical Association 295:314–17.
- Shamoo, A. S., and D. B. Resnik. 2009. Responsible Conduct of Research. 2nd edition. New York: Oxford University Press.
- Sieber, J. 2006. Quality and value: How can we research peer review? Nature. doi:10.1038/nature05006. http://www.nature.com/nature/peerreview/debate/nature05006.html
- Smith, R. 2006. Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine 99:178–82.
- Suls, J., and R. Martin. 2009. The air we breathe: A critical look at practices and alternatives in the peer review process. Perspectives on Psychological Science 4:40–50.
- Van Rooyen, S., et al. 1998. Effect of blinding and unmasking on the quality of peer review. Journal of the American Medical Association 280:234–237.
- Van Rooyen, S., et al. 1999. Effect of open peer review on quality of reviews and on reviewers’ recommendations: A randomised trial. British Medical Journal 317:23–27.
- Walsh, E., et al. 2000. Open peer review: A randomized controlled trial. The British Journal of Psychiatry 176: 47–51.Walsh, E., et al. 2000. Open peer review: A randomized controlled trial. The British Journal of Psychiatry 176: 47–5.