Rethinking peer review: an empty slate

 

John Yaphe*

*MD, Associate Professor, School of Health Sciences, University of Minho.

 

Peer review is an established feature of scientific publication but its validity and reliability have been called into question. Richard Smith¹ has called it an empty gun. It might also be compared to an empty slate. This suggests that we are free to redesign it for our purposes. How can we do this?

Smith says that it fails because it is slow and expensive. It misses errors. Anything can get published eventually. It is biased towards positive results from famous people. Reviewers may steal ideas. Blinding, opening the process, and training reviewers have failed. He suggests publishing everything and letting the public decide. Is this the end of peer review?

Jefferson² shows how peer review is widely used without proper testing. He suggests the following quality criteria.³ A good review should state if a study is important, useful, relevant, methodologically and ethically sound, complete and accurate.

The editorial board of this journal uses the FINER criteria⁴ (feasible, interesting, new, ethical, relevant) to determine if a manuscript should be sent out for peer review. Authors are advised to consider this when planning a project.

Faria-Vaz described the qualities of a publishable article as relevance, novelty and compliance with norms of the journal.⁵ Peer reviewers identify weaknesses in submitted manuscripts, judge innovation and suggest changes. The feedback function of peer review may create a conflict between timely publication and ensuring the quality of published articles,⁶ but that is a responsibility all editors must bear.

A good peer review is one which addresses the importance and originality of the research question, strengths and weaknesses of the method, the technical presentation of the paper, the interpretation of the results, and the degree extent to which the reviewer provided constructive substantiated comments.⁷ Review-ers who provide higher quality reviews may be younger in age and have training in epidemiology and statistics. However, academic rank or training may not predict quality reviews.⁸

Adding a statistical reviewer may help⁹ but providing checklists of statistical guidelines alone does not. Training courses do little to improve the quality of reviews. In one study,¹⁰ trained and untrained reviewers picked up only 3 of 9 major errors deliberately inserted in papers sent for review.

There are biases in big journals. A paper is more like to be accepted if it reports on a randomized controlled trial or if the author is from the same country as the journal.¹¹ With poor agreement between reviewers, editors of a big journal are more likely to reject reviewers’ recommendations to publish.¹²

Are authors satisfied with peer review? Predictably, satisfaction is associated with acceptance of the manuscript and not with the quality of reviews. Authors of rejected manuscripts are unhappy with delayed replies and a lack of constructive feedback.¹³

Are reviewers happy? Reviewers are likely to refuse to review a paper because of lack of time and accept a review because of interest in the topic.¹⁴ Reviewers picked by the authors of a paper may write high quality reviews, but are more likely to recommend acceptance of a paper.¹⁵

Second-order peer review is an alternative. Readers can be provided with the results of published high quality research by a panel of experts in evidence based medicine.¹⁶ Letters to the editor may also fulfill this function.¹⁷

Open review is another alternative. Godlee¹⁸ suggests that this is ethical, lacks adverse effects, is feasible, and has potential to balance accountability with credit for the work reviewers do. She hopes this will replace the current, unreliable system of peer review. Of course, we also need to prove the effectiveness of open review.

In this light, I would like to conclude with a 12-point proposal for improvement.

1. We need to attract new, young, academic reviewers.

2. We need to seek more expert statistical advice.

3. We need to check the quality of our reviews.

4. We need to promote publication of better studies including both negative and positive results.

5. We must encourage original research in family medicine especially among young family doctors.

6. We need to increase interest and relevance while preserving quality.

7. We must provide clear guidelines for authors and expert editing services to help improve writing.

8. We must act quickly as editors and provide constructive feedback to authors.

9. We need to select reviewers carefully by areas of interest and avoid burnout.

10. We may consider allowing authors to suggest reviewers.

11. We need to publish relevant clinical reviews on focused questions.

12. We need to consider open un-blinded review.

We have just filled the empty slate. I look forward to the challenges ahead and am curious to see how we can push our journal forward. We invite you to join us in this effort.

 

Conflict of Interest Statement

The author is a member of the editorial board of RPCG, a peer-reviewed journal, and has written and reviewed for peer-reviewed journals.

 

References

1. Smith R. Classical peer review: an empty gun. Breast Cancer Res 2010 Dec 20; 12 Suppl 4: S13.

2. Jefferson T, Alderson P, Wager E, Davidoff F. Effects of editorial peer review: a systematic review. JAMA 2002 Jun 5; 287 (21): 2784-6.

3. Jefferson T, Wager E, Davidoff F. Measuring the quality of editorial peer review. JAMA 2002 Jun 5; 287 (21): 2786-90.

4. Hulley SB, Cummings SR, Browner WS, Grady DG, Newman TB. Designing Clinical Research. 3rd ed. Philadelphia, PA: Lippincott, Williams & Wilkins; 2007. p. 19-22.

5. Faria-Vaz A. Praxis editorial. Rev Port Clin Geral 2010 Mai-Jun; 26 (3): 241-2.        [ Links ]

6. Shashok K. Content and communication: How can peer review provide helpful feedback about the writing? BMC Med Res Methodol 2008 Jan 31; 8: 3. doi:10.1186/1471-2288-8-3

7. Black N, van Rooyen S, Godlee F, Smith R, Evans S. What makes a good reviewer and a good review for a general medical journal? JAMA 1998 Jul 15; 280 (3): 231-3.

8. Callaham ML, Tercier J. The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Med 2007 Jan; 4 (1): e40. doi:10.1371/journal.pmed.0040040

9. Cobo E, Selva-O’Callagham A, Ribera JM, Cardellach F, Dominguez R, Vilardell M. Statistical reviewers improve reporting in biomedical articles: a randomized trial. PLoS One 2007 Mar 28; 2 (3): e332. doi:10.1371/journal.pone.0000332

10. Schroter S, Black N, Evans S, Godlee F, Osorio L, Smith R.. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med 2008 Oct; 101 (10): 507-14. doi:10.1258/jrsm.2008.080062.

11. Lee KP, Boyd EA, Holroyd-Leduc JM, Bacchetti P, Bero LA. Predictors of publication: characteristics of submitted manuscripts associated with acceptance at major biomedical journals. Med J Aust 2006 Jun 19; 184 (12): 621-6.

12. Marušic A, Lukic IK, Marušic M, McNamee D, Sharp D, Horton R. Peer review in a small and a big medical journal: case study of the Croatian Medical Journal and The Lancet. Croat Med J 2002 Jun; 43 (3): 286-9.

13. Weber EJ, Katz PP, Waeckerle JF, Callaham ML. Author perception of peer review: impact of review quality and acceptance on satisfaction. JAMA 2002 Jun 5; 287 (21): 2790-3.

14. Tite L, Schroter S. Why do peer reviewers decline to review? A survey. J Epidemiol Community Health 2007 Jan; 61 (1): 9-12. doi: 10.1136/jech.2006.049817

15. Schroter S, Tite L, Hutchings A, Black N. Differences in review quality and recommendations for publication between peer reviewers suggested by authors or by editors. JAMA 2006 Jan 18; 295 (3): 314-7.

16. Haynes RB, Cotoi C, Holland J, Walters L, Wilczynski N, Jadraszewski D, et al. Second-order peer review of the medical literature for clinical practitioners. JAMA 2006 Apr 19; 295 (15): 1801-8.

17. Faria-Vaz A. A alma de uma revista. Rev Port Clin Geral 2010 Jul-Ago; 26 (4): 349.

18. Godlee F. Making reviewers visible: openness, accountability, and credit. JAMA 2002 Jun 5; 287 (21): 2762-5.