Skip to content

Advances in Motion

Quality of Peer Reviews for a General Medical Journal Did Not Diminish During COVID-19 Pandemic

Journal

Originally published on JAMA Network Open

Key findings

  • This retrospective study of 5,013 manuscripts submitted to JAMA Network Open investigated whether peer review metrics changed with the onset of the COVID-19 pandemic
  • The pre-pandemic period studied was January 1, 2019, to March 10, 2020; the pandemic period studied was March 11, 2020, to June 29, 2021
  • The mean rate of reviewer acceptance of review requests remained stable (39.5% pre-pandemic vs. 38.4% during the pandemic; P=0.21), and the rate was substantially greater for COVID-19–focused manuscripts (67%)
  • Mean turnaround time was slightly but statistically significantly shorter during the pandemic than before (14.4 vs. 15.8 days; P<0.001), and for COVID-focused manuscripts it was 13.7 days
  • The quality of review improved somewhat; the mean number of reviews per manuscript that editors rated as very good or excellent increased from 1.3 before the pandemic to 1.5 during the pandemic (P<0.001)

The COVID-19 pandemic rekindled conversations about the viability of peer review, especially its burden on researchers, how best to ensure continued rigor and value of reviews and how to cope with the high volume of COVID-19–related manuscripts submitted by investigators who expected rapid publication.

Roy H. Perlis, MD, MSc, director of the Center for Quantitative Health in the Department of Psychiatry at Massachusetts General Hospital, and colleagues found the quality of peer review did not diminish during the pandemic at JAMA Network Open. In fact, their report cites a slightly shorter turnaround time during the first year of the pandemic and modestly increased editor-reported review quality.

Methods

The researchers retrospectively examined JAMA Network Open peer review data from two time periods:

  • Pre-pandemic—January 1, 2019, to March 10, 2020 (14.3 months)
  • Pandemic—March 11, 2020, to June 29, 2021 (15.6 months)

They included all 5,013 manuscripts received that were categorized as Original Investigations (n=4,295) or Research Letters (n=718) for which the editors sought at least one content review. 1,860 and 3,153 manuscripts were submitted during the pre-pandemic and pandemic periods, respectively.

Overall Results

On most measures, there were significant changes in the means before and after March 11, 2020:

  • Number of manuscripts reviewed per week—30 pre-pandemic period vs. 46 during the pandemic (P<0.001)
  • Number of reviewers invited per manuscript to achieve the minimum number of required reviews—6.0 vs. 7.0 (P<0.001)
  • Proportion of reviewers per manuscript who accepted invitations—39.5% vs. 38.4% (P=0.21)
  • Number of reviews returned per manuscript—1.6 vs. 1.7 (P<0.001)
  • Number of reviews per manuscript rated by editors as very good or excellent—1.3 vs. 1.5 (P<0.001)
  • Time to return reviews—15.8 vs. 14.4 days (P<0.001)

In multivariable linear regression adjusted for article type, study design, and direct submission versus transfer, time to return reviews remained shorter in the pandemic period (mean difference, −1.2 days; 95% CI, −0.7 to −1.6).

Results for COVID-19–focused Manuscripts

Similar patterns were observed when comparing manuscripts that concerned COVID-19 (n=915) with those that did not (n=2,238):

  • Number of reviewers invited per manuscript—6.4 vs. 7.2 (P<0.001)
  • Proportion of reviewers per manuscript who accepted invitations—67.2% vs. 64.8% (P=0.006)
  • Number of reviews per manuscript rated by editors as very good or excellent—1.5 vs. 1.4 (P<0.001)
  • Mean time to return reviews—13.7 vs. 14.6 (P=0.002)

Conclusions

The peer review process at this large open-access journal continued to function well during the pandemic despite changes in the volume of submissions and, presumably, many reviewers' work and home environments. Still, continued study of medical peer review is needed because there's abundant evidence COVID-19 has impacted researchers negatively, especially women.

30
manuscripts reviewed per week at JAMA Network Open before the COVID-19 pandemic

46
manuscripts reviewed per week at JAMA Network Open during the COVID-19 pandemic

40%
of reviewers for JAMA Network Open accepted invitations before the COVID-19 pandemic

67%
of reviewers for JAMA Network Open accepted invitations to review COVID-19–focused manuscripts

view original journal article Subscription may be required

Journal Article Published: Dates of Study: -

Explore research in the Department of Psychiatry

View the Department of Psychiatry's Guide to Mental Health Resources

Related topics

Related

Roy H. Perlis, MD, MSc, of the Department of Psychiatry, and colleagues compared features of major depressive disorder in individuals with or without prior COVID-19. They found indirect evidence that symptoms are a sequela of COVID-19 pathophysiology in a subset of individuals.

COVID-19, Depression, Neuroscience, Psychiatry

Journal

Related

Victor M. Castro, MS, Roy Perlis, MD, MSc, and colleagues determined that new-onset neuropsychiatric symptoms are common up to 150 days after COVID-19 hospitalization. Still, rates are no higher than those hospitalized for other indications during the same period—and rates of some symptoms are substantially lower.

Anxiety and Stress, COVID-19, Neuroscience, Psychiatry

Journal