PNAS2015-01-22 12:52 PM

Measuring the effectiveness of scientific gatekeeping

Kyle Silera,1, Kirby Leeb, and Lisa Beroc


Abstract

Peer review is the main institution responsible for the evaluation and gestation of scientific research. Although peer review is widely seen as vital to scientific evaluation, anecdotal evidence abounds of gatekeeping mistakes in leading journals, such as rejecting seminal contributions or accepting mediocre submissions. Systematic evidence regarding the effectiveness—or lack thereof—of scientific gatekeeping is scant, largely because access to rejected manuscripts from journals is rarely available. Using a dataset of 1,008 manuscripts submitted to three elite medical journals, we show differences in citation outcomes for articles that received different appraisals from editors and peer reviewers. Among rejected articles, desk-rejected manuscripts, deemed as unworthy of peer review by editors, received fewer citations than those sent for peer review. Among both rejected and accepted articles, manuscripts with lower scores from peer reviewers received relatively fewer citations when they were eventually published. However, hindsight reveals numerous questionable gatekeeping decisions. Of the 808 eventually published articles in our dataset, our three focal journals rejected many highly cited manuscripts, including the 14 most popular; roughly the top 2 percent. Of those 14 articles, 12 were desk-rejected. This finding raises concerns regarding whether peer review is ill-suited to recognize and gestate the most impactful ideas and research. Despite this finding, results show that in our case studies, on the whole, there was value added in peer review. Editors and peer reviewers generally—but not always—made good decisions regarding the identification and promotion of quality in scientific manuscripts.


Full Article

KEYWORDS

SHARE & LIKE

COMMENTS

ABOUT THE AUTHOR

PNAS

0 Following 4 Fans 0 Projects 58 Articles

SIMILAR ARTICLES

AbstractIn animal communication research, vocal labeling refers to incidents in which an animal consistently uses a specific acoustic signal when prese

Read More

AbstractWe show that easily accessible digital records of behavior, Facebook Likes, can be used to automatically and accurately predict a range of high

Read More

意义2010年 墨西哥湾北部深水地平线(MC252)灾难泄漏了超过400万桶原油。石油从海底架上升到了有许多大型中上层鱼类产卵的表层。在此我们描述了现场采集油样对包括蓝鳍金枪鱼,黄鳍金枪鱼和琥珀鱼在内的温水掠食动物迅速发育胚胎的影响。对于每个物种,环境相关MC252原油泄漏造成了心脏发育的严重缺陷。此

Read More

Abstract In the last decade there has been an exponential increase in knowledge about the genetic basis of complex human traits, including neuropsychi

Read More

AbstractGold mining has rapidly increased in western Amazonia, but the rates and ecological impacts of mining remain poorly known and potentially under

Read More

Abstract The interrelationships between hydrologically driven evolution of legacy landscapes downstream of major mining districts and the contaminatio

Read More

Abstract Changes in global (ocean and land) precipitation are among the most important and least well-understood consequences of climate change. Incre

Read More

AbstractAntiangiogenic therapy has shown clear activity and improved survival benefit for certain tumor types. However, an incomplete understanding of

Read More

AbstractThe biological mechanisms underlying long-term partner bonds in humans are unclear. The evolutionarily conserved neuropeptide oxytocin (OXT) is

Read More

Abstract Wine grapes present a unique biogeography model, wherein microbial biodiversity patterns across viticultural zones not only answer questions

Read More