Begin typing your search above and press return to search. Press Esc to cancel.

Automated manuscript-reviewer assignment and scoring

The peer review process suffers many biases related to gender, status, perceived novelty, and positive outcomes. Moreover, assigning reviewers to manuscript is a time consuming task that is known to produced more favorable reviews. In this project, we developed an automated reviewer-manuscript matching system solely based on the topics found in manuscripts and sets of reviewer’s sample articles. Our system was tested on COSYNE 2015 and 2016 where we found that automatically assigned reviewers were as experts as human assigned. This system is completely free and open access, supporting my goal of achieving reproducibility.

Only based on the abstracts of submitted manuscripts and sample abstract of reviewers, the system produces a reviewer-manuscript assignment that minimizes the total topic distance. It also produces assignments with constraints, such as limiting the number of reviews per manuscript and the number of reviews that each reviewer must complete.

How to combine the scores provided by a set of reviewers to produce a final manuscript decision? This part of our project produces an estimation of the final score of a manuscript while controlling for paper and reviewer-based biases. For example, reviewers that are consistently harsher that others have their review scores discounted. The approach is based on a Bayesian estimation algorithm and therefore it gracefully signals an additional estimation of its own uncertainty.

What's next

Automatic production of visit schedule for conferences and other time constraint events

The same principles that constraints assignment where there are conflicts of interests can be applied to time conflicts. We have tested this idea when we produced personalized time schedules for all attendees of the largest neuroscience conference in the world, SfN 2015. In this personalized schedule, each poster author was assigned a set of 50 posters to visit each day during the 5 day conference while controlling for time conflicts (e.g., not scheduling posters the same time they were presenting their own posters).

Applications to other conferences and reviewer-grant proposal assignment

I will be applying the same system to COSYNE 2016 and potentially other conferences. I know by personal communication that this system is sometimes being used to match reviewers to grant proposals at NIH.