In our most recent research, we have developed new methods to completely automate the reviewer-article assignment by using topic modeling. The system only needs the article’s abstracts and samples of abstracts written by the reviewers. This removes the biases introduced by either authors suggesting reviewers or reviewers bidding for their favorite articles to review.
Also, we have develop a new method for estimating the underlying score of an article based on judgments of a set of reviewers. Our method goes beyond a simple average of the reviewers’ scores, and it controls for the variances and biases of the reviewers. Sometimes reviewers are too harsh or too nice compared to other reviewers and the system regresses their scores to the mean accordingly. Also, some reviewers are very variables in their reviews or others are very consistent with others, and our system weighs their scores accordingly.