New peer review models show promising results but need careful consideration.

Effective peer review is a key component of scholarly communication. It evaluates research through close scrutiny by experts and funders and journals rely on it to determine the robustness of research findings or grant proposals.  While the peer review process continues to play a pivotal role in validating research results, it has also been widely criticised – it slows down the dissemination of research findings, sometimes fails to detect errors and studies suggest that it can introduce many types of bias.  [#pullquote#]new ways of applying technology to the peer review process show promising results.[#endpullquote#]However, new and innovative open peer review models have been developed to address these issues and new ways of applying technology to the peer review process show promising results.Open peer reviewNew peer review models are winning ground, but it is important to examine the benefits and potential biases that newer open systems of peer review may introduce. [#pullquote#]New peer review models are winning ground, but it is important to examine the benefits and potential biases [#endpullquote#]The version of the open-access scholarly publishing platform F1000Research is an example of full open peer review. This model discloses the identity of authors and reviewers and manuscripts, reports and comments are all publicly available throughout the review process.To explore the extent of judgement bias in the F1000Research model, we have set up a joint project with the University of Wolverhampton and F1000 to test if reviewers would judge research papers differently if they can see who else has reviewed the paper and what they have fed back.Since reviewers can view and read other reviewer reports before submitting their own for the same article, there is the possibility that previous comments influence subsequent reviewers. However, we have found little evidence that this is the case.  We also looked at whether a reviewer based in a specific country would assess the work of an author based in the same country more positively. We found a slight tendency for this. The most likely reason for this bias is that reviewers could be more likely to help or avoid problems with other researchers who they know.  While the results of our study are tentative and would need to be compared with other peer review models, we hope that it will contribute to an evidence base to inform decisions on how open peer review could be best applied.Using AI to support the peer review processAnother way traditional peer review is challenged is the use of artificial intelligence (AI). In recent years there has been growing interest in making the peer review process more efficient and effective.[#pullquote#]Numerous initiatives have looked at how elements of the peer review process can be automated through AI. [#endpullquote#]Numerous initiatives have looked at how elements of the peer review process can be automated through AI. For example, there are now AI-based tools that can help identify inconsistent statistical test results, detect plagiarism or find appropriate reviewers.  TripAdvisor algorithmOne aspect that can affect the effectiveness of peer review is the lack of consistency between reviewer comments and the overall recommendation.For example, if a reviewer provides a list of deficiencies and required improvements to the author of a manuscript and then recommends that the paper should be accepted for publication, or the reviewer is only positive about a research proposal and then recommends that it should not be funded. Working with the University of Wolverhampton, we have experimented with sentiment analysis of F1000Research open peer review reports to build PeerJudge – a tool which can detect positive and negative evaluations in reports.[#pullquote#]Sentiment analysis software can identify patterns in a text which are related to positive or negative words or phrases.[#endpullquote#]Sentiment analysis software can identify patterns in a text which are related to positive or negative words or phrases. In the same way that TripAdvisor works for hotels and restaurants, sentiment analysis has been widely used to automatically rate online user opinions about   peer review reports. Open peer review prediction toolIn the F1000Research open peer review model research articles are published after editorial checks but before peer review. The peer review reports are published alongside the article and when an article receives at least two reports with the overall rating of ‘approved’ it is submitted for scholarly indexing in bibliographic databases such as PubMed or Scopus.PeerJudge can help predict the overall reviewer decision and whether a paper will be ‘approved’, ‘approved with reservations’ or ‘not approved’. If further developed, the tool could be useful to notify reviewers that their peer review report and overall judgment are potentially inconsistent. It could help with journal reviewing consistency checks more generally, for example to compare different subsets such as between reviewers from different countries.[#pullquote#]Crucially, PeerJudge uses a transparent AI approach to detect judgments in the peer review reports. [#endpullquote#]It could also help with monitoring whether female authored articles in individual journals receive more critical evaluations. For example, recent research indicates that women in the field of economics are held to higher standards in peer review which might contribute to women publishing fewer papers. Crucially, PeerJudge uses a transparent AI approach to detect judgments in the peer review reports.The project also includes a briefing paper that gives an overview of recent developments in the automation of the peer review process (pdf) and discusses the opportunities for AI to support editors and reviewers. It also addresses some of the ethical challenges that arise with growing automation and the use of AI.The use of automation can support peer review in unexpected and promising ways but we need to tread carefully not to create unintended and potentially unwanted effects in the process.