Peer-review is a cornerstone of modern scholarly communication. It is a process that relies on the voluntary work of researchers to evaluate the manuscripts authored by their peers. However, poor quality reports, misconduct, fraud, bias and other ethical issues are a growing concern. This is witnessed by an increasing number of post-publication retractions of papers in recent years, largely fueled by the uncovering of large paper mills. Several computational approaches have been previously proposed to support editors and publishers in assessing the quality of peer-review reports. Further, several studies have shown that state-of-the-art natural language processing (NLP) methods can be used to extract certain features, including politeness or social biases, from texts. However, the literature is often focused solely on the technical aspects of the proposed approaches and lacks the inclusion of business requirements and the human operators, i.e., the editors and publishers. The study thus aimed to address this gap by establishing the requirements and design principles for an effective information system that supports editors and publishers in screening and assessing peer-review reports on a large scale.

The study was conducted using the design science research (DSR) methodology. This methodology aims to design, develop and evaluate information systems artifacts in real-world settings, while ensuring they are grounded in theory. The design of the system roots in the problem awareness, which is established through a literature review and interviews with industry experts. Based on the problem awareness, the system is designed and implemented as a proof-of-concept. The system’s validity is evaluated in a real world setting by implementing two screening tasks: (1) assessing the quality of peer-review reports, and (2) screening for unprofessional comments. The first task is based on features extracted via text mining and NLP techniques. The second task is based on a sentence by sentence screening approach, where each sentence is compared against a set of previously established negative examples using semantic text similarity (STS) search.

A quantitative evaluation is performed for the first task using a sample of 5’000 review reports from MDPI (Basel, Switzerland). The quantitative evaluation showed that text mining features can be used to train explainable regression models to predict the quality of peer-review reports. A base model using only token counts yielded an RMSE of 0.14650 and explained 30.7% of the target variable’s variance. The quality prediction can be improved using text mining features on the report-level and features extracted from the sentence by sentence screening approach (final model: RMSE 0.13615, 42.7% explained variance). The regression model was intentionally limited to features extracted from the free-text review comments. Further, a qualitative evaluation of the system confirms hybrid intelligence with a human feedback loop as a key requirement for an effective system design. The qualitative evaluation further shows that the system can be used to screen for unprofessional comments, but that the STS search needs to be improved to reduce the number of false positives in the screning task.