Cássia Roriz“I’m scared that this manuscript may contribute not so much towards the advance of the field of knowledge, but to its decline.”
“Did you have a convulsion when you wrote this sentence? Because that’s what happened to me while reading it.”
“This is an article fighting to stay alive.”
The harsh and rude comments above, which resemble insults commonly published on social media networks, were taken from reviews sent to researchers who submitted their articles for assessment by scientific journals or congresses. The authors expected their works would receive constructive criticism that would allow them to improve weak points and suggest new paths. After all, the reviewers are their peers, scientists working in the same field of knowledge who voluntarily gave up their time to read preliminary versions of their colleagues’ papers. However, instead of helping, these reviewers imposed a traumatic experience, which can be particularly daunting for young authors trying to find their feet in an academic career.
But, after all, the rude sentences were not written in vain. Compiled in four open databases by computer scientists from the Indian Institute of Technology, in Patna, they are part of a set of 2,716 sentences used in peer-review processes (some bad-tempered, others polite, or neutral). They were used to create an algorithm, a prototype for the time being, that, if advanced, could be useful for editors and reviewers who wish to analyze the tone and quality of their English language evaluations — and modify them, if necessary. The work is part of computer scientist Prabhat Kumar Bharti’s PhD and was published in February in the journal Scientometrics.
To train the tool, the researchers from India used computer language techniques and established two categories — constructiveness and politeness (see table) — into which an opinion can be classified. In the first step, the sentences were analyzed by four human annotators. They reached a consensus of 88.27% when classifying the levels of constructiveness and of 83.49% for politeness. Once trained, the computational model matched with the human classifications in 87.4% of the sentences. The study observed that, in general, the more constructive the comment, the more polite it usually is as well.
It is no surprise that large scientific databases are used to develop and refine algorithms and tools—in Brazil, computationally analyzed information from the Lattes website (a database containing the résumés of researchers and students throughout Brazil) is being used to broaden understanding about scientific activity in Brazil and identify trends (see Pesquisa FAPESP issue nº 233). Bharti explained in his LinkedIn profile that the purpose of the line of investigation is specifically to extract new understanding from a vast quantity of academic content. He and his colleagues highlight the model’s potential for developing a science communication system that unites over 70,000 journals with peer-reviewed articles — approximately 2 million manuscripts are analyzed each year. More than simply helping the work of the evaluating scientists, one of the aims of the study, according to the authors, is to warn of the damage caused by destructive judgments. “Some reviewers have shown unprofessional behavior, making depreciative comments about the author’s gender, race, ethnicity, and country of origin,” they wrote.
Computer scientist Jesús Mena-Chalco, of the Federal University of ABC (UFABC), notes that the creation of tools such as the one proposed by the group from India could be useful as an initial filter when editors have a large volume of reviews to assess. He says that he has been the victim of a comment that rudely demanded a professional English language review certificate. “I recognized the need to improve the work, but the suggestion could have been said in a different way. There are ways of asking: some will encourage us and others will do the opposite,” he says.
For linguist Maria José Borcony Finatto of the Federal University of Rio Grande do Sul (UFRGS), who is an expert in computer language, the Indian proposal sheds light on a crucial problem of peer reviews. “This topic needs to be discussed in different environments, from journals to funding agencies,” she says, having also been on the receiving end of inappropriate and impolite peer-review feedback. “The message was unpleasant and attacked me as a female researcher,” she recalls.
The size of the problem has already been investigated in other studies. One article published in 2019 showed that 58% of the 1,106 researchers from 46 countries claimed to have received at least one unprofessional review from periodicals. Some of the participants, all from areas of science, technology, mathematics, and engineering (STEM), shared excerpts from these reviews, such as “So and so tried this in the 1990s and failed, and he was more creative than you;” “This article is simply manure;” and “The condition of the author as a trans person has distorted their view about sex beyond biological reality.” This study from 2019 measured the perception of researchers about the impacts of these comments in three aspects: scientific aptitude (confidence as a scientist), productivity (publications per year), and career progress (ability to advance within the field). The results indicated that white men were more likely to say that the opinion did not affect them than Black men, women, and white and Black nonbinary people. In other words, the authors concluded that researchers of the female sex and from minority groups appear to feel more upset by these rude reviews.
Cássia Roriz
In search of constructive models
It is true that the majority of scientific journals have guidelines to assist peer reviews, but they do not guarantee that the result will be constructive and professional, reflects linguist Márcia Sipavicius Seide, of Western Paraná State University (UNIOESTE). Since 2020 she has coordinated writing clubs and editor training courses as university extension activities and she sometimes takes note of the feedback participants receive on submitted articles. “Some comments are rude and do not offer guidance,” says the linguist, who is also editor of the journal Onomástica desde América Latina (Onomastics from Latin America). “If the opinion does not point out what is wrong with the work and just rejects it, the author will not know where to go and this is harmful for science as a whole.”
This perception encouraged her to create models for assessing the quality of scientific opinions based on rubrics, sets of criteria used to measure the performance of a written text using points, like in university entrance exam essays. The aim is that they be pedagogic, polite, justified, and offer guidance for the research, in any field of knowledge. Seide proposes three models: for conditional approval of the article, in which adjustments are necessary; for rejection, when the paper is not accepted; and for direct approval, without adjustments. Each one has four criteria and five levels, and a total of between 10 and 20 points — the more points, the better the quality of the analysis. They were described in an article published in July 2023 in Revista Meta: Avaliação.
For conditional approval, one criterion for a good review considers whether the evaluator engages with the author, if they ask questions and explain what could be done to improve the text. “The text for rejection needs to be gentle, polite, and not discredit the author of the manuscript. And for direct approval, more points are given if the praise is justified, informing why the work was considered to be good,” she explains. For her, as the assessment of an article is a subjective process, the rubrics should make them more transparent. She has been using the methodology since 2023 to assess the comments of reviewers of the journal Onomástica. “I always check whether the evaluation — when it points out faults — offers suggestions aimed at resolving what was earmarked as the weak point of the article. I want the author to feel encouraged to continue writing.”
To minimize these issues, there are publications that seek to highlight the work of good reviewers. Since 2008, the American Physical Society (APS) has recognized around 150 peer reviewers “who have been exceptionally useful in analyzing manuscripts for publication in APS periodicals,” according to the institution’s website, which has a database of 91,600 active reviewers. In 2024, among the 156 researchers from 58 countries highlighted as outstanding referees is physicist Paulo Campos, of the Federal University of Pernambuco (UFPE), the sole Brazilian on this year’s list. For him, the key to constructive criticism is offering guidance so that the article is improved. “If it has potential for approval, you have to indicate directions and gaps. If it is a rejection, you must point out the problems,” observes Campos, who has been a peer reviewer of APS periodicals since 2001. He sees the role of the scientific editor as fundamental for filtering aggressive comments and requesting further details for those that are unconstructive.
Sigmar de Mello Rode, who is an editor of periodicals in the field of dentistry, explains that it is often necessary to reformulate the comments or even request further clarification from the reviewers who do not adequately support their suggestions. Rode, a researcher from São Paulo State University (UNESP), is president of the Brazilian Association of Scientific Editors (ABEC-BRASIL), which has created a course for scientific article reviewers. The initiative goes step by step through what needs to be observed in order to carry out critical and correct analysis. “We have had over 550 enrollments on the course in two years and we have realized that there is a demand for learning how to do it, as the instructions given to the reviewers by the journals are often incomplete,” says Rode.
“We still don’t discuss much the lack of transparency of the peer-review process, to which the authors often don’t have access,” comments technology and digital media expert Alex Mendonça, coordinator of the SciELO Preprints server. According to him, it is common to hear editors of periodicals in the SciELO library comment about problems involving destructive or succinct evaluations. “There is no silver bullet for dealing with this situation, but we have endorsed the migration towards open science including peer reviews. The more the reviews become public, the more the reviewers will think before making rude and prejudiced comments. In Brazil, he highlights the example of Bakhtiniana: Revista de Estudos do Discurso (Bakhtiniana: Journal of discourse studies), which began attaching the judgment of the reviewers to the content of the articles and showing their identity. Eventually, the comment can receive a DOI (Digital Object Identifier), a unique identifier for digital documents. “This way, it can be included on the peer reviewer’s résumé, valuing their work and encouraging them to make constructive evaluations,” suggests Mendonça.
