{"id":529871,"date":"2024-09-11T09:23:08","date_gmt":"2024-09-11T12:23:08","guid":{"rendered":"https:\/\/revistapesquisa.fapesp.br\/?p=529871"},"modified":"2024-09-11T11:34:41","modified_gmt":"2024-09-11T14:34:41","slug":"algorithm-encourages-constructive-and-respectful-peer-review","status":"publish","type":"post","link":"https:\/\/revistapesquisa.fapesp.br\/en\/algorithm-encourages-constructive-and-respectful-peer-review\/","title":{"rendered":"Algorithm encourages constructive and respectful peer review"},"content":{"rendered":"

\"\"C\u00e1ssia Roriz<\/span>\u201cI’m scared that this manuscript may contribute not so much towards the advance of the field of knowledge, but to its decline.\u201d<\/em><\/p>\n

\u201cDid you have a convulsion when you wrote this sentence? Because that’s what happened to me while reading it.”<\/em><\/p>\n

\u201cThis is an article fighting to stay alive.\u201d<\/em><\/p>\n

The harsh and rude comments above, which resemble insults commonly published on social media networks, were taken from reviews sent to researchers who submitted their articles for assessment by scientific journals or congresses. The authors expected their works would receive constructive criticism that would allow them to improve weak points and suggest new paths. After all, the reviewers are their peers, scientists working in the same field of knowledge who voluntarily gave up their time to read preliminary versions of their colleagues’ papers. However, instead of helping, these reviewers imposed a traumatic experience, which can be particularly daunting for young authors trying to find their feet in an academic career.<\/p>\n

But, after all, the rude sentences were not written in vain. Compiled in four open databases by computer scientists from the Indian Institute of Technology, in Patna, they are part of a set of 2,716 sentences used in peer-review processes (some bad-tempered, others polite, or neutral). They were used to create an algorithm, a prototype for the time being, that, if advanced, could be useful for editors and reviewers who wish to analyze the tone and quality of their English language evaluations \u2014 and modify them, if necessary. The work is part of computer scientist Prabhat Kumar Bharti’s PhD and was published in February in the journal Scientometrics<\/em>.<\/p>\n

To train the tool, the researchers from India used computer language techniques and established two categories \u2014 constructiveness and politeness (see table<\/em><\/a>) \u2014 into which an opinion can be classified. In the first step, the sentences were analyzed by four human annotators. They reached a consensus of 88.27% when classifying the levels of constructiveness and of 83.49% for politeness. Once trained, the computational model matched with the human classifications in 87.4% of the sentences. The study observed that, in general, the more constructive the comment, the more polite it usually is as well.<\/p>\n

It is no surprise that large scientific databases are used to develop and refine algorithms and tools\u2014in Brazil, computationally analyzed information from the Lattes website (a database containing the r\u00e9sum\u00e9s of researchers and students throughout Brazil) is being used to broaden understanding about scientific activity in Brazil and identify trends (see<\/em> Pesquisa FAPESP issue n\u00ba<\/em>\u00a0233<\/em><\/a>). Bharti explained in his LinkedIn profile that the purpose of the line of investigation is specifically to extract new understanding from a vast quantity of academic content. He and his colleagues highlight the model\u2019s potential for developing a science communication system that unites over 70,000 journals with peer-reviewed articles \u2014 approximately 2 million manuscripts are analyzed each year. More than simply helping the work of the evaluating scientists, one of the aims of the study, according to the authors, is to warn of the damage caused by destructive judgments. \u201cSome reviewers have shown unprofessional behavior, making depreciative comments about the author’s gender, race, ethnicity, and country of origin,\u201d they wrote.<\/p>\n

Computer scientist Jes\u00fas Mena-Chalco, of the Federal University of ABC (UFABC), notes that the creation of tools such as the one proposed by the group from India could be useful as an initial filter when editors have a large volume of reviews to assess. He says that he has been the victim of a comment that rudely demanded a professional English language review certificate. \u201cI recognized the need to improve the work, but the suggestion could have been said in a different way. There are ways of asking: some will encourage us and others will do the opposite,\u201d he says.<\/p>\n

For linguist Maria Jos\u00e9 Borcony Finatto of the Federal University of Rio Grande do Sul (UFRGS), who is an expert in computer language, the Indian proposal sheds light on a crucial problem of peer reviews. \u201cThis topic needs to be discussed in different environments, from journals to funding agencies,\u201d she says, having also been on the receiving end of inappropriate and impolite peer-review feedback. \u201cThe message was unpleasant and attacked me as a female researcher,\u201d she recalls.<\/p>\n

The size of the problem has already been investigated in other studies. One article published in 2019 showed that 58% of the 1,106 researchers from 46 countries claimed to have received at least one unprofessional review from periodicals. Some of the participants, all from areas of science, technology, mathematics, and engineering (STEM), shared excerpts from these reviews, such as \u201cSo and so tried this in the 1990s and failed, and he was more creative than you;\u201d \u201cThis article is simply manure;\u201d and \u201cThe condition of the author as a trans person has distorted their view about sex beyond biological reality.\u201d This study from 2019 measured the perception of researchers about the impacts of these comments in three aspects: scientific aptitude (confidence as a scientist), productivity (publications per year), and career progress (ability to advance within the field). The results indicated that white men were more likely to say that the opinion did not affect them than Black men, women, and white and Black nonbinary people. In other words, the authors concluded that researchers of the female sex and from minority groups appear to feel more upset by these rude reviews.<\/p>\n

\"\"C\u00e1ssia Roriz<\/span><\/p>\n

In search of constructive models
\n<\/strong>It is true that the majority of scientific journals have guidelines to assist peer reviews, but they do not guarantee that the result will be constructive and professional, reflects linguist M\u00e1rcia Sipavicius Seide, of Western Paran\u00e1 State University (UNIOESTE). Since 2020 she has coordinated writing clubs and editor training courses as university extension activities and she sometimes takes note of the feedback participants receive on submitted articles. \u201cSome comments are rude and do not offer guidance,\u201d says the linguist, who is also editor of the journal Onom\u00e1stica desde Am\u00e9rica Latina<\/em> (Onomastics from Latin America). \u201cIf the opinion does not point out what is wrong with the work and just rejects it, the author will not know where to go and this is harmful for science as a whole.\u201d<\/p>\n

This perception encouraged her to create models for assessing the quality of scientific opinions based on rubrics, sets of criteria used to measure the performance of a written text using points, like in university entrance exam essays. The aim is that they be pedagogic, polite, justified, and offer guidance for the research, in any field of knowledge. Seide proposes three models: for conditional approval of the article, in which adjustments are necessary; for rejection, when the paper is not accepted; and for direct approval, without adjustments. Each one has four criteria and five levels, and a total of between 10 and 20 points \u2014 the more points, the better the quality of the analysis. They were described in an article published in July 2023 in Revista Meta: Avalia\u00e7\u00e3o<\/em>.<\/p>\n

For conditional approval, one criterion for a good review considers whether the evaluator engages with the author, if they ask questions and explain what could be done to improve the text. \u201cThe text for rejection needs to be gentle, polite, and not discredit the author of the manuscript. And for direct approval, more points are given if the praise is justified, informing why the work was considered to be good,\u201d she explains. For her, as the assessment of an article is a subjective process, the rubrics should make them more transparent. She has been using the methodology since 2023 to assess the comments of reviewers of the journal Onom\u00e1stica<\/em>. \u201cI always check whether the evaluation \u2014 when it points out faults \u2014 offers suggestions aimed at resolving what was earmarked as the weak point of the article. I want the author to feel encouraged to continue writing.\u201d<\/p>\n

To minimize these issues, there are publications that seek to highlight the work of good reviewers. Since 2008, the American Physical Society<\/em> (APS) has recognized around 150 peer reviewers \u201cwho have been exceptionally useful in analyzing manuscripts for publication in APS<\/em> periodicals,\u201d according to the institution’s website, which has a database of 91,600 active reviewers. In 2024, among the 156 researchers from 58 countries highlighted as outstanding referees is physicist Paulo Campos, of the Federal University of Pernambuco (UFPE), the sole Brazilian on this year’s list. For him, the key to constructive criticism is offering guidance so that the article is improved. \u201cIf it has potential for approval, you have to indicate directions and gaps. If it is a rejection, you must point out the problems,\u201d observes Campos, who has been a peer reviewer of APS<\/em> periodicals since 2001. He sees the role of the scientific editor as fundamental for filtering aggressive comments and requesting further details for those that are unconstructive.<\/p>\n

Sigmar de Mello Rode, who is an editor of periodicals in the field of dentistry, explains that it is often necessary to reformulate the comments or even request further clarification from the reviewers who do not adequately support their suggestions. Rode, a researcher from S\u00e3o Paulo State University (UNESP), is president of the Brazilian Association of Scientific Editors (ABEC-BRASIL), which has created a course for scientific article reviewers. The initiative goes step by step through what needs to be observed in order to carry out critical and correct analysis. \u201cWe have had over 550 enrollments on the course in two years and we have realized that there is a demand for learning how to do it, as the instructions given to the reviewers by the journals are often incomplete,\u201d says Rode.<\/p>\n

\u201cWe still don’t discuss much the lack of transparency of the peer-review process, to which the authors often don’t have access,\u201d comments technology and digital media expert Alex Mendon\u00e7a, coordinator of the SciELO Preprints server. According to him, it is common to hear editors of periodicals in the SciELO library comment about problems involving destructive or succinct evaluations. \u201cThere is no silver bullet for dealing with this situation, but we have endorsed the migration towards open science including peer reviews. The more the reviews become public, the more the reviewers will think before making rude and prejudiced comments. In Brazil, he highlights the example of Bakhtiniana: Revista de Estudos do Discurso<\/em> (Bakhtiniana<\/em>:\u00a0Journal of discourse studies)<\/em>, which began attaching the judgment of the reviewers to the content of the articles and showing their identity. Eventually, the comment can receive a DOI (Digital Object Identifier), a unique identifier for digital documents. \u201cThis way, it can be included on the peer reviewer’s r\u00e9sum\u00e9, valuing their work and encouraging them to make constructive evaluations,\u201d suggests Mendon\u00e7a.<\/a><\/p>\n<\/div>

\n \n \n \n <\/picture>Alexandre Affonso \/ Revista Pesquisa FAPESP<\/span><\/div>
\n

Scientific articles<\/strong>
\nBHARTI, P. K. et al.<\/em> Please be polite to your peers: A multi task model for assessing the tone and objectivity of critiques of peer review comments<\/a>. Scientometrics<\/strong>. Feb. 2024
\nSILBIGER, N. J. & STUBLER, A. D. Unprofessional peer reviews disproportionately harm underrepresented groups in stem<\/a>. PeerJ<\/strong>. 7:e8247. Dec. 2019.
\nSEIDE, M. S. Rubricas de avalia\u00e7\u00e3o de parecer cient\u00edfico<\/a>. Revista Meta: Avalia\u00e7\u00e3o. Meta: Avalia\u00e7\u00e3o<\/strong>. Vol. 15, no. 48, pp. 620\u201340. July 2023.<\/p>\n","protected":false},"excerpt":{"rendered":"Rude, vague, and even discriminatory scientific opinions jeopardize the effectiveness of evaluations among scientists","protected":false},"author":684,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[166],"tags":[215],"coauthors":[2721],"class_list":["post-529871","post","type-post","status-publish","format-standard","hentry","category-policies-st-en","tag-scientometrics"],"acf":[],"_links":{"self":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts\/529871","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/users\/684"}],"replies":[{"embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/comments?post=529871"}],"version-history":[{"count":6,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts\/529871\/revisions"}],"predecessor-version":[{"id":531380,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts\/529871\/revisions\/531380"}],"wp:attachment":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/media?parent=529871"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/categories?post=529871"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/tags?post=529871"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/coauthors?post=529871"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}