In the early stages of the pandemic, two articles about the novel coronavirus published in respected scientific journals had a significant impact and wide-ranging repercussions. The first, published in the New England Journal of Medicine, on May 1, 2020, evaluated the effects of certain heart disease drugs on patients infected with SARS-CoV-2. The second, published in The Lancet on May 22, suggested that as well as being ineffective against COVID-19, hydroxychloroquine also increased the risk of death from heart problems. The two studies were retracted on June 5 due to suspicions of fraud by the company that provided the underlying data, the veracity of which was impossible to confirm, meaning the results could not be proven (see Pesquisa FAPESP issue no. 293). That should have put an end to the scientific trajectories of these papers, but it did not. Despite having been retracted, they continued to be cited in specialist literature as if they were valid references, according to a survey by researchers from Australia and Sweden.
Based on data from Retraction Watch, a website that tracks scientific papers retracted due to errors or misconduct, 212 COVID-19 articles retracted before January 2022 were analyzed. One quarter of them presented clinical findings related to treating people affected by the disease. The researchers found that the articles received 1,036 citations in total, 80% of which were made after they had been withdrawn. And 86% of the papers that cited them did not indicate that they had been retracted, propagating the false idea that they were still considered valid by the scientific community. “In theory, when people cite retracted studies, they should be citing them in a critical way, alluding to the fact that these papers have been retracted because the research is unreliable,” said political and social scientist Steve McDonald, a senior research fellow at Monash University’s School of Public Health and Preventive Medicine in Australia and one of the authors of the survey, published as a preprint on the medRxiv platform in June. “But what we found was that actually in a lot of these cases, they weren’t citing it as a retraction. They were using it as evidence that ‘this particular intervention is effective,’ or ‘there’s nothing wrong with that research.’ So they were uncritically citing retracted papers.”
The group analyzed retracted COVID-19 articles specifically, but “from my experience, this seems to be quite common for non-COVID research as well, especially since it’s rare for people to actually look through their citations before they submit a paper,” said epidemiologist Gideon Meyerowitz-Katz of the University of Wollongong’s School of Health and Society in Australia, another author of the survey, in an interview with The Scientist.
This type of carelessness is far from uncommon. In recent years, several studies have drawn attention to the phenomenon, which affects the integrity of science. One article, published in late 2020 by researchers from the University of Illinois, USA, and the University of Adelaide, Australia, described the trajectory of a paper by Wataru Matsuyama, a surgeon from Kagoshima University in Japan. Published in 2005, it presented the results of a clinical trial that identified positive effects of omega-3 polyunsaturated fat in people with chronic obstructive pulmonary disease. An investigation into Matsuyama’s work led to the paper’s retraction in 2008 for data falsification. However, the researchers found another 35 papers that cited it between 2010 and 2019—none mentioned that it had been retracted. More recently, researchers from Graz University of Technology and the University of Salzburg, both in Austria, examined articles by American anesthesiologist Scott Reuben, a former professor at Tufts University in Boston who was sentenced to six months in prison in 2009 for fabricating data in papers published in the 1990s and 2000s. They analyzed the citation rates of several of Reuben’s papers, some retracted and others not. They found that his work was cited 420 times between 2009 and 2019, including 360 citations of retracted papers, most with no indication that they were no longer considered valid.
There are plenty of other examples. Many papers continue to exert influence after being withdrawn and scholars from various fields have tried to understand the factors behind their lasting impact on the scientific literature. One thing that seems clear is that most of the time, the authors do not realize they are referencing retracted articles. This is partly because retracted papers are stored in multiple repositories, not all of which have mechanisms for updating the status of each manuscript, so when searching for a subject on Google Scholar, for example, scientists may be redirected to seemingly valid copies of withdrawn articles. Ideally, retractions should always be flagged. “Platforms like Research Gate could be more active in this regard,” says Dutch microbiologist Elisabeth Bik, who specializes in identifying signs of error and fraud in scientific papers.
The risk in these cases is that retracted articles compromise the validity of those that referenced them, which go on to mislead other scientists, polluting the scientific literature with erroneous and false knowledge that could even put people’s lives at risk. Bik says publishers need to make more of an effort to notify people of corrections and retractions, but she recognizes that it would be difficult to be 100% efficient. “It’s like a manufacturer recalling a defective product,” she explains. “It’s almost impossible to reach every single customer who bought it.” Ivan Oransky, founder of Retraction Watch, believes “the effort to prevent retracted papers from continuing to infiltrate the scientific literature must be shared by the entire academic community.”
Efforts to prevent the spread of retracted papers must be shared by the entire academic community
One such initiative is the project Reducing the Inadvertent Spread of Retracted Science (RISRS), launched in 2020 by information scientist Jodi Schneider of the School of Information Science at the University of Illinois Urbana-Champaign, USA. The objective is to unite scientists, editors, research integrity offices, consultants, and other representatives of the academic ecosystem to discuss the phenomenon and think up ways to combat it. The discussions held so far have already resulted in a series of recommendations, such as the creation of a platform for all editors to monitor retracted articles and check if they appear in the bibliographies of newly accepted manuscripts. “Many have already adopted systems that allow people to check whether the version of the article they are reading is the most recent one,” points out Bik.
Another achievement by the RISRS project was the 2021 formation of a panel of experts to help the US National Information Standards Organization (NISO) develop guidelines on how editors should inform people of retractions, corrections, and expressions of concern (statements signaling that the results of an article are being reviewed). “The initiative will focus on what happens once something has been retracted: what metadata should be updated, how the retraction should be displayed, and how that information should be communicated,” Schneider said in an interview with The Scholarly Kitchen. “If it is successful, publishers, preprint servers, and data repositories will have guidance on metadata and display standards.”
“This will be very useful for Brazilian editors,” says Sigmar de Mello Rode, president of the Brazilian Association of Scientific Editors (Abec Brasil). He stresses, however, that “the most efficient control mechanism for preventing the spread of retracted articles is still rigorous and ethical peer review.”
For paleontologist Alexander Kellner, director of the National Museum at the Federal University of Rio de Janeiro (UFRJ) and editor in chief of Anais da Academia Brasileira de Ciências, fighting this problem in Brazil demands the professionalization of scientific editors. “Brazilian journals are managed by researchers who have to deal with several other career demands, unlike the biggest publications in the USA and UK, which have specific resources and infrastructure for this job,” he says. “We do not have the human and technical resources needed to analyze every reference in the manuscripts submitted to us.” Bik points out that there are already computer programs capable of performing this task. Reference management software, such as LibKey, Zotero, and Endnote, can cross-reference the identification code of cited studies with Retraction Watch’s database of over 36,000 retracted articles.
Jacqueline Leta, a biologist from UFRJ’s Institute of Medical Biochemistry who specializes in scientometrics, agrees that it would be useful to adopt automated tools to monitor retracted papers. “Most journals already use software to check for plagiarism in manuscripts submitted for publication,” she says. “It wouldn’t be difficult to add reference checking to the editorial routine.” If a citation of a retracted paper is identified, the journal would alert the authors and give them the option of deleting the reference or changing it to a footnote to prevent it from being included in citation indices—which do not distinguish between positive and negative references, leading to distortions in one of the most used quantitative methods for assessing research performance: the h-index (see Pesquisa FAPESP issue no. 207).
Leta highlights another front that could be improved: “Retraction notes must not be ambiguous or evasive. They need to be more transparent and provide as much detail as possible about the problems with the article, including the reasons for the retraction, who requested it, and whether there was a consensus among the editors.”
Even critical citations of retracted papers give a voice to works that should be forgotten
In 1998, British physician Andrew Wakefield published an article in The Lancet suggesting that the MMR vaccine is associated with cases of autism in children. The paper was retracted in 2010 due to methodological issues and misconduct, but the damage was already done. The article continues to be cited by antivaccination movements and its results are often reported as valid on social media.
It also continues to be discussed by the scientific community. A study led by UFRJ biologist Jacqueline Leta indicates that the paper has received 1,577 citations in articles indexed in the Scopus database, 53% of them after it was retracted. She analyzed 60 articles that cited Wakefield’s study and were themselves cited more than 100 times, finding that the majority acknowledged the article was invalid and referred to it negatively, highlighting its methodological problems and harmful impacts on society. “Even so,” says Leta, “they help promote Wakefield by allowing him to appear in renowned journals.”
This exposes a paradoxical aspect of the criticism of retracted scientific articles. Despite emphasizing that the conclusions are flawed and seeking to discuss the harmful effects, these papers give a voice to articles that should be forgotten.
There is still no consensus among researchers on how to address the issue. Some argue that access to retracted articles should be removed, since they have lost their scientific value. Others say that they shouldn’t even be the subject of further studies.
Dutch microbiologist Elisabeth Bik is against this idea. “It is important that retracted papers can be studied,” she says. The practice is valid, but because author productivity indicators do not distinguish between positive and negative citations, the recommendation is that references to retracted articles be made in footnotes—that is what Leta did with Wakefield’s paper, even though the doctor’s name appears in the title of her work.
