Driven by a desire to spread new knowledge, researchers tend to grow apprehensive when articles in which they invested significant time and effort go unreferenced in papers by other authors. Their concerns are often justified, since the number of times an article is cited is considered a good measure of its impact—a high number indicates that the results are recognized by the scientific community, potentially serving as a reference for future studies. But the idea that uncited articles are not impactful is questioned by a paper in July’s issue of the journal Scientometrics. Jianhua Hou and Jiantao Ye, researchers at the School of Information Management at Sun Yat-Sen University in Guangzhou, China, analyzed 1,068 articles published in the respected journal PLOS ONE between 2006 and 2014, none of which had received a single citation five years after being published.
Hou and Ye found that despite never being cited, these papers had repercussions, which can be measured by how often they are viewed (PLOS ONE is an open access journal), downloaded, and saved using academic software, such as Mendeley and CiteULike, and to what extent their results are discussed in the journal’s own comments section or on social media, such as Facebook, Twitter, and Wikipedia. The resonance of the uncited articles in these media was compared to 586 highly cited articles published in PLOS ONE in the same period. The conclusion was that of those with no citations, the most read, downloaded, and discussed 1% performed better in these metrics than the top 31% of the highly cited papers. “These uncited articles have great influence on social media platforms,” wrote the authors.
Understanding why papers go uncited is a major challenge and has become a research topic of its own in the field of scientometry, which examines quantitative aspects of science. Studies on the subject reveal a highly nuanced picture. Many articles are not cited simply because they are not novel or they are of low quality. A study uploaded to the arXiv preprint repository in December 2019 analyzed a sample of studies published in 2014 in 250 scientific journals identified as “predatory,” meaning they do not use a proper peer review process and publish papers in exchange for money alone. According to the survey, conducted by researchers at the Hanken School of Economics and Tampere University in Finland, 6 of every 10 articles published in predatory journals had not been cited five years later. The result was compared with 250 articles selected at random from the Scopus database, composed of journals that follow strict quality standards. In this control group, only 6% of the papers had not been cited.
The set of variables that leads to an article having no apparent impact, however, involves much more than just quality. The chance of a paper not being cited varies widely depending on the field of knowledge, and reflects scientific communication practices specific to each discipline. In 2018, the magazine Times Higher Education (THE) analyzed 10,000 articles indexed in the Scopus database and found that articles on the visual and performing arts were most likely to remain uncited, with 77% of those published in 2012 not having been cited by 2017. Literature and literary theory came second, with 75%. At the opposite end of the spectrum, only 3% of articles on catalysis, a subfield of chemistry, had not been referenced in at least one other paper (see table).
The apparent disadvantage faced by the arts and humanities can be explained by the fact that the most robust output in these fields takes the form of books or book chapters, which are more commonly cited in other works. Even some fields of the hard sciences, such as engineering, have a significant number of uncited articles. “The focus in engineering is to create devices and tests, and to use these for real-world applications, which takes time. Publishing is not the first priority,” industrial engineer Frede Blaabjerg, from Aalborg University in Denmark, explained to THE. According to him, the engineering subfields are made up of small communities, which results in a limited number of researchers who might cite any given article.
Papers describing new living species are published in large numbers every year, and in general, they receive few citations—which does not detract from their importance. Michael MacRoberts, a botanist at Louisiana State University in the USA, published an article in 2010 about the shortcomings of citation analysis, including a reference to one of his own papers: the description of a moss species found in the state of Texas. This was the only time that his description of the species had been cited, but the moss is registered in plant atlases and online databases, establishing itself as a valuable contribution to the field. Dentist Sigmar de Mello Rode, president of the Brazilian Association of Scientific Editors (ABEC), says that the weight attributed to citations has fallen as alternative impact indicators have emerged. “When an author successfully communicates the knowledge he has produced via social and scientific media, and as a result, generates benefits for society, the number of citations the work has obtained may be of secondary importance, even if this metric is important in academia,” says Rode, who is a researcher at São Paulo State University (UNESP) in São José dos Campos.
Articles in interdisciplinary journals tend to be more frequently cited, but are also influenced by characteristics of the field to which the journal is most closely linked. In 2017, researchers from the School of Public Health (FSP) at the University of São Paulo (USP) analyzed uncited articles in eight Brazilian public health journals and observed a major disparity between them. The periodical with the most uncited articles was the one most closely associated with the humanities: História, Ciência, Saúde – Manguinhos (History, science, health), published by Casa de Oswaldo Cruz since 1994. Of the articles it published between 2008 and 2012, 58% had not been cited within the following three years. The journals with the least uncited articles, at just 6%, were Revista de Saúde Pública (Public Health Journal) and Cadernos de Saúde Pública (Notes on Public Health), which more strongly identify with the field of biomedicine.
The group is still investigating the reasons why an article is referenced or not. As part of her PhD, nutritionist Milena Maria de Araújo Lima Barbosa is studying the attributes of frequently cited nutrition articles. She notes, for example, that papers resulting from research backed by funding agencies receive more citations. “This is probably due to the credibility it lends the article, suggesting that the research has already undergone a level of quality assessment,” says group leader Angela M. Belloni Cuenca, a researcher at FSP-USP and executive editor of Revista de Saúde Pública. Articles are also cited more if they are published by multiple authors or in interdisciplinary journals, or involve international collaborations. There were also various factors linked to low citations rates, such as long titles that only describe the topic of the article without giving the reader any idea of the study’s conclusions. Longer abstracts were associated with a greater number of citations, but the authors point out that “when writing an abstract, it is important not to waste words; carefully choose words that increase the chance of the article being found by search engines and boost its visibility online. Avoiding repetition by using synonyms also helps.”
Articles published via open access tend to be more cited than those only available through subscriptions, but this is only the case for reputable journals that are among the most highly cited in each discipline. A study published in the journal Scientometrics in July by Spanish researchers from the University of Las Palmas de Gran Canaria showed that high-impact open access journals tend to attract more citations than subscription journals. But this advantage is not seen in open access publications with a lower impact, which includes many so-called predatory journals.
The origin of the paper and the language it is written in are also decisive factors in whether it is likely to be cited or not. Papers in English, the lingua franca of the science world, are generally more cited than those written in other languages, just as those by authors from core countries, such as the USA and Europe, are more likely to be mentioned than articles by people from countries like China, Russia, India, and Brazil. “If international databases included more journals in local languages, the number of papers with no citations would be much lower,” information scientist Vincent Larivière, from the University of Montreal, Canada, told the journal Nature. Being absolutely certain that an article has never been cited is itself a challenge, since every database determines this information in its own particular way. In 2011, a survey published in the Journal of the Association for Information Science and Technology estimated that 10% of all articles written by Nobel laureates have never been cited. The following year, Czech biologist Petr Heneberg, from Charles University in Prague, analyzed papers written by 10 Nobel prize winners indexed in the Web of Science database, finding that the rate was much lower, at around 1.6%. When he cross-checked the data against the Google Scholar database, he obtained an even lower figure, at 0.3%.
The idea that some scientific literature has no impact has worried researchers since the early 1990s, when a study by the journal Science estimated at 55% of all documents published between 1981 and 1985 had not been cited within five years. The research, however, covered not only scientific articles, but also letters, corrections, and summaries of scientific meetings published in journals—works that are not usually cited. In 2017, Larivière revealed a more encouraging situation. He found that of the 39 million original and review articles published between 1900 and the end of 2015 and indexed in the Web of Science database, 21% had never been cited, most of which were in low-impact journals. In well-known and reputable journals, the figure was very small. According to the researcher, there has been a steady decline in the number of uncited articles over recent decades, with the internet making it easier to find and cite relevant papers. But, according to Larivière, this does not necessarily mean that the science produced today has more of an impact. With the exponential worldwide growth of scientific output in the last two decades, the number of references made in articles has exploded, and being cited has become much more likely.
For Sigmar Rode, the high number of uncited articles is also a reflection of an aspect of scientific communication that he believes needs to be changed—the excessive number of journals across so many fields, which leads to a tendency to publish papers that offer few new contributions and are unlikely to be cited by others. As an example, he mentions a journal he edited for many years, Brazilian Oral Research, which was the result of a merger in the 1990s of three scientific publications on dentistry that were published by different groups at USP. “The merger was driven by the Brazilian Dental Research Society, which funded the new journal. Many groups of journals discuss similar moves, but they often face the same problem: who will give up their periodical to merge with the other?” says Rode, who believes that scientific journals wishing to compete internationally need to be separated from those that make important contributions, but are of interest only to particular communities. “The best example is Revista da Associação Paulista dos Cirurgiões Dentistas (the journal of the São Paulo association of dental surgeons), which for years has been renowned for publishing Portuguese articles on clinical cases targeted specifically at dentists. It has a huge impact for thousands of Brazilian dentists, but it is not indexed in international databases and does not measure its impact factor based on citations, which have no relevance to its objective.”
Republish