Imprimir Republish

Indicators

The fascination with rankings

Lists of the best universities gain in popularity, despite their methodological limitations

A contradiction runs right through every one of the more than 10 international rankings of universities adopted annually as a reference point: these lists were never as popular and influential as now, but it also adds to the perception that, because of their methodological limitations, no ranking singlehandedly perfectly portrays scientific and academic excellence. A debate on March 11 in Hong Kong, at a conference organized by the British Council, brought together those responsible for two of these rankings and researchers critical of the methodology adopted. What was seen was an exposé of the usefulness and limitations of rankings. Dzulkifli Abdul Razak, vice president of the Sains Malaysia University, for example, criticized the rankings because they divert the attention of universities away from their primary missions, which are finding solutions for combating hunger or the effects of climate change, for example. Phil Baty, editor of the supplement, the Times Higher Education (THE) ranking, and John Molony, from the QS ranking, both from the United Kingdom, admitted that methodologies needed to be improved, but emphasized that the lists introduced objectivity to the comparison of institutions, which had previously been assessed purely on the basis of their reputation. “It was clear in the debate that the rankings are inaccurate and should not be used for formulating scientific policy,” says physicist, Leandro Tessler, a professor from the State University of Campinas (Unicamp), who attended the debate in Hong Kong. “But provided their limitations are understood they can be a tool for universities to find new paths.”

The disclosure last month of the results of a reputation survey that is used by the THE ranking was evidence of the paradox. The data indicate that there are no Latin American universities in the list of the 200 most respected universities in the world, which is dominated by institutions from the United States and England (Harvard is first and Cambridge is sixth). The University of São Paulo (USP) almost made the list, according to the ranking’s organizers, and is a little lower than the 200th place. The data sparked a debate in the media and academia about the ills of Brazilian universities, while countries like Russia, India and China managed a place among the best.

Other rankings, however, show a different picture. Taking the case of USP, it would be among the 100 best when its capacity for forming CEOs of major corporations is evaluated (19th place in the ranking of École de Mines de Paris and Fortune magazine). It also stands out in the production of scientific articles (19th place in the SCImago ranking, which is based on the volume of publications, their quality and international collaboration), is prodigious when it comes to placing its academic production on the Internet (51st place in the Webometrics ranking) and has world-class clinical medical research (96th place in the ranking of this field of knowledge, according to the Shangai Jiao Tong University). In other areas its situation is more vulnerable, which is the case for example in the impact of its research (451st place in the ranking of citations from Leiden University in the Netherlands). “Of course USP has a lot of room to improve, such as in its relationship with the production sector and with government, but the balance of these rankings is favorable to the university,” says Marco Antonio Zago, Dean of Research at USP.

For Hernan Chaimovich, a retired professor from USP’s Institute of Chemistry, the university should discuss what it wants for the future: “To become a global player USP must, for example, take part in defining global standards of scientific and academic quality, instead of passively watching as the ranking results are disclosed”, says Chaimovich, who is also coordinator of FAPESP’s Centers of Research, Innovation and Dissemination (Cepid) and superintendent of the Butantan Foundation. According to the professor, some criteria adopted in the surveys, such as research reputation, are of limited use. “What importance is it for USP if it is not known by a researcher interviewed in Switzerland?” he asks. For Marco Antonio Zago, the rankings are useful for comparing institutions, but there are factors privileged by these lists that are of no interest to USP. “Some of them value the number of foreign graduate students. This contingent, which is around 2% at USP, could increase but never reach the 30% of a British university, which depends on attracting students from abroad for its funding. In our case, this would mean using the money from São Paulo taxpayers to educate more foreigners, which would not make sense.”

Each ranking has its own methodology. “You have to consider that these rankings are heterogeneous and not just because of the methodology,” says Zago. “Some look for data on the Internet, while others ask for information from the institutions, but that’s not all. The parameters and weights are also different, and some of these parameters change from one year to the next,” he says. The THE ranking comprises a mixture of criteria that undergo changes and adaptations every year. Research has the biggest weight in the overall picture. The volume of articles, the investment of each institution and the reputation of the research carried out by 13,388 researchers from 131 countries, which is compiled by Thomson Reuters and disclosed in March, has a weight of 30%. The research impact, measured by article citation indicators, is worth another 32.5%. The learning environment has a weight of 30% and considers various indicators. Professors and students of other nationalities, which indicate the degree of internationalization, is worth 5%. Finally, 2.5% of the weight relates to innovation, measured by the investment from industry.

Nobel
The methodology of the Shangai Jiao Tong University ranking combines research indicators, such as the volume of research and citations with quality data, like the number of researchers who have won Nobel Prizes or Field Medals, or scientists who have highly cited articles. “These are criteria that are a portrait of the Ivy League universities in the United States,” says Valdemar Sguissardi, a professor on the post-graduate program in education at the Methodist University of Piracicaba. “In a globalized world rankings are a reality, but we need to look at their data in relative terms. When they favor article citations, they undoubtedly have an Anglo-Saxon bias, because they favor researchers from countries where English is spoken. Other lists, on the other hand, favor scientific indicators from the ‘hard’ areas, thus prejudicing universities that are strong in social sciences and the humanities,” says Sguissardi. He also remembers that the initial function of various rankings was to serve as a reference point for students interested in studying abroad. “We have to bear in mind that these rankings form part of the logic of big business. Imagine the impact on the finances of the University of Cambridge from its privileged position in the rankings, when almost 40% of its students are foreigners and, with the exception of those from the European Union, pay the full university fees.”

In an article published in the journal, The Chronicle of Higher Education, Ellen Hazelkorn, research vice president at the Dublin Institute of Technology, summed up the difficulty with the rankings: “The dispute between the organizations of classification has not solved a fundamental question: is it possible to measure or compare institutions as a whole, considering their different missions and their national and financial contexts?”. Despite this limitation, she observes, countries like China, Finland, France, Germany, India, Japan, Malaysia, Russia, Spain, South Korea, Taiwan and Vietnam have introduced policies for creating their own “world-class universities,” using definitions inspired by the rankings.

The president of Unicamp, Fernando Costa, considers it important to monitor what the rankings say. “The question is not to choose one of them and want to be among the first 100, but to use these surveys to reflect upon the qualities of a world-class university,” he says. “There are obviously criteria, which, while desirable, are unfortunately far from the Brazilian reality, such as having Nobel Prize or Field Medal winners.” According to him, the São Paulo state universities incorporate several of the requirements of the world’s best universities and have indicated their willingness to move forward in this area. “For example, we have regular and sufficient funds, albeit at a lower level than North American universities. We are also very concerned with rewarding professors on merit, but the characteristics of our institutions do not allow for the same approach seen in North American universities”, he says. Unicamp has made an effort over the last twenty years to assess the performance of its professors. Every three years all professors need to prepare a report of their activities. “Those whose performance is unsatisfactory cannot be dismissed, because they have security of tenure by law, but they can have their salaries reduced. Several professors have lost their fulltime status because their performance was not up to scratch,” he says. Another front is that of internationalization; attracting students and professors from other countries. “We make an effort to bring good researchers to Brazil and we have various foreign professors, but there are limits. The entry system is by admission exam and it’s not easy to convince a foreigner to apply for it. Now we’re giving them the opportunity of a traineeship and the possibility of doing the exam in English to facilitate the process,” he says.

Recognition
For the president of Paulista State University, Julio Cezar Durigan, rankings are important for providing visibility to universities that have created scientific excellence but are little known. “In our case it allowed Unesp to gain international recognition and encouraged us to improve”, he says, referring to the institution’s performance in the survey by the Shangai Jiao Tong University, in which it has risen by more than 100 positions over the last two years: currently Unesp is in the group of the 350 best universities in the world. “We have advanced a lot in the number of PhDs: we’re turning out 800 a year, which is more than many North American universities. We still have a long way to go in the fields of the globalization of research and innovation. One of our targets is to send at least 20% of our students to study abroad; today this is just 6%,” says Durigan.

Hernan Chaimovich from USP draws attention to the definition of a world-class university. “It’s tautological,” he declares. “It’s clear that the 10 that head up most of the rankings are world-class universities. With the rest, we don’t know if the term means something or if it’s a marketing tool.” According to Chaimovich there is not always a connection between the qualities of a great university and the criteria considered by these lists. “The factors assessed by the rankings are research, the number of citations of articles and the institution’s public exposure. They lack a methodology for evaluating teaching and learning processes. These need to be developed,” he says.

Chaimovich says he would like to have a Harvard in Brazil “and preferably without having to wait the 500 years it took to become the best.” However, he cites a series of characteristics of Harvard that do not apply to the reality of Brazilian public universities. “In a world-class university the professors are not selected by public entrance exam. Harvard selects its teachers thinking about the future, in order to develop priority fronts of knowledge. I know of no world-class university that chooses its president by vote or that does not openly support meritocracy by way of salary, laboratory space or power. I know no world-class university that does not look to its own country, even though it brings together the world’s best brains for thinking. And I know no world-class university that is not deeply concerned with graduation courses, which is a great source of prestige,” says the professor.

Republish