In order to better meet the data requirements of oversight bodies and national and international ranking organizations, an increasing number of Brazilian universities are creating committees or offices dedicated to collecting and analyzing data on university performance. A recent example is the Office for Strategic Institutional Data created in September by the Federal University of S\u00e3o Paulo (UNIFESP). \u201cInformation is scattered across different departments within the university. Our mission is to organize this data and develop better metrics to evaluate teaching and research performance,\u201d explains Juliana Garcia Cespedes, associate dean for planning at UNIFESP.<\/p>\n
The university, along with other federal education and research institutions, is required to provide the Ministry of Education (MEC) and the Federal Audit Court (TCU) with a comprehensive and detailed set of information including budget performance, cost per student, the number of students who graduate within the time limits for their programs, and a \u201cfaculty qualification index.\u201d The data is compiled under the supervision of information technology officer Lidiane Cristina da Silva, and has yielded important dividends for the university that have underlined the importance of expanding and better structuring data collection efforts.<\/p>\n
Last April, the London-based magazine Times Higher Education <\/em>(THE<\/em>) launched a new academic ranking model that recognizes universities\u2019 social impact. Besides teaching and research indicators, the new ranking measures university performance against some of the United Nations (UN) Sustainable Development Goals, such as reduced inequalities, sustainable institutions, and impact on urban development. Many major universities in Brazil and around the world have been left out of the ranking primarily because they are unable to provide data on aspects such as the amount of waste they recycle or initiatives in place against violence.<\/p>\n But UNIFESP decided to take up the challenge and was rewarded with a place among the top 200 universities in the world in this ranking, alongside two other Brazilian universities\u2014the Federal University of ABC (UFABC) and the Federal University of Cear\u00e1 (UFC). \u201cWe were able to quickly gather the data required for the ranking thanks to the expertise of our information technology and planning departments,\u201d says dean Soraya Smaili. Having a woman as dean earned the university extra points for reducing inequalities, while providing medical care to communities, a tradition at the institution since its days as the S\u00e3o Paulo School of Medicine, also helped performance.<\/p>\n Academic performance indicators support managers in decision-making and help universities to report on their results to society\u2014which is ultimately the source of university funding. \u201cAnalyzing our indicators helps us to design better metrics for evaluating teaching and research performance in different fields of knowledge. It also helps universities to reflect on their relationship with the public,\u201d says Alu\u00edsio Cotrim Segurado, a professor at the University of S\u00e3o Paulo School of Medicine (FM-USP) and head of the institution\u2019s Office for Academic Performance Indicators (EGIDA). The office, created in 2018, is dedicated to improving data collection and interpretation in support of planning.<\/p>\n EGIDA also liaises with organizations publishing academic rankings. The importance of intensifying this engagement became evident, says Segurado, when USP found that its performance in British consultancy QS\u2019s world rankings for academic and employer reputation had been affected by the university\u2019s failure to update its list of researchers named to respond to the survey. \u201cSituations like these demonstrate how important these offices are for universities,\u201d says Segurado. Another challenge is designing new research metrics for areas where bibliometric indicators\u2014such as the number of published papers and citation indexes\u2014are not ideal, as is the case in the humanities. \u201cWe need to develop evaluation indicators that are meaningful for fields such as the arts, music, and cinema,\u201d he says.<\/p>\n Inputs from universities\u2019 performance metrics offices are helping FAPESP to update its Science, Technology, and Innovation (ST&I) Indicators for S\u00e3o Paulo, which it has compiled and reported since the late 1990s. The Foundation is working with university metrics offices and committees to collect data on research spending and development, sources of funding, and the number and profiles of researchers working at universities, research institutes, and companies in the state. \u201cThese offices have been a godsend. Before, universities lacked a single point of contact for information. Now it is easier to get data that is organized and consistent,\u201d explains economist Sin\u00e9sio Pires Ferreira, head of indicator studies at FAPESP.<\/p>\n Initiatives similar to those at USP and UNIFESP have been implemented at other public universities in Brazil. The Strategic Planning and Management Department at UFC, in Fortaleza, is currently developing new strategies for analyzing and monitoring metrics for academic performance. \u201cThe data we have collected and analyzed so far is helping us to design more effective institutional development plans focused on transparent management,\u201d says coordinator Roberta Queir\u00f3s Viana Maia.<\/p>\n USP, UNICAMP, and UNESP are working together to create metrics depicting the economic, social, and cultural impact of their activities<\/p><\/blockquote>\n In March 2017, S\u00e3o Paulo State University (UNESP) created a committee of administration, information science, and metrics experts to assess the institution\u2019s performance in several university rankings, including the THE World University Rankings and the Center for Science and Technology Studies ranking produced at Leiden University, in the Netherlands. \u201cWe can now monitor our standings, interpret the data, and determine how they can be acted upon,\u201d explains Jos\u00e9 Augusto Chaves Guimar\u00e3es, a professor at the Information Science Department at the UNESP campus in Mar\u00edlia.<\/p>\n During their monitoring efforts, the UNESP committee found that many researchers were publishing papers in Portuguese, limiting their international impact. It was also difficult to provide ranking organizations with a comprehensive picture of the institution\u2019s research, as the English translation of the university\u2019s name had not been standardized in scientific papers. The committee identified more than 90 different interpretations of the university\u2019s acronym in English.<\/p>\n To address these issues, UNESP launched a program, called Propetips, to encourage researchers to publish in English and in international journals, and to use the correct English translation for the university\u2019s name. \u201cThere are other obvious dos and don\u2019ts, such as not publishing in predatory, low-quality journals,\u201d explains Guimar\u00e3es. Several steps had already been taken before the commission was created. One was requiring all researchers to register for an Open Researcher and Contributor ID (ORCID), a persistent digital identifier in the global scientific environment that prevents researchers from being mistaken for others who share the same name<\/a>.<\/p>\n University metrics offices in Brazil have drawn inspiration from successful experiences at Harvard University, the University of Miami, and Cambridge. Data collection and analysis can inform strategies and policies that help to maintain or capture additional funding for research. They also track the career paths of undergraduate and graduate alumni, supplying ranking organizations with a diverse set of information.<\/p>\n But as Brazilian universities step up their efforts to measure ranking performance, Marisa Beppu, who served as associate dean for university development at the University of Campinas (UNICAMP) between 2017 and 2019, cautions that institutions should not be subservient to rankings. \u201cUniversity rankings should be used as tools for calibrating teaching and research activities,\u201d she says. \u201cUniversities need to critically analyze the rankings but also go beyond these data to include metrics that are meaningful in the Brazilian setting, says Milena Serafim, a staff member at UNICAMP\u2019s Coordination Office, the body running the university\u2019s institutional data management initiatives, under the leadership of deputy dean Teresa Atvars. Serafim and Atvars note that UNICAMP has worked to develop metrics over the years to improve its governance and aid decision-making involving administrative matters and the careers of professors and researchers. \u201cThe university has also invested in improving engagement with ranking organizations in order to develop metrics that are also meaningful for society,\u201d says Beppu. UNICAMP\u2019s impact on society is manifold, and is especially significant in areas such as intellectual property production and protection, technology transfer, and entrepreneurship\u2014the university last month announced that in 2019 it had spun out 114 new \u201cdaughter companies,\u201d including ventures created by students, alumni, and other individuals linked to the institution, businesses built around a technology licensed by the university, or firms that graduated from the university\u2019s technological incubator. In all, 717 daughter companies are currently in operation, generating a total of R$7.9 billion in revenue this year.<\/p>\n At USP, the EGIDA program runs a comprehensive impact assessment based on searches in external databases. Information compiled from the Lattes platform is used to estimate the impact USP has through training undergraduate and graduate professors to work at other universities in Brazil. \u201cAt some universities, more than half of professors received their training at USP,\u201d says Segurado.<\/p>\n The three state universities in S\u00e3o Paulo have developed a suite of metrics to assess economic, social, and cultural impact, as part of a program led by Jacques Marcovitch of the School of Economics, Management, and Accounting at USP (FEA-USP). The goal is to create indicators to track the extent to which these universities are fulfilling their missions. \u201cThere were already ongoing efforts to compile data for international rankings. We are taking advantage of the momentum to design indicators of university impact on economic development, innovation, community well-being, and other outcomes from investment with public funding,\u201d says Renato Pedrosa, a professor at UNICAMP and a member of the research team led by Marcovitch. The plan is for universities to generate data using similar and comparable methodologies, says Pedrosa, who also heads the FAPESP Science, Technology, and Innovation Indicators program. \u201cProfessor Marcovitch\u2019s program did exactly what FAPESP expects from a public policy research project: it identified a challenge, assembled a team of trained researchers, and secured buy-in from an organization interested in using the research results, in this case the Council of Deans of State Universities in S\u00e3o Paulo (CRUESP),\u201d says FAPESP Science Director Carlos Henrique de Brito Cruz.<\/p>\n