CAPES assessment finds that quality indicators for master’s and doctoral courses in Brazil have improved : Revista Pesquisa Fapesp

The Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES), linked to the Ministry of Education (MEC), released the results from its four-year assessment of graduate programs last December, a year overdue—the CAPES assessment has been used since the 1970s to measure the quality of master’s and doctoral programs and to inform the distribution of research grants and fellowships. Despite facing significant funding cuts and eroded fellowship awards over the past few years, many programs showed an improvement in their performance indicators. Out of the 4,512 programs assessed from 2017 to 2020, 34% improved their assessment scores. The number of programs rated as world class—those receiving the highest assessment scores of 6 and 7—rose by 37%, from 490 to 671.

The majority of these programs are located in Brazil’s South and Southeast states, especially São Paulo, Rio de Janeiro, Rio Grande do Sul, and Minas Gerais. But other regions have been catching up. In the Northeast, for instance, the number of world-class programs rose from 37 to 60, a gain of 62%. The North went from having no programs with a score of 7 in the previous assessment to three programs in the current assessment—two at the Federal University of Pará (UFPA) and one at the Brazilian Institute for Amazon Research (INPA). “We were expecting our scores to improve, but they exceeded our expectations and this will greatly enhance our ability to get funding for our graduate programs,” said psychologist Emmanuel Zagury Tourinho, dean of the university, in a video posted on UFPA’s social media.

The current edition’s results are consistent with trends seen in previous assessments. The University of São Paulo (USP) remains the institution with the highest number of programs rated 6 and 7—a total of 114, largely in the fields of life sciences, engineering, and medicine. The Federal University of ABC (UFABC), founded just 17 years ago, also celebrated a strong performance in the current assessment. Of the 29 programs evaluated, 10 improved their scores, with two programs receiving a score of 6—nanomaterials and advanced chemistry and science and technology. This was “a notable achievement for a relatively young institution,” says Charles Morphy, the university’s associate dean for graduate student affairs. Meanwhile, São Paulo State University’s (UNESP) graduate department had 10 programs with a top score of 7, including programs in physics, chemistry, and dentistry. “This reflects our practice of setting targets and tracking program performance against those targets over each four-year period,” says chemist Maria Valnice Boldrin, UNESP’s associate dean for graduate affairs.

At the University of Campinas (UNICAMP), 30 programs received higher scores than in the previous cycle, with 37 receiving world-class ratings, including in fields such as the humanities, food science, engineering, medicine, and dentistry. The office of the associate dean for graduate student affairs said in a statement that they will work with the coordinators of programs scoring 3 and 4 to create strategies for improvement in the next assessment.

Highly ranked programs not only gain academic prestige but also enjoy greater independence and are eligible for funding directly from CAPES via the Academic Excellence Program (ProEx). Although programs with scores of 3 to 5 are also eligible for funding from the Graduate Support Program (ProAP), the awards are smaller. In addition to funding, high-scoring programs have a better chance of having their research projects selected in calls for proposals organized by CAPES and the Brazilian National Council for Scientific and Technological Development (CNPq).

Inevitably, the funding constraints for grants and projects in recent years have affected well and poorly ranked programs alike. “Graduate programs have had to navigate a number of hurdles caused by the pandemic and the limited funding for science and education,” says political scientist Rachel Meneguello, associate dean for graduate student affairs at UNICAMP. “Insufficient fellowship awards, eroded by inflation, are one example.” Fellowship awards have not been adjusted for inflation since 2013, while the General Market Price Index (IGP-M) has increased by 125.1% since then. Currently, master’s fellows receive R$1,500, and doctoral fellows receive R$2,200.

Initiatives such as the Institutional Program to Internationalize Brazilian Universities (PrInt)—launched by CAPES in 2017 to establish international collaborations for master’s and doctoral degree programs—have also been affected. Student mobility grants were not awarded in 2020, and the program was suspended entirely in 2021. Boldrin explains that the program, which was slated to run until 2021, has now been extended to 2024, and grants that were meant to be paid in 2020 and 2021 have been rescheduled for 2023 and 2024.

The reasons for the strong performance of highly ranked programs are not clear, but several hypotheses have been suggested. One possibility is that the CAPES assessment approach has been modified to focus more on qualitative aspects such as the quality of training provided and the intellectual property generated by a program. Previously, program managers completed a questionnaire that provided information about the program’s approach, faculty qualifications, student profiles, and the number of papers published in the highest tiers of Qualis, a journal ranking system used by CAPES. “But in the most recent assessment, programs were also asked to name their students’ and researchers’ top papers for consideration,” notes epidemiology researcher Rita Barradas Barata, a former director at CAPES who participated in the assessment of graduate programs in the field of public health.

Assessed programs reported 4.7 million papers authored from 2017 to 2020. Of these, 280,000 were submitted by program coordinators and assessed on impact, originality, and innovation. “This may have helped programs perform better than they would have if the assessment had focused on quantitative aspects only,” says Barata.

Alexandre Affonso / Revista Pesquisa FAPESP

Experts interviewed for this article said there has been a trend within CAPES’s Technical and Scientific Council for Higher Education (CTC), the highest decision-making body on graduate assessments, to give subject-matter coordinators greater autonomy in recommending scores for programs. In previous years’ assessments, CAPES created a special committee to review any scores of 6 and 7.

It is also possible that programs’ quality metrics have improved as a result of investments made during the previous four-year period, and which were not affected by recent constraints. “A program’s performance in each assessment is often the result of past efforts,” says Morphy of UFABC. “The first half of the 2010s was marked by a policy that valued and worked to expand graduate education, and it is possible that current results are an aftercrop from this.”

Rita Barata believes CAPES will need to adapt its criteria for the next assessment round, for the period from 2021 to 2024. “The impacts on programs from the lack of funding were compounded by the pandemic, resulting in more students dropping out of master’s and doctoral programs, projects being put on hold, and theses, dissertations, and scientific papers being delayed,” she says. “This all affects program performance and research output in the following years and the agency needs to take this into account.”

She believes CAPES will also need to reexamine the effectiveness of its assessment model. “The four-year assessments are designed to identify nuances and programs demonstrating outstanding performance, helping CAPES, CNPq, and other agencies set investment priorities and design strategies for developing graduate education,” she explains. “But as more programs are upgraded to higher scores, the assessment scale is no longer able to distinguish excellence.”

Alexandre Affonso / Revista Pesquisa FAPESP

The scientific community has proposed a three-dimensional model that assigns different scores across a set of five performance dimensions: education and training, research, international and regional engagement, innovation, and impact on society (see Pesquisa FAPESP issue no. 286). “This has been discussed for several years now and has garnered wide support from the community, but we still don’t know whether it will be implemented,” says Marcio de Castro Silva Filho, associate dean for graduate education at USP. “A new approach could help to improve the system by providing a more accurate and detailed assessment that takes account of programs’ individual strengths,” adds Meneguello.

The last phase of the most recent CAPES assessment was marked by a series of setbacks, with the most significant occurring in September 2021. In response to a public civil action brought by the Federal Prosecution Service (MPF), judge Andrea de Araújo Peixoto, of the 32^nd Federal Court of Rio de Janeiro, issued an injunction suspending the assessment even though it was nearly completed and due to be released in December. The action alleged that CAPES had modified the assessment criteria midway in the four-year assessment process, and that this had created legal insecurity for graduate programs. The prosecutors recommended applying the same criteria in the 2021 assessment as were used in the previous assessment, for the four-year period ended 2016 (see Pesquisa FAPESP issue no. 309).

In December, judge Antonio Henrique Correa da Silva, also from the 32^nd Federal Court of Rio de Janeiro, granted a motion from the Attorney General’s Office to overturn the injunction and allow the assessment to resume, but the results remained withheld from release. Meanwhile, 80 researchers serving as coordinators and consultants in chemistry, mathematics, and physics resigned, citing the agency’s lack of effort to overturn the court’s decision, and pressure to approve new programs.

In September 2022, CAPES settled the public civil action with the MPF to allow the assessment results to be published. Initially, only program coordinators and associate deans for graduate affairs received the scores and could request reconsideration if they were not satisfied with the preliminary results. The settlement, however, drew criticism from the Brazilian Society for the Advancement of Science (SBPC) as it allowed programs to maintain their previous four-year assessment score if they had been downgraded in the current assessment. CAPES has not disclosed which programs took advantage of this provision to avoid being downgraded. In a statement, the agency said it had received 975 requests for reconsideration, with 707 requesting to keep their score from 2017. “Still, we believe the assessment scores accurately reflect individual program strengths,” the agency wrote. In all, only 189 programs received lower scores than in 2017.

The delayed assessment results caused setbacks for program planning. “We went through two years of instability, with successive changes in CAPES management, the stalled assessment process, and the late release of the assessment results. All of this delayed the program planning process,” says Boldrin. “We’re already midway into the next four-year assessment and have only now been able to assess our strengths and weaknesses and set new targets.” The pandemic has also delayed discussions on the National Graduate Education Plan (PNPG) for 2021–2030, which will establish a set of goals and ambitions for the coming years. “This task has been left to CAPES’s new management, now led by biologist Mercedes Bustamante,” says Meneguello from UNICAMP.

CAPES scores diluted the performance of some universities

The University of São Paulo (USP), São Paulo State University (UNESP), and the Federal University of Rio de Janeiro (UFRJ) have the country’s largest number of graduate programs, according to official data compiled from the most recent four-year assessment performed by the Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES) and released in December. In total, 265 USP programs were assessed, slightly more than the sum of UNESP (128) and UFRJ (125) programs.

However, the methodology employed by CAPES to disclose the scores watered down the performance of some universities, namely USP and UNESP, by reproducing the non-standardized way the institutions registered their graduate programs in the assessment process. Those that reported their programs by grouping them together obtained greater relevance in the assessment than those that declared their programs separately, by unit.

Rodrigo Cunha / Revista Pesquisa FAPESP

The assessment divided USP into seven different units, assigning each a corresponding fraction of the university’s programs under assessment. Even so, the institution ranked 1^st on the list, thanks to the 174 programs at its São Paulo campuses. Additionally, CAPES assessed 50 programs at the Ribeirão Preto campus, 17 at the São Carlos campus, 17 at the Luiz de Queiroz College of Agriculture in Piracicaba, 4 at the Lorena School of Engineering, 2 at the Bauru School of Dentistry, and 1 at the Center for Nuclear Energy in Agriculture (CENA), in Piracicaba.

However, in the case of UNESP, this methodology erased its contribution. The university is not even mentioned in the table prepared by CAPES and reproduced in this article, which shows the 30 institutions that stood out most in the graduate program assessment. With campuses in 24 cities within the state of São Paulo, UNESP has graduate programs in 19 different units, which were fragmented in the ranking. The university is first mentioned in 48^th place, in reference to the group of 27 graduate programs assessed at the Botucatu campus. Then, UNESP reappears on the list of institutions with 10 to 19 assessed programs—thanks to the Araraquara and Bauru campuses, which each have 13 programs, the São José do Rio Preto campus with 11 programs, and the Rio Claro campus with 10. And so on, with the campuses in Jaboticabal (9 programs), Ilha Solteira and Presidente Prudente (7 each), Marília (5), Araçatuba, Franca and São Paulo/Dean’s Office (4 each), Assis, Guaratinguetá and São José dos Campos (3 each), Sorocaba (2), São Vicente, Tupã and the Institute of Theoretical Physics in São Paulo (1 program each).

UFRJ, on the other hand, did not face this problem, as it declared all 125 of its programs by grouping them. Among the institutions with many programs, the following are of note: the University of Brasília (UnB) and the Federal University of Rio Grande do Sul (UFRGS), with 90 each, the Federal University of Minas Gerais (UFMG) and the Federal University of Pará (UFPA), with 87 each, the Federal University of Rio Grande do Norte (UFRN), with 81, the Federal University of Pernambuco (UFPE), with 80, the University of Campinas (UNICAMP), with 79, and the Federal Universities of Bahia (UFBA), Santa Catarina (UFSC), Paraná (UFPR), and Fluminense (UFF), with 78 each.

Republish