A four-year assessment of graduate programs in Brazil, used to gage the quality of the country’s 6,500 master’s and doctoral degree programs and to inform the distribution of grants and funding, is being challenged in court for the first time in 45 years since the assessment was first introduced. In response to a public civil action brought by the Federal Prosecution Service (MPF), judge Andrea de Araújo Peixoto, of the 32nd Federal Court of Rio de Janeiro, issued an injunction suspending the 2017-2021 assessment even though it was nearly completed and due to be released in December.
Prosecutors Jessé Ambrósio dos Santos Jr. and Antônio do Passo Cabral allege that the Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES), a Ministry of Education (MEC) agency responsible for evaluating and ranking graduate programs, had modified the assessment criteria during the course of the four-year assessment process, and that this had created legal insecurity for graduate programs. “They have retroactively applied new criteria, often established at the end of the assessment period, thus preventing institutions from timely reviewing and adjusting their previous processes and routines to the new criteria by which they would be assessed,” wrote the prosecutors. “This violates the right of every society to adequate government oversight and fair and impartial distribution of public funding, a right that is eroded by CAPES’s illegal assessment of graduate programs.”
They recommend applying the same criteria in the 2021 assessment as were used in the previous assessment for the four-year period ended 2016. The public civil action follows a 2018 MPF investigation into the assessment rules used by CAPES. The investigation was launched in response to criticisms from graduate program managers, who not uncommonly complain about the assessment methodology or the scores they have received when the results are released. The Rio de Janeiro court gave CAPES 30 days in which to provide a report on its assessment criteria.
The suspension of the assessment has left the academic community perplexed. “This creates significant insecurity,” says Ricardo Pimenta Bertolla, associate dean for graduate student affairs at the Federal University of São Paulo (UNIFESP). “Program managers had completed lengthy reports over those four years and most of them had developed, invested in, and perfected their programs to achieve better scores. The graduate community is now very familiar with the assessment workflow. The suspension sends all this effort down the drain.” Maria Valnice Boldrin, associate dean for graduate student affairs at São Paulo State University (UNESP), notes that in 2020, 24 of her institution’s programs were merged into 12 new programs. The goal in combining these master’s and doctoral programs was to provide more robust and comprehensive training to students. “We were looking forward to seeing how these and our other 116 programs would perform, and the assessment would have been an opportunity to review our targets and develop more effective policies.”
CAPES has been responsible for monitoring graduate program quality in Brazil since 1976, and for more than two decades has used a model in which program managers periodically complete a questionnaire with information on a range of topics. The questionnaire responses are first reviewed by subject-matter committees across 49 areas of knowledge, which assess program performance and recommend scores. CAPES’s Technical and Scientific Council for Higher Education (CTC), a body of subject-matter coordinators, then reviews the assessment results and assigns final scores.
The court injunction creates an impasse, as the system which the prosecutors have found fault in is, conversely, considered sound by the scientific community. Carlos Gilberto Carlotti Júnior, associate dean for graduate student affairs at the University of São Paulo (USP), says the claim that the assessment criteria are set retroactively is misleading. “The assessment has three areas of focus: program structure, faculty research, and student research. This never changes,” he explains. Midway into the four-year period, a mid-term meeting is held to assess progress on data collection and approve the final documentation with the firmed-up assessment criteria.
The only thing that is calibrated at the end of the four-year period is the weight assigned to student and faculty research. For programs in which research is measured in terms of papers, these scores are based on a journal rating table, called Qualis Periódicos, which in turn is based on journal citation data. The weights assigned to research papers can be adjusted by committees formed of subject-matter coordinators, depending on the data collected on research output. “This is left to the end because the assessment is also for benchmarking purposes and shows how different programs compared to each other in 2021,” explains Carlotti. “We first need to know a little about the body of research during the four-year period so we can design metrics that take the nature of that research into account.” The risk in using the same criteria as in the previous assessment is that many programs will have evolved and the assessment would be unable to identify nuances to determine what excellence looks like. Programs are scored on a scale of 3 to 7. “It also wouldn’t make sense to use data from the previous assessment because journal impact factors, which are an input into the journal rating table, change every year. We would risk overrating a journal whose reputation has diminished, or underrating an important research paper.”
For Renato Janine Ribeiro, chairman of the Brazilian Society for the Advancement of Science (SBPC) and formally assessment director at CAPES, taking the matter to court creates a delicate problem, which is a judge deciding on a matter she may not be in a position to fully understand and properly assess. “It’s like a judge challenging the result of a competitive process to select candidates for a tenured position as professor of Greek. It is highly unlikely that a judge will know more about the requisite skills than the assessment panel,” he says. Ribeiro believes the turbulence caused by the suspension could adversely affect future research effort, as graduate programs use the assessment to plan their future.
“Brazil’s science output is umbilically linked to its graduate education system,” says neuroscientist Luiz Eugênio Mello, scientific director at FAPESP. “Halting the CAPES assessment has left Brazil’s research community—an important lever for the country’s development—without a compass.” Helena Nader, deputy chair of the Brazilian Academy of Science (ABC), believes graduate program assessments have played a key role in driving excellence in graduate education, where universities do research at the highest level and where more than 24,000 PhDs are trained each year. “The criteria are defined by researchers and discussed exhaustively. It’s to be expected that some programs will be unhappy with their scores or feel they have been unfair. Complaining while suggesting improvements is one thing. Suspending the assessment is quite another. It erodes one of the pillars of Brazil’s science enterprise.”
Asked to comment on the suspension, CAPES President Cláudia Mansani Queda de Toledo said in a statement that she is seeking solutions to continue the assessment. She also defended the assessment process, noting that it involves more than 1,500 researchers across 49 subject-matter committees. “I would like to publicly thank the council members and appraisers for their efforts over the years at each stage of the assessment, and for the sense of civic responsibility that each of them has demonstrated.”
Complaining while suggesting improvements is one thing. Suspending the assessment is quite another. It erodes one of the pillars of Brazil’s scientific community,” says Helena Nader of ABC
The suspension has come at a turbulent time for CAPES. In the days preceding the court decision, all members of the Technical and Scientific Counsel (CTC), a decision-making body on graduate assessments, were removed. The decision was made on the argument that the Council had two more subject-matter committee representatives than the 18 established in the agency’s bylaws. There were fears that this could affect the assessment process, but the following week an election was held that reinstated 18 of the original members—two members voluntarily resigned to adjust the Council composition for compliance with the bylaws.
But there had already been tensions between the CTC and agency directors even before the Council was dissolved. On September 10, CAPES’s president published final rules on journal ranking for the 2021 assessment, adding flexibility for subject-matter coordinators to modulate the weights assigned to different scientific journals, against the advice of the CTC. Other controversial provisions were also added, such as the possibility for some fields to choose between the journal-ranking rules for 2021 or 2016. The rules were also criticized for neglecting certain aspects that some hoped would be revisited, but were left unchanged. “The assessment is still based on journal impact and not on the actual citations received by each paper produced by graduate students and faculty,” says Abel Packer, who manages the Brazilian open-access journal library SciELO Brasil. “But what was most disappointing was that the rules shut the door on open science by excluding journals using open peer review, such as the British Medical Journal, as these journals are deemed to violate good publishing practices. This appears to extend to the growing number of journals that accept non-peer-reviewed preprints on preprint servers.”
Boldrin, of UNESP, notes that there’s more to the tensions between CAPES and the broader scientific community than just the impasse over the assessment process. “Over the previous four years, we’ve already had four different presidents at CAPES and this has created uncertainty around core policies,” she says. She cites as an example the Institutional Program to Internationalize Brazilian Universities (PrInt), launched by CAPES in 2017 to establish international partnerships for master’s and doctoral degree programs. The agency selected 36 institutions to participate in the PrInt program and receive federal funding, including UNESP. “In 2020 no mobility grants were awarded due to the pandemic, and now, in 2021, the program has been suspended. We have reached the end of the year and the funds have yet to be disbursed,” she says. “It’s sad because this initiative had been carefully planned and promised to more effectively internationalize our graduate programs and foster collaborations among research groups across 70 networks of local and international researchers. We hope everything normalizes after the November PrInt assessment.” For Boldrin, the suspended assessment was just the tip of the iceberg. “The MPF investigation had been ongoing since 2018, and only reached this impasse now. It appears that the graduate education system’s currently weakened position, with declining investments and the dissolution of the CTC, opened a flank for the lawsuit.”
Carlotti, of USP, believes the court-ordered suspension has added additional fuel to a fire that is undermining Brazil’s wider graduate education system. “Our system is grounded on three pillars, and all three have now been weakened,” he says. The first pillar, says Carlotti, is funding for grants, which has been shrunk by successive budget cuts, especially at federal agencies. The second is the 10-year National Graduate Education Plan, which provides a compass to guide the development of graduate programs. “The most recent plan expired in 2020, and a new one is yet to be discussed and developed,” he says. The third pillar is the program ranking system, which has now been challenged. “The program assessment gives students assurance about which programs have the highest quality. Without a national plan, without adequate funding and with the program assessment being challenged, the entire system is impaired.”
Republish
