{"id":478106,"date":"2023-05-19T15:25:58","date_gmt":"2023-05-19T18:25:58","guid":{"rendered":"https:\/\/revistapesquisa.fapesp.br\/?p=478106"},"modified":"2024-06-05T16:27:50","modified_gmt":"2024-06-05T19:27:50","slug":"capes-assessment-finds-that-quality-indicators-for-masters-and-doctoral-courses-in-brazil-have-improved","status":"publish","type":"post","link":"https:\/\/revistapesquisa.fapesp.br\/en\/capes-assessment-finds-that-quality-indicators-for-masters-and-doctoral-courses-in-brazil-have-improved\/","title":{"rendered":"CAPES assessment finds that quality indicators for master’s and doctoral courses in Brazil have improved"},"content":{"rendered":"

The Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES), linked to the Ministry of Education (MEC), released the results from its four-year assessment of graduate programs last December, a year overdue\u2014the CAPES assessment has been used since the 1970s to measure the quality of master’s and doctoral programs and to inform the distribution of research grants and fellowships. Despite facing significant funding cuts and eroded fellowship awards over the past few years, many programs showed an improvement in their performance indicators. Out of the 4,512 programs assessed from 2017 to 2020, 34% improved their assessment scores. The number of programs rated as world class\u2014those receiving the highest assessment scores of 6 and 7\u2014rose by 37%, from 490 to 671.<\/p>\n

The majority of these programs are located in Brazil\u2019s South and Southeast states, especially S\u00e3o Paulo, Rio de Janeiro, Rio Grande do Sul, and Minas Gerais. But other regions have been catching up. In the Northeast, for instance, the number of world-class programs rose from 37 to 60, a gain of 62%. The North went from having no programs with a score of 7 in the previous assessment to three programs in the current assessment\u2014two at the Federal University of Par\u00e1 (UFPA) and one at the Brazilian Institute for Amazon Research (INPA). \u201cWe were expecting our scores to improve, but they exceeded our expectations and this will greatly enhance our ability to get funding for our graduate programs,\u201d said psychologist Emmanuel Zagury Tourinho, dean of the university, in a video posted on UFPA\u2019s social media.<\/p>\n

The current edition\u2019s results are consistent with trends seen in previous assessments. The University of S\u00e3o Paulo (USP) remains the institution with the highest number of programs rated 6 and 7\u2014a total of 114, largely in the fields of life sciences, engineering, and medicine. The Federal University of ABC (UFABC), founded just 17 years ago, also celebrated a strong performance in the current assessment. Of the 29 programs evaluated, 10 improved their scores, with two programs receiving a score of 6\u2014nanomaterials and advanced chemistry and science and technology. This was \u201ca notable achievement for a relatively young institution,\u201d says Charles Morphy, the university\u2019s associate dean for graduate student affairs. Meanwhile, S\u00e3o Paulo State University\u2019s (UNESP) graduate department had 10 programs with a top score of 7, including programs in physics, chemistry, and dentistry. \u201cThis reflects our practice of setting targets and tracking program performance against those targets over each four-year period,\u201d says chemist Maria Valnice Boldrin, UNESP\u2019s associate dean for graduate affairs.<\/p>\n

At the University of Campinas (UNICAMP), 30 programs received higher scores than in the previous cycle, with 37 receiving world-class ratings, including in fields such as the humanities, food science, engineering, medicine, and dentistry. The office of the associate dean for graduate student affairs said in a statement that they will work with the coordinators of programs scoring 3 and 4 to create strategies for improvement in the next assessment.<\/p>\n<\/div>

\n \n \n \n <\/picture>Alexandre Affonso \/ Revista Pesquisa FAPESP<\/span><\/div>
\n

Highly ranked programs not only gain academic prestige but also enjoy greater independence and are eligible for funding directly from CAPES via the Academic Excellence Program (ProEx). Although programs with scores of 3 to 5 are also eligible for funding from the Graduate Support Program (ProAP), the awards are smaller. In addition to funding, high-scoring programs have a better chance of having their research projects selected in calls for proposals organized by CAPES and the Brazilian National Council for Scientific and Technological Development (CNPq).<\/p>\n

Inevitably, the funding constraints for grants and projects in recent years have affected well and poorly ranked programs alike. \u201cGraduate programs have had to navigate a number of hurdles caused by the pandemic and the limited funding for science and education,\u201d says political scientist Rachel Meneguello, associate dean for graduate student affairs at UNICAMP. \u201cInsufficient fellowship awards, eroded by inflation, are one example.\u201d Fellowship awards have not been adjusted for inflation since 2013, while the General Market Price Index (IGP-M) has increased by 125.1% since then. Currently, master’s fellows receive R$1,500, and doctoral fellows receive R$2,200.<\/p>\n

Initiatives such as the Institutional Program to Internationalize Brazilian Universities (PrInt)\u2014launched by CAPES in 2017 to establish international collaborations for master\u2019s and doctoral degree programs\u2014have also been affected. Student mobility grants were not awarded in 2020, and the program was suspended entirely in 2021. Boldrin explains that the program, which was slated to run until 2021, has now been extended to 2024, and grants that were meant to be paid in 2020 and 2021 have been rescheduled for 2023 and 2024.<\/p>\n

The reasons for the strong performance of highly ranked programs are not clear, but several hypotheses have been suggested. One possibility is that the CAPES assessment approach has been modified to focus more on qualitative aspects such as the quality of training provided and the intellectual property generated by a program. Previously, program managers completed a questionnaire that provided information about the program\u2019s approach, faculty qualifications, student profiles, and the number of papers published in the highest tiers of Qualis, a journal ranking system used by CAPES. \u201cBut in the most recent assessment, programs were also asked to name their students\u2019 and researchers\u2019 top papers for consideration,\u201d notes epidemiology researcher Rita Barradas Barata, a former director at CAPES who participated in the assessment of graduate programs in the field of public health.<\/p>\n

Assessed programs reported 4.7 million papers authored from 2017 to 2020. Of these, 280,000 were submitted by program coordinators and assessed on impact, originality, and innovation. \u201cThis may have helped programs perform better than they would have if the assessment had focused on quantitative aspects only,\u201d says Barata.<\/p>\n\n \n \n \n <\/picture>Alexandre Affonso \/ Revista Pesquisa FAPESP<\/span>\n

Experts interviewed for this article said there has been a trend within CAPES\u2019s Technical and Scientific Council for Higher Education (CTC), the highest decision-making body on graduate assessments, to give subject-matter coordinators greater autonomy in recommending scores for programs. In previous years\u2019 assessments, CAPES created a special committee to review any scores of 6 and 7.<\/p>\n

It is also possible that programs\u2019 quality metrics have improved as a result of investments made during the previous four-year period, and which were not affected by recent constraints. \u201cA program\u2019s performance in each assessment is often the result of past efforts,\u201d says Morphy of UFABC. \u201cThe first half of the 2010s was marked by a policy that valued and worked to expand graduate education, and it is possible that current results are an aftercrop from this.\u201d<\/p>\n

Rita Barata believes CAPES will need to adapt its criteria for the next assessment round, for the period from 2021 to 2024. \u201cThe impacts on programs from the lack of funding were compounded by the pandemic, resulting in more students dropping out of master\u2019s and doctoral programs, projects being put on hold, and theses, dissertations, and scientific papers being delayed,\u201d she says. \u201cThis all affects program performance and research output in the following years and the agency needs to take this into account.\u201d<\/p>\n

She believes CAPES will also need to reexamine the effectiveness of its assessment model. \u201cThe four-year assessments are designed to identify nuances and programs demonstrating outstanding performance, helping CAPES, CNPq, and other agencies set investment priorities and design strategies for developing graduate education,\u201d she explains. \u201cBut as more programs are upgraded to higher scores, the assessment scale is no longer able to distinguish excellence.\u201d<\/p>\n\n \n \n \n <\/picture>Alexandre Affonso \/ Revista Pesquisa FAPESP<\/span>\n

The scientific community has proposed a three-dimensional model that assigns different scores across a set of five performance dimensions: education and training, research, international and regional engagement, innovation, and impact on society (see<\/em> Pesquisa FAPESP issue no. 286<\/em><\/a>). \u201cThis has been discussed for several years now and has garnered wide support from the community, but we still don\u2019t know whether it will be implemented,\u201d says Marcio de Castro Silva Filho, associate dean for graduate education at USP. \u201cA new approach could help to improve the system by providing a more accurate and detailed assessment that takes account of programs\u2019 individual strengths,\u201d adds Meneguello.<\/p>\n

The last phase of the most recent CAPES assessment was marked by a series of setbacks, with the most significant occurring in September 2021. In response to a public civil action brought by the Federal Prosecution Service (MPF), judge Andrea de Ara\u00fajo Peixoto, of the 32nd<\/sup> Federal Court of Rio de Janeiro, issued an injunction suspending the assessment even though it was nearly completed and due to be released in December. The action alleged that CAPES had modified the assessment criteria midway in the four-year assessment process, and that this had created legal insecurity for graduate programs. The prosecutors recommended applying the same criteria in the 2021 assessment as were used in the previous assessment, for the four-year period ended 2016 (see<\/em> Pesquisa FAPESP issue no. 309<\/em><\/a>).<\/p>\n

In December, judge Antonio Henrique Correa da Silva, also from the 32nd<\/sup> Federal Court of Rio de Janeiro, granted a motion from the Attorney General\u2019s Office to overturn the injunction and allow the assessment to resume, but the results remained withheld from release. Meanwhile, 80 researchers serving as coordinators and consultants in chemistry, mathematics, and physics resigned, citing the agency\u2019s lack of effort to overturn the court\u2019s decision, and pressure to approve new programs.<\/p>\n

In September 2022, CAPES settled the public civil action with the MPF to allow the assessment results to be published. Initially, only program coordinators and associate deans for graduate affairs received the scores and could request reconsideration if they were not satisfied with the preliminary results. The settlement, however, drew criticism from the Brazilian Society for the Advancement of Science (SBPC) as it allowed programs to maintain their previous four-year assessment score if they had been downgraded in the current assessment. CAPES has not disclosed which programs took advantage of this provision to avoid being downgraded. In a statement, the agency said it had received 975 requests for reconsideration, with 707 requesting to keep their score from 2017. \u201cStill, we believe the assessment scores accurately reflect individual program strengths,\u201d the agency wrote. In all, only 189 programs received lower scores than in 2017.<\/p>\n

The delayed assessment results caused setbacks for program planning. \u201cWe went through two years of instability, with successive changes in CAPES management, the stalled assessment process, and the late release of the assessment results. All of this delayed the program planning process,\u201d says Boldrin. \u201cWe\u2019re already midway into the next four-year assessment and have only now been able to assess our strengths and weaknesses and set new targets.\u201d The pandemic has also delayed discussions on the National Graduate Education Plan (PNPG) for 2021\u20132030, which will establish a set of goals and ambitions for the coming years. \u201cThis task has been left to CAPES\u2019s new management, now led by biologist Mercedes Bustamante,\u201d says Meneguello from UNICAMP.<\/p>\n

CAPES scores diluted the performance of some universities<\/strong><\/p>\n