{"id":421473,"date":"2022-01-19T12:56:28","date_gmt":"2022-01-19T15:56:28","guid":{"rendered":"https:\/\/revistapesquisa.fapesp.br\/?p=421473"},"modified":"2022-01-19T13:59:12","modified_gmt":"2022-01-19T16:59:12","slug":"turbulence-on-the-horizon-for-postgraduate-education","status":"publish","type":"post","link":"https:\/\/revistapesquisa.fapesp.br\/en\/turbulence-on-the-horizon-for-postgraduate-education\/","title":{"rendered":"Turbulence on the horizon for postgraduate education"},"content":{"rendered":"

A four-year assessment of graduate programs in Brazil, used to gage the quality of the country\u2019s 6,500 master\u2019s and doctoral degree programs and to inform the distribution of grants and funding, is being challenged in court for the first time in 45 years since the assessment was first introduced. In response to a public civil action brought by the Federal Prosecution Service (MPF), judge Andrea de Ara\u00fajo Peixoto, of the 32nd<\/sup> Federal Court of Rio de Janeiro, issued an injunction suspending the 2017-2021 assessment even though it was nearly completed and due to be released in December.<\/p>\n

Prosecutors Jess\u00e9 Ambr\u00f3sio dos Santos Jr. and Ant\u00f4nio do Passo Cabral allege that the Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES), a Ministry of Education (MEC) agency responsible for evaluating and ranking graduate programs, had modified the assessment criteria during the course of the four-year assessment process, and that this had created legal insecurity for graduate programs. \u201cThey have retroactively applied new criteria, often established at the end of the assessment period, thus preventing institutions from timely reviewing and adjusting their previous processes and routines to the new criteria by which they would be assessed,\u201d wrote the prosecutors. \u201cThis violates the right of every society to adequate government oversight and fair and impartial distribution of public funding, a right that is eroded by CAPES\u2019s illegal assessment of graduate programs.\u201d<\/p>\n

They recommend applying the same criteria in the 2021 assessment as were used in the previous assessment for the four-year period ended 2016. The public civil action follows a 2018 MPF investigation into the assessment rules used by CAPES. The investigation was launched in response to criticisms from graduate program managers, who not uncommonly complain about the assessment methodology or the scores they have received when the results are released. The Rio de Janeiro court gave CAPES 30 days in which to provide a report on its assessment criteria.<\/p>\n

The suspension of the assessment has left the academic community perplexed. \u201cThis creates significant insecurity,\u201d says Ricardo Pimenta Bertolla, associate dean for graduate student affairs at the Federal University of S\u00e3o Paulo (UNIFESP). \u201cProgram managers had completed lengthy reports over those four years and most of them had developed, invested in, and perfected their programs to achieve better scores. The graduate community is now very familiar with the assessment workflow. The suspension sends all this effort down the drain.\u201d Maria Valnice Boldrin, associate dean for graduate student affairs at S\u00e3o Paulo State University (UNESP), notes that in 2020, 24 of her institution\u2019s programs were merged into 12 new programs. The goal in combining these master\u2019s and doctoral programs was to provide more robust and comprehensive training to students. \u201cWe were looking forward to seeing how these and our other 116 programs would perform, and the assessment would have been an opportunity to review our targets and develop more effective policies.\u201d<\/p>\n

CAPES has been responsible for monitoring graduate program quality in Brazil since 1976, and for more than two decades has used a model in which program managers periodically complete a questionnaire with information on a range of topics. The questionnaire responses are first reviewed by subject-matter committees across 49 areas of knowledge, which assess program performance and recommend scores. CAPES\u2019s Technical and Scientific Council for Higher Education (CTC), a body of subject-matter coordinators, then reviews the assessment results and assigns final scores.<\/p>\n<\/div>

\n \n \n \n <\/picture>Alexandre Affonso<\/span><\/div>
\n

The court injunction creates an impasse, as the system which the prosecutors have found fault in is, conversely, considered sound by the scientific community. Carlos Gilberto Carlotti J\u00fanior, associate dean for graduate student affairs at the University of S\u00e3o Paulo (USP), says the claim that the assessment criteria are set retroactively is misleading. \u201cThe assessment has three areas of focus: program structure, faculty research, and student research. This never changes,\u201d he explains. Midway into the four-year period, a mid-term meeting is held to assess progress on data collection and approve the final documentation with the firmed-up assessment criteria.<\/p>\n

The only thing that is calibrated at the end of the four-year period is the weight assigned to student and faculty research. For programs in which research is measured in terms of papers, these scores are based on a journal rating table, called Qualis Peri\u00f3dicos<\/em>, which in turn is based on journal citation data. The weights assigned to research papers can be adjusted by committees formed of subject-matter coordinators, depending on the data collected on research output. \u201cThis is left to the end because the assessment is also for benchmarking purposes and shows how different programs compared to each other in 2021,\u201d explains Carlotti. \u201cWe first need to know a little about the body of research during the four-year period so we can design metrics that take the nature of that research into account.\u201d The risk in using the same criteria as in the previous assessment is that many programs will have evolved and the assessment would be unable to identify nuances to determine what excellence looks like. Programs are scored on a scale of 3 to 7. \u201cIt also wouldn\u2019t make sense to use data from the previous assessment because journal impact factors, which are an input into the journal rating table, change every year. We would risk overrating a journal whose reputation has diminished, or underrating an important research paper.\u201d<\/p>\n

For Renato Janine Ribeiro, chairman of the Brazilian Society for the Advancement of Science (SBPC) and formally assessment director at CAPES, taking the matter to court creates a delicate problem, which is a judge deciding on a matter she may not be in a position to fully understand and properly assess. \u201cIt\u2019s like a judge challenging the result of a competitive process to select candidates for a tenured position as professor of Greek. It is highly unlikely that a judge will know more about the requisite skills than the assessment panel,\u201d he says. Ribeiro believes the turbulence caused by the suspension could adversely affect future research effort, as graduate programs use the assessment to plan their future.<\/p>\n

\u201cBrazil\u2019s science output is umbilically linked to its graduate education system,\u201d says neuroscientist Luiz Eug\u00eanio Mello, scientific director at FAPESP. \u201cHalting the CAPES assessment has left Brazil\u2019s research community\u2014an important lever for the country\u2019s development\u2014without a compass.\u201d Helena Nader, deputy chair of the Brazilian Academy of Science (ABC), believes graduate program assessments have played a key role in driving excellence in graduate education, where universities do research at the highest level and where more than 24,000 PhDs are trained each year. \u201cThe criteria are defined by researchers and discussed exhaustively. It\u2019s to be expected that some programs will be unhappy with their scores or feel they have been unfair. Complaining while suggesting improvements is one thing. Suspending the assessment is quite another. It erodes one of the pillars of Brazil\u2019s science enterprise.\u201d<\/p>\n

Asked to comment on the suspension, CAPES President Cl\u00e1udia Mansani Queda de Toledo said in a statement that she is seeking solutions to continue the assessment. She also defended the assessment process, noting that it involves more than 1,500 researchers across 49 subject-matter committees. \u201cI would like to publicly thank the council members and appraisers for their efforts over the years at each stage of the assessment, and for the sense of civic responsibility that each of them has demonstrated.\u201d<\/p>\n

Complaining while suggesting improvements is one thing. Suspending the assessment is quite another. It erodes one of the pillars of Brazil\u2019s scientific community,\u201d says Helena Nader of ABC<\/p><\/blockquote>\n

The suspension has come at a turbulent time for CAPES. In the days preceding the court decision, all members of the Technical and Scientific Counsel (CTC), a decision-making body on graduate assessments, were removed. The decision was made on the argument that the Council had two more subject-matter committee representatives than the 18 established in the agency\u2019s bylaws. There were fears that this could affect the assessment process, but the following week an election was held that reinstated 18 of the original members\u2014two members voluntarily resigned to adjust the Council composition for compliance with the bylaws.<\/p>\n

But there had already been tensions between the CTC and agency directors even before the Council was dissolved. On September 10, CAPES\u2019s president published final rules on journal ranking for the 2021 assessment, adding flexibility for subject-matter coordinators to modulate the weights assigned to different scientific journals, against the advice of the CTC. Other controversial provisions were also added, such as the possibility for some fields to choose between the journal-ranking rules for 2021 or 2016. The rules were also criticized for neglecting certain aspects that some hoped would be revisited, but were left unchanged. \u201cThe assessment is still based on journal impact and not on the actual citations received by each paper produced by graduate students and faculty,\u201d says Abel Packer, who manages the Brazilian open-access journal library SciELO Brasil. \u201cBut what was most disappointing was that the rules shut the door on open science by excluding journals using open peer review, such as the British Medical Journal, <\/em>as these journals are deemed to violate good publishing practices. This appears to extend to the growing number of journals that accept non-peer-reviewed preprints on preprint servers.\u201d<\/p>\n

Boldrin, of UNESP, notes that there\u2019s more to the tensions between CAPES and the broader scientific community than just the impasse over the assessment process. \u201cOver the previous four years, we\u2019ve already had four different presidents at CAPES and this has created uncertainty around core policies,\u201d she says. She cites as an example the Institutional Program to Internationalize Brazilian Universities (PrInt), launched by CAPES in 2017 to establish international partnerships for master\u2019s and doctoral degree programs. The agency selected 36 institutions to participate in the PrInt program and receive federal funding, including UNESP. \u201cIn 2020 no mobility grants were awarded due to the pandemic, and now, in 2021, the program has been suspended. We have reached the end of the year and the funds have yet to be disbursed,\u201d she says. \u201cIt\u2019s sad because this initiative had been carefully planned and promised to more effectively internationalize our graduate programs and foster collaborations among research groups across 70 networks of local and international researchers. We hope everything normalizes after the November PrInt assessment.\u201d For Boldrin, the suspended assessment was just the tip of the iceberg. \u201cThe MPF investigation had been ongoing since 2018, and only reached this impasse now. It appears that the graduate education system\u2019s currently weakened position, with declining investments and the dissolution of the CTC, opened a flank for the lawsuit.\u201d<\/p>\n

Carlotti, of USP, believes the court-ordered suspension has added additional fuel to a fire that is undermining Brazil\u2019s wider graduate education system. \u201cOur system is grounded on three pillars, and all three have now been weakened,\u201d he says. The first pillar, says Carlotti, is funding for grants, which has been shrunk by successive budget cuts, especially at federal agencies. The second is the 10-year National Graduate Education Plan, which provides a compass to guide the development of graduate programs. \u201cThe most recent plan expired in 2020, and a new one is yet to be discussed and developed,\u201d he says. The third pillar is the program ranking system, which has now been challenged. \u201cThe program assessment gives students assurance about which programs have the highest quality. Without a national plan, without adequate funding and with the program assessment being challenged, the entire system is impaired.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"Uncertainties arise as assessment of Brazilian graduate programs goes to court","protected":false},"author":11,"featured_media":420757,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[166],"tags":[226,234,256],"coauthors":[98],"class_list":["post-421473","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-policies-st-en","tag-education","tag-finance","tag-public-policies","position_at_home-sumario"],"acf":[],"_links":{"self":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts\/421473","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/comments?post=421473"}],"version-history":[{"count":2,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts\/421473\/revisions"}],"predecessor-version":[{"id":421717,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts\/421473\/revisions\/421717"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/media\/420757"}],"wp:attachment":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/media?parent=421473"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/categories?post=421473"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/tags?post=421473"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/coauthors?post=421473"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}