{"id":332994,"date":"2020-02-19T16:57:01","date_gmt":"2020-02-19T19:57:01","guid":{"rendered":"https:\/\/revistapesquisa.fapesp.br\/?p=332994"},"modified":"2020-03-18T19:01:26","modified_gmt":"2020-03-18T22:01:26","slug":"5-dimensional-benchmarking","status":"publish","type":"post","link":"https:\/\/revistapesquisa.fapesp.br\/en\/5-dimensional-benchmarking\/","title":{"rendered":"5-dimensional benchmarking"},"content":{"rendered":"

The framework for benchmarking graduate programs in Brazil will see its most extensive overhaul in more than two decades. The Brazilian Federal Agency for Support and Evaluation of Graduate Education (CAPES), a Ministry of Education agency responsible for assessing the quality of master\u2019s and doctoral programs, has announced that the country\u2019s more than 4,000 active programs will no longer be benchmarked using a single composite score\u2014currently on a scale of 3 to 7 for recommended and recognized programs\u2014but will now be assessed across five different dimensions: Education and Training, Research, International and Regional Engagement, Innovation, and Societal Impact. The details of the new framework will be fleshed out over the coming months by experts from the academic community and the technical department at CAPES. The changes will be effective for the assessment cycle beginning in 2021, and the results will be reported in 2025.<\/p>\n

The goal is to provide more meaningful and comprehensive insight into program quality across different aspects, revealing strengths and weaknesses. CAPES chairman Anderson Ribeiro Correia says the current model, with its focus on research and education metrics, has played an important role in expanding graduate education in Brazil since the 1960s, but has become too narrow to accommodate the growing diversity of programs. \u201cWe\u2019ve created high-quality graduate programs that have different strengths. But we\u2019ve continued to measure their quality on the basis of scholarly publishing only,\u201d he says. Correia notes that the system in place gives little weight, for example, to how university training benefits the lives of alumni, or the development of local communities. \u201cThere are programs that give a significant boost to students\u2019 careers, supporting better pay and social mobility. Other programs have a notable impact on local economies, even if they are not quite at the state of the art. By recognizing and encouraging these contributions, we can amplify the impact that graduate education has on Brazil\u2019s development.\u201d<\/p>\n

The idea of radically changing the current model has been two years in the making. After announcing results for the 2013\u20132016 assessment cycle<\/a>, CAPES managers commissioned a study from the institution’s Technical-Scientific Board on aspects of the system that needed improvement. Experts and organizations in the field of graduate education were consulted and offered two key insights: the importance of universities self-assessing their programs\u2014describing their initial expectations and whether they have been met; and the need to implement metrics that reflect the different missions that graduate programs can pursue other than high-quality research, such as supporting regional development and collaborating with industry.<\/p>\n

In late 2018, CAPES sent a delegation to Germany and the Netherlands, including SBPC\u2019s honorary chairwoman, Helena Nader, and the agency\u2019s Assessment Director, Sonia Ba\u00f3, to learn more about U-Multirank, a university benchmarking tool that seemed to align with their aspirations for change in Brazil\u2019s graduate education model. U-Multirank measures universities\u2019 strengths and weaknesses across five different dimensions: teaching and learning, research, knowledge transfer, international orientation, and regional engagement. The results are displayed in a five-color sunburst chart<\/a> representing the five assessment dimensions, with bars showing the level of performance for each indicator.<\/p>\n

The framework was created in 2014 by the Center for Higher Education Development in G\u00fctersloh, Germany, and the Center for Higher Education Policy Studies at Universiteit Twente, the Netherlands, as part of a European Commission initiative to develop an approach that provides a better measure of European universities\u2019 individual strengths than available international rankings. It is currently used by universities in 96 countries, including Brazil, where it has been adopted by the three state universities in S\u00e3o Paulo<\/a>. Brazil\u2019s new graduate benchmarking model will likely draw inspiration from U-Multirank, but will be adapted to CAPES\u2019 evaluation goals. \u201cU-Multirank was developed to provide a comprehensive assessment of universities and has many indicators that are specifically for undergraduate education. Purpose-appropriate indicators will need to be developed to measure the quality of graduate programs,\u201d says Ab\u00edlio Baeta Neves, who served as chairman at CAPES between 2016 and 2018, when plans were first made for the new framework.<\/p>\n

The changes announced in the CAPES assessment model have been well received by the academic community. Zoologist Telma Berchielli, associate dean for graduate student affairs at S\u00e3o Paulo State University (UNESP), says that an approach that recognizes all the different facets of graduate education has been well overdue. \u201cThe previous approach was extremely rigid. It seemed primarily concerned with people being trained to work at universities, whereas today many PhDs go on to work in industry. They contribute to society just the same, but in a different way,\u201d she says. UNESP has a broad offering of programs, she notes\u2014some very well ranked, and others that have yet to reach maturity, but all generating regional impact through the universities\u2019 network of campuses in 24 cities. She cites as examples the animal biology program at the university\u2019s campus in S\u00e3o Jos\u00e9 do Rio Preto, which is less than 10 years old and yet earned the top score (7) in the most recent edition of the CAPES assessment; and the coastal biodiversity program offered at the campus in the seaside city of S\u00e3o Vicente, which received a more modest score of 4. \u201cBut both provide local impact by creating knowledge and training specialists, and this has not been adequately recognized.\u201d Berchielli especially welcomes the introduction of self-assessments in the new model. \u201cThis exercise will show what each program\u2019s strengths are, what the university\u2019s mission is, and where its best research is focused.\u201d<\/p>\n

<\/a>Statistician Nancy Lopes Garcia, associate dean for graduate student affairs at the University of Campinas (UNICAMP), agrees that the evaluation model needs an overhaul, but has a number of concerns. \u201cToo much focus could be placed on the economic and social impact of a program. How would this dimension apply to basic science programs, which can take years to develop applications, but are essential in making those applications possible? How can one measure the economic and social contribution of a program, for example, in mathematics?\u201d she wonders. She believes UNICAMP will not be significantly affected by the changes, however. \u201cWe have consistently invested in the quality of our programs, which for the most part are very solid and internationally competitive. Just under half of our students are in graduate programs, the highest proportion among universities in S\u00e3o Paulo.\u201d<\/p>\n\n \n \n \n <\/picture>\n

While the impact from the new guidelines is difficult to predict before the details are known, Baeta Neves believes the potential implications could be substantial. \u201cIt won\u2019t be a simple matter as it will upset a deeply rooted assessment culture that has influenced the design of current programs. They will have to orient themselves without the composite scoring system they use today, and this is uncharted territory for them,\u201d he explains. The CAPES assessment, which is used as a basis for awarding grants and funding, has shaped the functioning of Brazil\u2019s graduate education system over time. Before 1996, programs were ranked using alphabetical ratings from A to E. In subsequent years, a score of 1 to 5 was assigned to programs offering only master\u2019s degrees, and up to 7 for programs including doctoral degrees. In 1998, differential scores of either 6 or 7 were introduced to reflect aspects such as international engagement and faculty experience abroad. By then, internationalization had come to be recognized as a key lever in research\u2014studies had shown that research in collaboration with foreign institutions had a four times greater citation impact than research done locally.<\/p>\n

In the past 20 years, the number of programs has almost tripled and society has become increasingly demanding in terms of the impacts science is expected to produce, but the assessment system has not kept up with the times. Baeta Neves says the future model could be highly transformational if it abandons what he sees as an excess of normative requirements in the current model. \u201cUniversities, especially the most prestigious ones, need to be allowed more leeway to experiment. The current system is highly controlling and imposes requirements that are entirely unrelated to excellence. It requires, for example, a minimum number of professors in each program. But why would having 12 faculty be better than having, say, 10 or 9?\u201d<\/p>\n

Brazil\u2019s graduate assessment system is globally unique. In the US, program accreditation and assessment is done in a decentralized manner by scientific associations. The most recent countrywide assessment of doctoral programs in the US was carried out in 2010. The report, published by the National Academy of Sciences, examined data for the 2005\u20132006 academic year from 5,000 programs in 62 fields at 212 institutions. Compared with data from the mid-1990s, the numbers of doctoral students had increased by 4% in engineering and 9% in the physical sciences, but declined by 5% in the social sciences and 12% in the humanities. Interestingly, the report found that doctoral education in the US was dominated by programs in public universities\u201472% of the doctoral programs in the study were in public universities. Of the 37 universities that produced the most PhDs from 2002\u20132006, only 12 were private universities.<\/p>\n

The Brazilian assessment framework is similar in scale to the system used to monitor research quality at universities in the UK, where assessments are conducted every five years to determine the allocation of research funding for the following period<\/a>. \u201cUK assessments are on the basis of research only as there are no graduate programs proper to be evaluated as in Brazil. To do doctoral research, students approach potential supervisors with a proposed project. If accepted, their research will count toward a PhD degree,\u201d explains Baeta Neves.<\/p>\n

Not only will the CAPES model see radical changes starting in 2021, but the current assessment process (for the period 2017\u20132020) has also been reformulated so that it values the quality of research over quantity. The assessment form now has three items instead of five, and different weights. For example, it now recognizes the scientific output of alumni, and not only faculty. \u201cUntil the previous edition, indicators related to the student body were more quantitative in nature\u2014such as the number of students enrolled and the average time taken to earn degrees. The assessment now measures students\u2019 scholarly output within five years of completing their master\u2019s or doctoral degrees. While citations will only be captured in the short term, this is progress nonetheless,\u201d says Nancy Garcia of UNICAMP.<\/p>\n

Qualis, the system CAPES uses to benchmark the quality of research produced by graduate programs, is also being reformulated. Science journals are rated on their impact and prestige in their respective fields, and programs that publish in the most prestigious journals score additional points in the assessment. The goal in the reformulation is to make the Qualis system less subjective. Whereas previously a journal could be rated high in one field but low in another, in the following assessment cycle a single rating will be used across \u200ball fields. But each field will be permitted to partly adjust the rating, either up or down two levels for 10% of journals, or up or down one level for 20% of journals.<\/p>\n

The most significant change relates to the quality of research. Rather than reporting the number of papers at the highest tiers of the Qualis assessment, programs will now submit a selection of what they believe to be their highest-quality papers and documents for review. Information scientist Rog\u00e9rio Mugnaini, of the School of Communications and Arts at the University of S\u00e3o Paulo (USP), says that valuing the quality of research over quantity can have a beneficial effect. \u201cIn the CAPES assessment, there are some fields that place too much emphasis on quantitative metrics, leading researchers to push to publish as much as they can at any cost, often at the expense of quality. This can also congest the peer review system,\u201d he says. Mugnaini notes that similar reformulations are being implemented internationally. The UK\u2019s assessment system, which previously used more quantitative criteria given the amount of effort it would take to review all university research over a five-year period, has reintroduced criteria based primarily on peer review. \u201cChanges in the CAPES system are often proposed by experts in different fields and do not always reflect recent global developments in science assessment. When considering prospective changes, it is important that current best practice be taken into account.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"CAPES unveils a new, more comprehensive and diverse set of quality metrics for master\u2019s and doctoral programs","protected":false},"author":11,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_exactmetrics_skip_tracking":false,"_exactmetrics_sitenote_active":false,"_exactmetrics_sitenote_note":"","_exactmetrics_sitenote_category":0,"footnotes":""},"categories":[166],"tags":[226,215],"coauthors":[98],"class_list":["post-332994","post","type-post","status-publish","format-standard","hentry","category-policies-st-en","tag-education","tag-scientometrics"],"acf":[],"_links":{"self":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts\/332994","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/comments?post=332994"}],"version-history":[{"count":4,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts\/332994\/revisions"}],"predecessor-version":[{"id":334411,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/posts\/332994\/revisions\/334411"}],"wp:attachment":[{"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/media?parent=332994"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/categories?post=332994"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/tags?post=332994"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/revistapesquisa.fapesp.br\/en\/wp-json\/wp\/v2\/coauthors?post=332994"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}