Imagezoo/GettyImagesThe University of São Paulo (USP), Brazil’s largest public institution of higher education—and the most prominent in international rankings—begins 2017 with a new model for measuring the performance and quality of the work of its teaching staff that numbers nearly 6,000. On November 8, 2016, the University Council approved a revamped assessment system designed to coordinate existing evaluation mechanisms and, more importantly, establish objectives to be achieved by professors, departments and units working together.
The change was implemented through an overhaul of the Permanent Assessment Committee (CPA) and the Faculty Bylaws. The assessments will be carried out every five years, based on the guiding principles of the academic plans established for each faculty member. These plans will define the expected performance in teaching activities at the undergraduate and graduate levels, student orientation, research, extension and business administration courses, and must be approved by the department to which the faculty member is assigned. Each department, in turn, will have a collective academic plan whose objectives apply to its entire teaching staff. And the Department projects will be coordinated with larger projects carried out at each school or institute.
In putting together their academic plans, professors may, through negotiations with the department, define specific tasks, such as their overarching interest in conducting research or devoting themselves to teaching activities. “This is a major change because it addresses the variety of backgrounds among professors that a university like USP needs to have,” says President Marco Antonio Zago. “It corrects distortions such as requiring that faculty members who are researchers become involved in an inordinate number of administrative or teaching activities, or that call for a professor recognized for her excellence in teaching to produce a long list of scientific papers.”
If the faculty member fails to comply with what is specified in her individual plan, she can sign an agreement to commit to redirecting her work within a three-year period and re-establish the expected academic level. If she fails again, she would be subject to an administrative procedure that may result in a warning, reprimand and dismissal. Faculty members who sign such an agreement once will not be entitled to use this resource in the next two assessment cycles. “There need to be consequences for those cases in which academic plans were not fulfilled, but the focus of the new system is not to punish,” says Alexandre Nolasco de Carvalho, a professor at the Institute of Mathematical Sciences and Computation (ICMC) in São Carlos, chairman of the committee tasked with proposing the model. “The idea is to have goals that foster the development of academic quality among professors, departments, schools and the university as a whole.”
One source of inspiration for the USP model is the triennial assessment process for graduate programs that has been carried out for more than 20 years by the Brazilian Federal Agency for the Support and Evaluation of Graduate Education (Capes), under the Ministry of Education (MEC). “The Capes model is recognized for its ability to promote continuous improvement in the quality of graduate programs. It is performed by peers and the requirements are increased in subsequent cycles,” says Maria Paula Dallari Bucci, a professor in the USP Law School, member of the committee charged with proposing the system and former secretary of higher education at MEC from 2008-2010.
Imagezoo/GettyImagesThe model is intended to refine the previous system, which consisted of three distinct assessment bodies. One of them was the university’s institutional assessment, which the CPA performed every five years and submitted to the State Department of Education. That assessment is conducted through opinions written by advisors who evaluate quantitative indicators, self-assessment forms and interviews with staff, students and university employees. “It is the university’s obligation as a public entity,” Nolasco says. “But, on its own, this assessment has never had the power to guide institutional development at USP.” Now, the purpose of the CPA has been expanded. It is presently composed of a general assembly and two specific chambers—one focused on teaching activities and the other on institutional assessment (of departments and units)—and its mission is to formulate the assessment process of USP entities and establish guidelines and the timetable for assessments.
There was also an assessment committee focused on career development of faculty members as well as one for monitoring the employee probation period for staff and faculty who passed the civil service exam and were admitted to the university. The probation period used to be six years–during which time, the faculty member was evaluated to be confirmed in the position and earn tenure. In the new configuration, the probation period has been reduced to three years. According to Alexandre Nolasco, the three stages were somewhat flawed in that they did not operate in a mutually related way. “There was no connection between the assessments and the university’s institutional development,” he says. “Assessment cannot be an end in and of itself. It has to be part of a process to improve quality, in which objectives are planned and periodically measured so as to determine what has or has not been achieved, and then allow for course corrections.”
Discussions about the model have elicited various critiques from the academic community. Some have led to modifications in the original proposal, but others remain. What Maria Arminda do Nascimento Arruda, the director of the USP School of Philosophy, Literature and Human Sciences (FFLCH) finds concerning is centralization of the assessment process for the entire university in a single body: the CPA. “The assessment process has the ability to synchronize the departmental plan with the faculty member’s main focus: teaching, research and extension activities that will serve as their assessment parameters. But the university model we are accustomed to grants broad autonomy to the various schools and each of them has their own criteria for judgment and assessment. Centralization introduces the risk of expanding standardization and bureaucratization at the university and constricting a system that is already rigid,” she says. According to the professor, although the project provides for the presence of representatives from the social sciences and the humanities on the assessment committee, there is real concern that criteria so unlike the tradition in these fields will be used to measure the quality and performance of faculty members. “I’m completely in favor of assessment because USP is a public university and has to be held accountable. But the question that really needs to be asked is: will excessive standardization compromise cultural, scientific and academic activities at the university?”
Discussions concerning the changes in the evaluation system began back in 2014, when President Marco Antonio Zago established a committee tasked with updating the system. In 2016, at least three versions of the regulations were produced through a process of public hearings, incorporating contributions made during unit meetings and in meetings held on the Ribeirão Preto, São Carlos and São Paulo campuses. “We received more than 800 pages of suggestions, which were in large part incorporated into the final text approved by the University Council, says Bucci.
Initially, linking assessments to the type of work arrangement was considered–86% USP faculty fall under the full-time teaching and research arrangement (RDIDP). But that idea was abandoned. “There was a completely mistaken interpretation in some quarters of the university that suggested that the President’s office might be interested in reducing the number of RDIDP professors to save money,” says President Marco Antonio Zago. “Since this was not true and the RDIDP plays an essential role in the university, we decided to take type of work arrangement off the table, even though the system clearly needs to be updated.” If the notion of tying evaluations to type of work arrangement had gone forward, it would not have been the first time. Back in the 1990s, the University of Campinas (Unicamp), for example, instituted a faculty performance assessment format that could have resulted in the elimination of the RDIDP, and a resulting reduction in salary for professors whose assessments indicated that their performance had not reached the level determined through established criteria. There are always vacancies for part-time faculty at Unicamp. Those who pass civil service exams, after submitting an activity plan, may be authorized to enter the full-time track, but that does not ensure them the right to definitely remain in that category. Another sensitive topic was the makeup of the CPA, which initially would have had all its members indicated directly by the Office of the President. The approved text specifies that only three of the nine members will be appointed by the President. The rest will be named by units and faculty groups.
Imagezoo/GettyImagesAccording to Elizabeth Balbachevsky, a professor in the Department of Political Science at the School of Philosophy, Literature and Human Sciences at USP, the new system is an improvement on several fronts. “For the first time, it includes something that was always considered taboo at USP: accountability for faculty performance,” says the professor, remembering that back in 1988, the leaking of a list of USP professors who had written no papers in previous years left a lasting impression. “This accountability is very clear and specifies that the faculty member will be assessed according to solid and predefined parameters,” says Balbachevsky, who takes part in research studies about the academic profession in Brazil and in other countries.
Renato Pedrosa, a professor in the Department of Political Science and Technology of the Geosciences Institute of Unicamp, is critical of one provision in the new USP Bylaws—the one which states that at the end of the three-year probation period, the recently-hired faculty member will be evaluated by a committee made up of three people, at least one of whom is associated with the department to which the staff member is assigned. “If the committee has at least one member who is a colleague of the professor being evaluated, that means there is the possibility of forming a committee with two or three members from the same department. This seems unlikely to prevent the endogeny that is a feature of many Brazilian universities and the formation of interest groups that maintain the departmental status quo,” he says.
Pedrosa also believes that there need to be more criteria about what constitutes acceptable performance in assessing faculty and departments. Elizabeth Balbachevsky agrees that the new system is vague regarding these criteria, but says that this was done intentionally. “Clearly, it is a solution that contemplates the diversity of academic backgrounds that make up the units at USP, which attracts faculty from professional schools to international level professional enclaves,” says the professor. President Marco Antonio Zago asserts that the objective of the reform was to establish the system’s general framework, which can now be discussed and refined by the new assessment bodies. “It would make no sense to establish every last detail because it would become more complicated to correct what does not work and refine the model over time,” he says. The presence of representatives from various fields on the CPA, says Zago, is important for reaching consensuses and lending validity to the process. “There is a tendency for researchers from some fields to fail to recognize the assessment criteria used by others, or even to reject such criteria. We have to get beyond that.”
Elizabeth Balbachevsky points out that the solution adopted by USP follows university governance models used in other countries, based on mechanisms for valuing and monitoring faculty performance. “The difference is that at British, U.S. and Australian universities, the assessment processes are authorized by experts from outside of the institutions, many of whom are from other countries. This is missing from the USP model and could serve as a counterweight to possible corporate dynamics that the new type of evaluations could engender.”
Republish