Imprimir Republish

EVALUATION

Reconstructed Trajectories

Group evaluates FAPESP programs and shows where grant recipients supported by the Foundation came from and where they went after completing their projects

An unprecedented research project that examined the profiles and career paths of grantees who took part in undergraduate, master’s, and doctoral projects at São Paulo universities and institutions financed by FAPESP and other research-sponsoring agencies between 1995 and 2009 identified four distinct groups of students, classified according to income, family structure and education. The first group is composed predominantly of people from Brazilian socioeconomic classes D and E. They came from public schools and average-size families and did undergraduate research. The second group consisted of people from classes A and B, who had attended private schools in the state of São Paulo and who also did undergraduate research. The third is composed of students from class A in Brazil’s North and Northeast regions who had begun research during their master’s degree studies. The fourth group, individuals from classes C and D and also beginning to do research while studying for their master’s, had parents who had little schooling.

This classification into four groups is one of the results of a recently-concluded research project funded by FAPESP that produced in-depth evaluations of three of the Foundation’s programs: the grants for undergraduate, master’s, and doctoral research; the BIOTA-FAPESP program conducted to develop a detailed account of the biodiversity of São Paulo State; and the Multi-user Equipment Program (EMU) that arranges for the purchase of very expensive equipment that can be used by researchers from several institutions. The former FAPESP grantees came predominately from groups 1, 2, and 3, explains Sergio Salles-Filho, who was responsible for the evaluation and served as coordinator of the Study Group on Organization of Research and Innovation (Geopi), affiliated with the Department of Science and Technology Policy (DPCT) of the Institute of Geosciences at the University of Campinas (Unicamp).

Ana Paula Campos

The evaluation of the grant program was based on a comparison between a group who had received grants from FAPESP between 1995 and 2009 and a control group composed of individuals who had received the benefit from other agencies. Using databases from FAPESP and the Coordinating Agency for the Improvement of Higher Education Personnel (Capes), researchers from Unicamp sent e-mails to 39,765 former grantees in the undergraduate, master’s, and doctoral programs. They were invited to answer a questionnaire that, to make responding easier, was already partially filled in with information obtained on the Lattes platform and also from the databases used. A total of 12,343 responses were received. “It’s true that the sample has a bias, because it is hard to find those grantees who ended their academic careers at the undergraduate level and have no current record in the databases we consulted,” says Salles-Filho, who is also assistant coordinator of special programs for the scientific director of FAPESP. The evaluation produced a panorama of the origins and destinations of former grantees. It showed, for example, that they came from diverse socioeconomic strata—there were significant numbers of students from all social classes—and that although the majority were concentrated in São Paulo State, after they graduated or earned their master’s and PhD a reasonable contingent went to work in other states. In 2011, 32% of the funds disbursed by FAPESP financed grants—those for undergraduate, master’s, and doctoral studies were responsible for 87% of the grants awarded in Brazil.

Characteristic profiles of the grant recipients emerged from questionnaires, such as the one answered by Carlos Alberto Moreno Chaves, 30, candidate for a PhD in geophysics from the Institute of Astronomy, Geophysics and Atmospheric Sciences (IAG) of the University of São Paulo (USP). Son of a day laborer, he always attended public schools. Since 2006, when he was doing undergraduate work at the IAG, he has been a FAPESP grantee, first in undergraduate research, then in the master’s program, and now working on his PhD under the guidance of Professor Naomi Ussami. “Since I always attended university full-time, I relied on grants from FAPESP for my support. Recently I even got married,” he said. He once worked as a furniture assembler while preparing for the college entrance exam—he was accepted by USP on his second attempt.

Ana Paula CamposBárbara Medeiros Fonseca, 36, from Espírito Santo State, went directly to the doctoral program in ecology of land and aquatic ecosystems at USP after getting her undergraduate degree. She, too, won a grant from FAPESP. A native of Conceição da Barra in Espírito Santo, she went to school in a neighboring city in the state of Bahia. “I had to travel 80 kilometers every day,” she says. She had moved to Brasília to do undergraduate work in biology at the University of Brasília when she received a grant from the National Council for Scientific and Technological Development (CNPq). Between 2001 and 2006 she obtained her PhD at USP, with Carlos Eduardo de Mattos Bicudo as her advisor. Now she teaches at the Catholic University of Brasília.

The largest number of former FAPESP grantees graduated in the state of São Paulo, but when researchers looked at their origins starting with high school, the Northeast region gains importance, a sign of the flow of northeastern students to São Paulo universities. Most former FAPESP grantees continued to work in São Paulo State after their grants expired. “But the percentage who remained in this state fell as individuals earned additional degrees. There was an increase in the percentage of grantees, particularly PhDs, who settled in other states of the Southeast, and also in the Northeast and North of Brazil,” says Adriana Bin, an advisor in the post-graduate program at Unicamp’s School of Applied Sciences, member of the team, and responsible for evaluating the FAPESP grant program. While the current places of employment of 89% of the former undergraduate research grantees bear a São Paulo address, that proportion falls to 69% among PhDs. “One theory is that this finding reflects an increase in demand for teaching staff in other Brazilian states. FAPESP may be playing the role of trainer of personnel for other states, not just for São Paulo.” More than half the former PhD grantees are teaching in public universities, while holders of bachelor’s and master’s degrees are employed in other sectors of the job market. In all three categories of former grant recipients it was found, consistently, that around 8% of them are engaged in research not associated with education; they are working for companies, at research institutions, or independently.

Ana Paula CamposAccording to the results of the evaluation, the number of researchers pursuing post-doctoral studies is 60% higher among former FAPESP grantees than among those from other agencies. Students supported by the Foundation also forged more international collaborative relationships involving data sharing (38.9% in the case of the PhDs) than in the control group (28%). Another significant figure refers to the number of papers published in co-authorship with foreign researchers: 43.3% of former FAPESP doctoral grantees have published with co-authors from other countries, compared with 31.1% of grantees from other agencies. Students funded by FAPESP published more often in high impact journals, applied for more registrations of intellectual property, and generated more innovations through their research work than other grant recipients, although here we must consider the striking differences among the various fields of knowledge.

Entrepreneurial spirit
FAPESP grantees are distinguishing themselves with respect to entrepreneurial activities. Those who interrupted their academic careers right after completing their undergraduate research work are working more often as businessmen or independently than the former recipients of grants from other agencies. But among researchers who received a PhD, the majority of whom are on the teaching staff of public universities, the opposite was observed; there are more entrepreneurs among the control group. The starting salaries of former recipients of grants from the São Paulo agency exceeded the earnings of the other former grantees in the master’s programs. But the trend in income over the course of a grantee’s career favored those from FAPESP among recipients of grants in both the master’s and doctoral programs.

The Foundation’s former grantees followed a regular academic path from undergraduate to doctorate, passing through the master’s level, and this was true almost 50% more frequently than among the others. It may seem paradoxical, but the survey also showed that the percentage of FAPESP grantees who stopped at undergraduate studies instead of continuing an academic career was almost three times higher than the percentage found in the other group. “As we have already seen, a good number of those who do undergraduate work go directly into the labor market, sometimes as entrepreneurs,” Adriana Bin says. “But those who go for a master’s degree with a grant from the Foundation usually continue and pursue a PhD.” The evaluation report makes a series of recommendations for improving the efficiency of the grants program, such as creating mechanisms to attract more grantees from other states, and integrating PhD grant recipients more closely into research and development activities at companies, besides strengthening international collaborations, an area in which FAPESP grantees already shine.

The goal of the Geopi project was to develop and apply methodologies for evaluating the results and impacts of scientific, technological, and innovation programs. Besides focusing on the grants, BIOTA, and EMU programs, the project also sought to establish rules and criteria for continuous evaluation of four other FAPESP programs that had previously been looked at: The Program to Support Research in Small Business (Pipe), the Partnership for Technological Innovation (PITE), the Young Investigators Awards Program, and the Program for Research into Public Policies (see Pesquisa FAPESP Issue No. 147).This front, headed by Ana Maria Carneiro, researcher at the Center for Public Policies Studies (NEPP), also at Unicamp, complements the evaluation work already done, since it permits continued monitoring of the programs. It is intended that the same will occur with the grant, Biota, and EMU programs in the future.

Ana Paula CamposThe methodology developed by the group is innovative and resulted in submission of an article to a journal in the field of scientometry, which is a field of knowledge that seeks to generate data to encourage the overcoming of the challenges posed by science. Other articles are being drafted. One of the challenges facing the Unicamp researchers was to design control groups to serve as comparison with the data on the FAPESP programs. In the case of the grants program, the comparison was made with a group of students in undergraduate, master’s, and doctoral research who had applied for grants from FAPESP but were denied and managed to obtain aid from other sponsoring agencies. “The premise is that if they were able to have their application evaluated by FAPESP and received a grant from another source they are comparable to former FAPESP grantees,” explains Fernando Colugnati, a Geopi researcher responsible for the study’s sample design and the statistical analyses. Similarly, the projects in the BIOTA-FAPESP Program were compared with a group of biodiversity projects supported by the Foundation that, for various reasons, were not included in that program. The data for that control group needed to be calibrated in order to circumvent an important bias: while the Biota program features a significant number of thematic projects that involve more human and financial resources, most of the projects in the control group were aids to research efforts of shorter duration. So an attempt was made to use a methodology that could statistically decouple the effects of the Biota thematic projects.

Biota-FAPESP
With regard to the Biota-FAPESP program, an effort begun in 1999 to identify the biodiversity of São Paulo, the result of the evaluation was quite positive, both in terms of scientific productivity and in its ability to provide a foundation for new public policies, even though it showed mediocre results in the prospecting for chemical compounds that might have a potential use in development of products in industries such as pharmaceuticals and cosmetics. Among its contributions, it was found that Biota produced good results in the field of taxonomy: its projects identified 524 taxons per project, three times the number found in the control group. The program was also successful in attracting more professionals to the field of biodiversity. “We also observed that the researchers in the program published almost twice as many articles, even after allowance was made for the thematic bias,” says Paula Drummond de Castro, also a researcher in the group and responsible for evaluating the Biota and the EMU programs. Another key finding was performance in terms of the publication of scientific articles, which soared approximately one year after the projects began. The articles generated by Biota involved twice as many international co-authorships as did projects in the control group. Publication of the results of the program outside the academic world was significant: 66% of the projects in the program reported engaging in dissemination activities, compared with 44% of the control group.

According to the evaluation, BIOTA-FAPESP performed below expectations in its ambition to develop new substances derived from biodiversity that have market potential. Many projects found potentially active substances, but the inclusion of the program into more advanced stages of technological development, such as pre-clinical, clinical, and marketing studies proved to be practically non-existent. “The criticism is valid and applies to the entire field of bio-prospecting in Brazil,” says Carlos Joly, coordinator of BIOTA-FAPESP. “In part, that results from the freezing of activity in that area brought about by the infamous Provisional Measure 2.186-16 that regulates access to genetic resources and/or the associated traditional knowledge.” According to Joly, few research groups have been able to bring their collections of bioactive substances identified from native flora and fauna into compliance with that legislation. “Consequently, the production sector could not be certain that these molecules could legally be used.”

Ana Paula Campos
Multiuser equipment
The Multiuser Equipment financing effort has been part of the Research Infrastructure Support Program since 1996. Its objective is to purchase very expensive equipment that can be used by researchers from several institutions. Initially, projects were evaluated as they were submitted, creating a continuous flow. That changed in 2004 when the first EMU request for proposal was announced. The Geopi evaluation did not include projects approved under the second request for proposal, in 2009, that had incorporated several changes in order to ensure involvement of the institutions in the purposes of the program. Principal investigators and associates who had benefited from the EMU projects were compared with others whose applications had been denied by FAPESP, but who ultimately obtained similar equipment from other sources.

The EMU program projects generated more articles, theses, and dissertations, measured in absolute numbers, than did those of the control group. In both groups there was an increase in the average number of articles published around four years after the equipment was installed, followed by a sharp decline after six years. “There are indications that lead us to believe that the EMU program strengthens groups whose use is closer to the equipment, since the partnerships that formed were with partners within the project itself,” says Salles-Filho. The group/laboratory with which the principal investigator was affiliated was the primary user of the equipment during the five-year period. “This finding provoked a discussion about the multi-user nature of the equipment,” the professor says. “This may perhaps be the most challenging aspect, and possibly the most important and controversial. It is important since it refers to the objectives that justify the existence of the program. It is controversial because the kinds of equipment made available by FAPESP vary so widely that it is hard to make a more precise evaluation of their differential impact.”

Republish