CATARINA BESSELA paper published by Brazilian researchers in the journal Scientometrics demonstrates how a certain methodology can be used to evaluate the impact of research programs, especially when the number of projects to be analyzed is limited and the control group – the reference group used as a basis for comparison – has different characteristics from those of the group targeted by the study. The paper, authored by Fernando Colugnati from the Federal University of Juiz de Fora (UFJF), Sergio Firpo from the Fundação Getúlio Vargas (FGV), and Paula Drummond and Sergio Salles-Filho from the University of Campinas (Unicamp), gives a detailed description of this innovative methodology designed to evaluate the Biota-FAPESP Program, which has been studying the biodiversity of São Paulo State since 1999. To gather data about the program’s impact, the group compared Biota-FAPESP projects with others that have equivalent characteristics, but are not part of the program. Preventing certain biases from confounding the results was a major challenge. The higher number of projects in the control group – not to mention their characteristics that differ somewhat from the target group – is just one of these biases. Whereas the Biota-FAPESP Program sponsors a significant number of thematic projects, which receive larger amounts of funding and human resources and can last for up to five years, many projects in the control group were regular line of research projects, whose duration and resources are more modest.
So the researchers designed a methodology that could statistically dissociate the effects of Biota’s thematic projects. To estimate the probability that each project in the control group would be comparable to a Biota Program project, they turned to a statistical theory designed in the 1980s. A group of variables that could be potential sources of bias in the study was defined. In other words, these variables – including the age of the principal investigator, the number of papers published and the size of the research team – could increase a project’s probability of belonging to the Biota group. This probability, called the propensity score (PS), is calculated for each project in the control group. The inverse of PS provides a kind of weighting factor that can be used to correct distortions. As additional challenges, the researchers needed to compensate for the two groups’ differences in size and composition, as well as the small size of the sample universe from which the projects were selected. “Once the PS had been estimated, the thematic projects in the control group were assigned a higher weight because they were relatively rare. By doing this, we were able to achieve a more uniform and balanced distribution, permitting a less biased comparison between the groups, using statistical models,” says Colugnati.
To select projects for the control group, the researchers accessed roughly 1,400 biological sciences projects in FAPESP’s database that were not included in the Biota-FAPESP Program. A search based on keywords (such as “biodiversity” or “biomes”) reduced the sample to approximately 300 projects, and a case-by-case analysis led to a final sample of 117 projects within this group, including thematic, regular line of research, and young investigator projects. Adding the 66 projects in the Biota group, the total sample universe consisted of 183 projects. The respective principal investigators were encouraged to fill out an online questionnaire to support the assessment. A total of 142 researchers responded, representing 56 Biota-FAPESP Program projects and 86 control group projects. Propensity scoring and statistical modeling were then applied to these data.
Prospecting
The final balance was positive for Biota-FAPESP, both in terms of scientific productivity and ability to support new public policies, although the program still produces modest results when it comes to prospecting for compounds that could potentially be used in product development for the pharmaceutical, cosmetic, and other industries (see Pesquisa FAPESP Issue No. 210).
This evaluation of the Biota-FAPESP Program was supported by a grant from FAPESP and was conducted by the Study Group for Organization of Research and Innovation (GEOPI), affiliated with the Department of Science and Technology Policy (DPCT) of the Institute of Geosciences at Unicamp, coordinated by Professor Sergio Salles-Filho, who is also responsible for evaluating other FAPESP programs, including scholarships, Young Investigator and Multi-user Equipment Program (EMU) grants, and the Innovative Research in Small Businesses Program (PIPE), among others. Salles-Filho, who is a supervising member of the FAPESP Area Panel of Special Programs, for program evaluations, says that the particular characteristics of each program mean that they can only be evaluated after a specific methodology has been designed. “And this is sometimes not necessary, but we take the opportunity to test new hypotheses and methods,” he says. “The biggest challenge, when evaluating the impact of a program, is guaranteeing the attribution of causality; in other words, making sure that the measurements obtained will be effectively attributed to the investment made by the program, attempting to isolate other factors that may influence a program’s impact. Whenever possible, we recommend using a control group. But the control group must be reliable, which explains the usefulness of the methodology based on propensity scores,” he explains. The methodology has very well-defined applications. “Our group worked on an evaluation of companies that used incentives from Brazil’s legislation on information technology, but no control group was possible because the vast majority of Brazilian companies use these incentives, and so there was no way to build a reliable control group,” he says.
According to Salles-Filho, designing new methodologies gives evaluation processes the opportunity to generate knowledge. The scientific contribution generated by the Biota-FAPESP evaluation will not be limited to the paper in Scientometrics. The research group at Geopi expects to complete another study by the end of 2014, comparing two distinct methodologies using the results from the Biota evaluation. One is the methodology that relies on a control group, as described in Scientometrics. The other, known as additionality with verification of causality, aims to measure impact without using a control group, by comparing data from the start and end of a project. “Our goal is to see whether the two methodologies produce equivalent results, or if differences will emerge,” says Salles-Filho.
Republish