catarina besselPublished in october 2014
A paper published by Brazilian researchers in the journal Scientometrics demonstrates how a certain method can be used to evaluate the impact of research programs, especially when the number of projects to be analyzed is limited and the control group – the reference group used as a basis for comparison – has characteristics that differ from those of the group targeted by the study. The paper, which was authored by Fernando Colugnati of the Federal University of Juiz de Fora (UFJF), Sergio Firpo of the Fundação Getúlio Vargas (FGV), and Paula Drummond and Sergio Salles-Filho of the University of Campinas (Unicamp), provides a detailed description of this innovative method that was designed to evaluate the Biota-FAPESP Program, which has been studying the biodiversity of São Paulo State since 1999. To gather data about the program’s impact, the group compared Biota-FAPESP projects with others that have equivalent characteristics but are not part of the program. Contradicting certain biases from confounding the results was a major challenge. The higher number of projects in the control group – not to mention their characteristics, which differed somewhat from the target group – was just one of these biases. Whereas the Biota-FAPESP Program sponsors a significant number of thematic projects, which receive large amounts of funding and human resources and can last for up to five years, many projects in the control group were standard research projects, whose duration and resources are more modest.
Thus, the researchers designed a method that could statistically dissociate the effects of Biota’s thematic projects. To estimate the probability that each project in the control group could be compared with a Biota Program project, they employed a statistical theory designed in the 1980s. A group of variables that could be potential sources of bias in the study was defined. These variables – including the age of the principal investigator, the number of papers published and the size of the research team – could increase a project’s probability of belonging to the Biota group. This probability, which is called the propensity score (PS), was calculated for each project in the control group. The inverse of the PS provides a type of weighting factor that can be used to correct distortions. As additional challenges, the researchers needed to compensate for the two groups’ differences in terms of size and composition and the small size of the sample universe from which the projects were selected. “Once the PS had been estimated, the thematic projects in the control group were assigned a higher weight because they were relatively rare. By doing this, we were able to achieve a more uniform and balanced distribution, permitting a less biased comparison between the groups, using statistical models,” says Colugnati.
To select projects for the control group, the researchers accessed approximately 1,400 biological sciences projects in FAPESP’s database that were not included in the Biota-FAPESP Program. A search based on keywords (such as “biodiversity” or “biomes”) reduced the sample to approximately 300 projects, and a case-by-case analysis led to a final sample of 117 projects within this group, including thematic, regular research, and young investigator projects. Adding the 66 projects in the Biota group, the total sample universe consisted of 183 projects. The respective principal investigators were encouraged to complete an online questionnaire to support the assessment. A total of 142 researchers responded, representing 56 Biota-FAPESP Program projects and 86 control group projects. Propensity scoring and statistical modeling were then applied to these data.
Prospecting
The final judgment was positive for Biota-FAPESP, both in terms of its scientific productivity and ability to support new public policies, although the program still produces modest results in regard to prospecting for compounds that could potentially be used in the pharmaceutical, cosmetic, and other industries (see Pesquisa FAPESP Issue nº 210).
This evaluation of the Biota-FAPESP Program was supported by a grant from FAPESP and was conducted by the Study Group for Organization of Research and Innovation (GEOPI), which is affilia-ted with the Department of Science and Technology Policy (DPCT) of the Institute of Geosciences at Unicamp. The group is coordinated by Professor Sergio Salles-Filho, member of FAPESP’s Special Programs Panel, including fellowships, Young Investigator and Multi-user Equipment Program (EMU) grants, and the Innovative Research in Small Businesses Program (PIPE), among others. Salles-Filho, who is a supervising member of the FAPESP Area Panel of Special Programs, for program evaluations, says that the particular characteristics of each program mean that they can only be evaluated after a specific method has been designed. “And this is sometimes not necessary, but we take the opportunity to test new hypotheses and methods,” he says. “The biggest challenge, when evaluating the impact of a program, is guaranteeing the attribution of causality; in other words, making sure that the measurements obtained will be effectively attributed to the investment made by the program, attempting to isolate other factors that may influence a program’s impact. Whenever possible, we recommend using a control group, but the control group must be reliable, which explains the usefulness of the methodology based on propensity scores,” he explains. The method has very well-defined applications. “Our group worked on an evaluation of companies that used incentives from Brazil’s legislation on information technology, but no control group was possible because the vast majority of Brazilian companies use these incentives, and so there was no way to build a reliable control group,” he says.
According to Salles-Filho, designing new methods gives evaluation processes the opportunity to generate knowledge. The scientific contribution generated by the Biota-FAPESP evaluation will not be limited to the paper in Scientometrics. The research group at GEOPI expects to complete another study that will compare two distinct methods using the results from the Biota evaluation. One is the method that relies on a control group, as described in the article. The other method, which known as additionality with verification of causality, aims to measure a project’s impact without using a control group by comparing data from the start and end of a project. “Our goal is to see whether the two methodologies produce equivalent results, or if differences will emerge,” says Salles-Filho.
Republish