Published in October 2012
GABRIEL BITTARThe Audit Court of the State of Mato Grosso (TCE/MT) monitors spending on education and health in 141 municipalities and the impact of these policies on the quality of services rendered to citizens through georeferenced databases developed by the Center for Metropolitan Studies (CEM). “This model allows us not only to assess whether municipalities are spending what the constitution requires, but also whether their performance has been improving,” says Marta Arretche, a professor in the Political Science Department of the University of São Paulo (USP) and director of the CEM.
The methodology, created five years ago at the request of the TCE/MT, was transferred to the state and began to be used as a tool for evaluating the performance of municipal administrations. The information resulting from the application of the model, the Health and Education Performance Index for the cities of Mato Grosso, is available on the TCE/MT web site and can be accessed by any citizen interested in obtaining information about spending and services in these areas in their municipality.
Communicating the results of policy assessments is the primary mission of the CMS, one of the Research, Innovation and Dissemination Centers (RIDCs) supported by FAPESP (São Paulo Research Foundation) and established by the Brazilian Center for Analysis and Planning (CEBRAP) and the USP School of Philosophy, Letters and Humanities (FFLCH) and School of Communication and Arts (ECA). Since its inception in 2003, over 30 research projects have been implemented that involve methodology transfer to the public sector.
The partnership between the TCE/MT and the CEM began in 2007, when the court decided to conduct an investigation that went beyond checking the appropriateness of public spending, including analyses of legality, legitimacy and economic viability. “The court also felt that we needed to assess whether spending had a positive impact on society,” says Volmar Bucco Jr., a public auditor at the TCE/MT. This investigation gave rise to the notion of evaluating public policies using performance indicators, and the court signed a partnership agreement with the CEM. “This is a valuable tool. Through the portrayal of the results of public policies on health and education in the state and its municipalities, the efficient management of public resources can be evaluated through external control,” he explains. Thus, the government can assess its own performance by evaluating progress and results compared to the Brazilian average. This facilitates decision making, and citizens can monitor the performance of public officials and demand improved services.
The Health and Education Performance Index is actually a georeferenced database with information provided by the ministries of education and health and permits a comparison of the performance levels of different Brazilian municipalities. The assessment matrix considers 10 items for each of the areas in question. The analysis of health systems, for example, tallies neonatal mortality, health coverage for pregnant women, children’s health indicators, deaths from infectious contagious diseases (such as dengue fever and tuberculosis) and the number of women’s health visits. The data are collected by the Unified Health Care System database (Datasus). The information used to analyze the education system was collected on grade repetition rates up to the 4th grade and for the 5th to 8th grades, among other data. The National Institute for Educational Studies and Research (INEP) and the School Census (systems for evaluation of basic education in Brazil) provided this information.
GABRIEL BITTARNational Data
The maps on the right and on page 113, for example, allow us to compare the performance of the health and educational systems in different municipalities throughout the country. The reddish areas indicate situations that are classified as “very bad.” Dark blue areas represent circumstances rated as “very good.” These two tones and the color gradient between them consolidate data collected at the municipal level from across the country.
According to Marta Arretche, this type of analysis was only possible because the country has comprehensive and free public information systems. “To learn how the municipal or state governments are spending, you need to build a reliable system. And this information already exists. Only Japan compares to Brazil in terms of the availability of information on public systems. “In the United States,” she adds, “there are no national information systems containing city data. There, to assemble a similar system one would need to go city by city, collect the desired information, then standardize it. Here we have the School Census, the Brazil Exam, Datasus, and the Brazilian Institute of Geography and Statistics (IBGE) municipal information surveys.”
This information must be standardized to permit comparisons. For each indicator, the municipality receives a score from 0 to 1. The indicator scores are summed to obtain the total score. The conditions are always analyzed for three-year periods (the maps are for the periods 2004–2006 and 2007–2009). “We work with a three year average because, if an extreme event were to occur in a city in a given year, for example, distortions could be corrected.”
The result of the comparative analyses for the period reveals that, in education, a substantial number of municipalities exhibited extremely poor performance at the beginning of the decade, while a small number of cities represented “islands of excellence” that have increased in size over the decade. Conversely, the Brazilian health sector does not exhibit these contrasts. “There is less inequality between municipalities, but we have no points of excellence, even though significant improvements have been seen over the years,” says Arretche.
The quality of life indicator
The expectation of the TCE/MT’s public auditor is that this analytical methodology should be made available to any public agency in Brazil, or even to private ones. According to Marta Arretche, the study may help measure the degree of economic development and quality of life provided to the population of a country the size of Brazil. In her opinion, the Human Development Index (HDI) does not represent a good tool for the assessment of decentralized public service delivery systems. Because the CEM methodology permits much more comprehensive and realistic evaluations of the performance of health and education policies, it supports the development of more precisely targeted public policies that will have more substantial effects on the affected population.
Mato Grosso, for example, is beginning to see results. The diagnostic indicators consolidated by this system have already resulted in improvements statewide. “Since the TCE/MT began using this methodology to assess the accounts of public administrative officials as well, societal control has intensified and the tool has begun to show the status of each municipality compared with others,” he explains. “Government officials then had to take steps to improve their indices.”
The successful implementation of the methodology had other outcomes: the CEM was asked to develop a panel for monitoring the number of homicides for the state Secretary of Public Safety, based on a set of indicators that measure the performance of security actions and also allow comparisons among different regions in the state. The panel serves as a sort of alarm whenever it registers negative changes in the average indicators.
“This information methodology has become an important tool for assessing and supporting the formulation of public policy,” said Eduardo Marques, a CEM researcher and professor in the USP Political Science Department. The CEM has conducted over 30 studies focused on providing support the formulation and implementation of public policies. Among these are a Map of Social Vulnerability, which allowed deficiencies to be detected by analyzing the distribution of socioeconomic structures, and Environmental indicators and urban management: sustainability challenges in the city of São Paulo, which enabled the administration to identify the substantial variability in environmental problems and pressures in different urban spaces; both of these studies concerned the City of São Paulo. Two other studies conducted for São Paulo concerned housing vulnerability, which were used to develop the city’s Housing Plan. This last experience led the Ministry of Cities to commission the development of a methodology for the analysis of shantytowns, which generated information about the housing needs of a selected group in 560 municipalities.
Partnerships with public agencies developed gradually through contacts between managers. “The CEM was one of the first to digitalize databases,” says Marques. “We started with the city of Mauá in 2003. The information circulates, and demands arise.”
Republish