Five cities in inland São Paulo State are using a method capable of flagging inequalities across their public education networks, with simplified indication of schools posting outstanding performance, and those in need of attention and interventions. The model, created by the Sérgio Henrique Ferreira Chair of the Institute of Advanced Studies at the University of São Paulo (IEA-USP), in Ribeirão Preto, is based on a matrix of indicators that reflect different dimensions of pedagogical development, such as student approval rates, age-year distortion (number of students two or more years older than that expected for each school year), schools’ scores in external assessments, and their performance in mathematics and Portuguese language. Combined into a single index, these measures demonstrate whether or not municipal public schools are overcoming the learning challenges and problems arising from the socioeconomic situation around them.
Application of this method, funded by FAPESP, produced georeferenced maps of the cities of Ribeirão Preto, Cordeirópolis, Jundiaí, Francisco Morato, and Batatais, in which schools are represented in these municipalities by colors summarizing their performance; from red (indicators requiring attention), through orange, yellow (intermediate), to green (best). “The aim is to improve the efficiency of educational management. The methodology is based on statistical modeling and data science to inform education departments and administrators, and facilitate the decision-making process,” explains chemical engineer Mozart Neves Ramos, coordinator of the initiative, and incumbent of the Sérgio Henrique Ferreira Chair, which seeks out ways to contribute to public policies in medium-sized cities, primarily in the area of education. The method has been used since last year to assess students in the 5th and 9th year of elementary education (6–15 years), and enabled comparison between performance evolution at the schools between 2017 and 2019, helping administrators to identify issues and their causes.
Biologist Rafael Naime Ruggiero, the chair’s data analysis group coordinator, says that one of the differentials of this method is the focus on small and medium-sized municipalities, and the use of statistics to analyze the relationship between different educational indicators. “The aim is to gauge the inequality of performance in a network,” he explains. The next step in the initiative will be to apply the methodology for the first time in a big city — the São Paulo municipal education network — providing a more accurate diagnosis of educational inequalities across the 13 administrative regions of the municipality. A partnership agreement has been signed between the chair and the São Paulo Municipal Education Department.
According to Mozart Ramos, the diagnosis also helps to assess whether the causes of poor performance come from within or outside the school. “When a school is located in a community with high rates of violence or drug use, the impact on proficiency is very high. The children live under stress,” he states. Ramos goes on to emphasize that these solutions are not necessarily simple. “The diagnosis allows the department to assess whether there is a management problem, and try to correct it. However, problems are frequently associated with a scarcity of teachers qualified in certain subjects. In some places, math teachers have to give physics and chemistry classes.”
Chair researchers analyzed data from the network of 109 municipal schools in Ribeirão Preto, a city in the northeast of São Paulo State with a population of more than 700,000, and observed, for example, a drop in the performance of one of these units, the Vereador José Delibo School, when comparing indicators from 2017 and 2019. The percentage of students with satisfactory learning in the Portuguese language plummeted from 93% to 77%, and in math, from 82% to 64%. The researchers visited the college and found that during this time frame, Portuguese and math teachers had been moved into management roles and replaced by other less experienced teachers. Another important factor was that in this period the school did not run an enrollment selection process, taking on local students at different learning levels.
In the city of Jundiaí, 57 km from the São Paulo state capital and with a population of 423,000, a municipal school network of 104 schools and 38,000 students, improvements were seen in performance across the educational facilities, attributed to a public policy developed over the two most recent mayoral terms. This is down to the “school unwalling” method, aimed at adapting elementary-level education curricula to get students outside the classroom and closer to nature and the city in which they live, in addition to using digital technologies to put them in virtual contact with museum collections from other countries, for example. According to Vastí Ferrari, the municipal education secretary, this methodology has helped to demonstrate how much each school has been able to draw on the pedagogical proposal.
“It was a nice surprise to see a school that had an average external assessment score below 7 (on a scale of 1 to 10) improve and score 8 after they started to work with the unwalling methodology,” says the secretary. “When I ask a school’s management what happened, they very clearly respond that they put the children to work with aspects of nature to improve their scientific repertoire, using recreational strategies to improve their level of writing, and held monthly meetings with teachers to address issues at the school and monitor the performance of each pupil,” she explains.
Daniel AlmeidaFerrari goes on to say that at schools that underwent internal changes, where the methodology was still in the implementation phase, she witnessed a shift in the performance measured using the IEA-USP Chair method. “The schools are very sensitive to change, such as the retirement of experienced teachers and their replacement by younger ones, where it is not always possible to reproduce strategies already in place,” she says. The challenge now, says Vasti, is to spread the word on the solutions and innovations adopted successfully among schools experiencing difficulties. “This knowledge needs to be shared across the whole network, and data- and evidence-based methodology gives a solid basis to show what is working well.”
In Cordeirópolis, in the Piracicaba Metropolitan Region, the methodology has become a tool for monitoring the performance of 14 elementary education units, which cover 2,900 students. The school network has also benefited from a supplementary initiative by the Sérgio Henrique Ferreira Chair in the form of an introductory course in educational data analysis for school principals and managers, instructing on how to obtain and interpret information and use it in decision-making. “Managers are not generally prepared to analyze data. Our primary education coordinator took the course and immediately began to apply the knowledge to our network,” explains Cordeirópolis education secretary Angelita Ortolan. “As ours is a small town, today we can run very specific evaluations and analyze the difficulties experienced by each school, each year, and each student.”
In 2022 the first edition of the course, delivered online, enabled 33 managers from across 10 municipalities, and an in-person edition last year in Ribeirão Preto qualified 29 people from 17 towns and cities. “We want our managers to independently obtain and analyze the data. The idea behind this type of training is to spread a data-analysis culture,” says Ruggiero, who coordinates the offer of these courses for the Chair.
The strategy is now enshrined in municipal legislation in Cordeirópolis and Batatais. In June, the two city halls approved laws instituting public, evidence-based education policies in their municipal systems. “A law of this kind is essential for the municipality, regardless of who is in power, to implement educational actions based on scientific research and studies, driving learning and fairness, and reducing educational inequalities,” says Victor Hugo Junqueira, Education Secretary for Batatais, whose educational network covers 4,500 students across eleven schools and nine crèches. The rationale for these law bills, supported by the Chair, is that consolidated research groups can help to improve the quality of education, providing evidence-based diagnosis. “If managers don’t support those dealing with the numbers, everything ends up being based on guesswork, and the children don’t learn,” says Mozart Ramos, an aficionado of data use in the formulation of educational policies since his time as executive president of the movement Todos pela Educação (All for Education); Ramos is currently a member of the FAPESP Board of Trustees.
Data and evidence used in education management is spreading — Ceará State, known for its innovative pedagogical experiments and public policies, provides an example: an initiative of the State Education Department, led by mathematician Jorge Lira, of the Federal University of Ceará, conducted tests 2 to 3 times per year with high school–level students (in Brazil, age 15–18), and identified specific failings in math learning, for example basic arithmetic, which they carried forward from primary-level education. These deficits are being addressed through reconstitution of learning for students, and specific training for teachers. The initiative is funded by the Ceará State Education Department in collaboration with the Chief-Scientist Program of the Ceará Scientific & Technological Development Foundation (FUNCAP), which seeks to approximate academia to public administration by deploying researchers to help solve problems faced by government bodies (see Pesquisa FAPESP issue nº 274).
Lira says that the strategy was born of efforts to understand why the performance of Brazilian students in math generally becomes stagnant or declines during the final years of middle-level education and during high school, contrary to what happens with other subjects. “Our student assessments revealed that the low levels of consolidation in certain basic knowledge areas, such as how to deal with arithmetic operations, was a deep-seated problem, Lira explains. Another finding is that failings in teacher qualification contribute to the issue — not all have instructional mathematics tools to help the students.
After identifying the basic knowledge areas presenting difficulties for students, the teachers collaborate to formulate didactic sequences to resume and review content. “We produce structured materials that can be adapted to the different learning pathways, and offer suggestions on how to implement them.” The outcome of this strategy, based on assessment of over 300,000 pupils from the three years of high school education, was measurable, including during the COVID-19 pandemic. “Unexpectedly, there was no drop in performance in math among state-network students during social isolation. Grade averages were maintained, and exams demonstrated that bottlenecks currently faced by students are related to more complex high school education content, and no longer to those at primary/middle level — a notable qualitative improvement,” concludes Lira.
The story above was published with the title “The educational inequality map” in issue 342 of august/2024.
Project
Use of statistical modeling and data science to improve learning and reduce educational inequalities in the public education system (nº 22/06522-8) Grant Mechanism Regular Research Grant – Research in Public Policies; Principal Investigator Mozart Neves Ramos (IEA-USP); Investment R$376,947.22.
