Over a span of just a few months, Brazil has dropped seven positions in the ranking of nations with the largest supercomputer arsenals. In November 2015, the country came in at 10th place, with six systems among the 500 highest performers in the world. In June 2016, it fell to 17th place, with four computers. Brazil’s best showing was in 2004 when it had nine representatives on the list. The two Brazilian supercomputers that have fallen out of the ranking belong to Petrobras and to the National Institute for Space Research (INPE). Both continue to produce complex and swift calculations at around trillions of operations per second, and are indispensable for simulating oil field drilling and producing accurate climate studies and weather forecasts. But they are no longer on the list of the 500 fastest because they were surpassed by others of more recent manufacture. Inaugurated in 2016, the world’s fastest computer, China’s Sunway TaihuLight, is capable of performing 93 quadrillion calculations per second – performance that is three times faster than the second-place computer, also from China, the Tinhae-2. That Asian country has 168 systems in the TOP500, a ranking published since 1993 by the University of Tennessee and Germany’s University of Mannheim.
The Brazilian supercomputers cited in the ranking were inaugurated last year. Three of them are part of the Santos Dumont cluster, in operation at the National Scientific Computing Laboratory (LNCC), in the state of Rio de Janeiro. The other, christened Yemoja, was installed at the Center for Integrated Manufacturing and Technology, a unit of the National Industrial Training Service (SENAI-CIMATEC), in the city of Salvador, Bahia State. “The drop in positions illustrates the fact that Brazil is updating its supercomputer arsenal at a slower rate than that of other countries,” says Pedro Leite da Silva Dias, a professor at the Institute of Astronomy, Geophysics and Atmospheric Sciences of the University of São Paulo (IAG-USP), who headed the LNCC from 2007 to 2015. The laboratory is affiliated with to the Ministry of Science, Technology, Innovations and Communications. “On the other hand, for a long time, we only had one computer on that dynamic and quickly changing list. In order to stay on top, or at least maintain our position in the rankings, we need to invest a lot in modernizing our high-performance computing arsenal,” Dias continues. Brazil’s decline in the ranking is sounding alarm bells though. “Having dominance in computer processing technologies gives us a competitive advantage because supercomputers help develop cutting-edge research studies,” say Augusto Gadelha, current director of the LNCC. “More powerful machines encourage researchers to attempt to solve more complex problems.” During the second half of the 1990s, Brazil led the list in terms of number of supercomputers among the nations that now make up the BRICS [Brazil, India, China, Russia and South Africa] bloc. At the moment, Brazil is only ahead of South Africa. “In recent years, we have watched the advance of China, which has surpassed the United States and now has supremacy in terms of arsenal. The rivalry between the two countries has created a sort of supercomputing Cold War,” notes Dias. There is some a battle to see who will be the first to have a computer capable of achieving exaflops, in other words, getting to nearly one quintillion mathematical operations per second.
Besides being powerful research tools, supercomputers also provide support to areas many nations consider strategic, such as defense, nuclear energy and health. In 2014, the Oak Ridge National Laboratory and the Lawrence Livermore National Laboratory, both in the United States, invested $325 million in the purchase of IBM supercomputers that researchers are now using to study climate change, new materials and renewable energy. The infrastructure is shared with government agencies such as the Department of Energy (DOE). In the case of China, percentage of its supercomputers is used by government institutions. “They help on various fronts, such as in planning construction of hydroelectric plants,” Dias explains.
Leader in latin america
Brazil is the only country in Latin America to appear on the TOP500 ranking. It possesses only 0.8% of the world’s 500 supercomputers, a percentage less than the country’s contribution to world science, which is just over 2% of the articles published in indexed journals. High performance computing is relatively new in Brazil: the first acquisition dates back to the late 1980s. At the time, the Brazilian Innovation Agency (FINEP) began a program to promote the purchase of supercomputers at universities and research institutions. The first to benefit was the Federal University of Rio Grande do Sul (UFRGS). In 1994, the Center for Weather Forecasting and Climate Studies (CPTEC) at INPE purchased its first supercomputer, a 3.2 gigaflop NEC Sx-3. “Weather forecasting requires complex equations to be solved in just a few hours. And this can only be done with extremely fast processing times,” explains Luiz Flávio Rodrigues, head of the supercomputing and support unit of CPTEC-INPE.
To handle these demands, supercomputers have to be replaced every four or five years. Since 1994, INPE has had five systems. The most recent is a Cray XT-6 called Tupã (the Guaraní word for thunder), purchased in 2010 for R$50 million – R$15 million from FAPESP and R$35 million from the federal government. The supercomputer has 256 teraflops of capacity. “Now, with the crisis, there are no prospects for replacing it in the short term. Because the computer is already coming to the end of its useful life, we updated some of its components, which should keep it operating for two more years,” Rodrigues says.
The Tupã splits its time working on two fronts: half of the time it is used for CPTEC operations, which include weather forecasting, and the other half is devoted to scientific research. One of Tupã’s applications is processing the Brazilian Earth System Model (BESM) one of the pillars of FAPESP’s Research Program on Global Climate Change (PFPMCG) launched in 2008. Under the coordination of INPE, the BESM was developed on the basis of South America’s climate parameters, incorporating regional cloud formation processes, in addition to knowledge developed in Brazil regarding the influence of the Amazon Rainforest on global climate. In 2012, the Brazilian model starting contributing to the Intergovernmental Panel on Climate (IPCC).
The oil and gas industry also makes intensive use of supercomputers to calculate such things as tank capacity in an effort to reduce the risk of spills. Petrobras’ two main computers – the Grifo04 and Grifo06 – are used to generate high resolution maps of the subsoil obtained through the emission of sound waves. The company also uses the processing capacity of partner institutions like the Alberto Luiz Coimbra Institute for Graduate Studies and Engineering Research (COPPE-UFRJ). Many Petrobras projects are headed by researchers at the Center for Advanced High-Performance Computing (NACAD) at COPPE and at the Petrobras Research Center (Cenpes). In July 2016, COPPE inaugurated the Lobo Carneiro, a U.S. made SGI supercomputer with 226 teraflop capacity – it can perform 226 trillion mathematical operations per second. Just like the Santos Dumont computer at the LNCC, the device will also serve the High-Performance Computing for Energy Project (HPC4E) that brings together Brazilian and European Union research institutions and companies engaged in improving energy sector efficiency. “In addition to clients in the oil and gas industry, like Petrobras, many of our users operate in the field of engineering, especially in investigations of different forms of energy like solar and wind,” explains Guilherme Horta Travassos, a professor in the Systems and Computing Program at COPPE-UFRJ and president of the NACAD management committee. The Lobo Carneiro cost $4.2 million. Funds came from sources such as the UFRJ and the National Agency of Petroleum, Natural Gas and Biofuels (ANP).
Reinforcements from bahia
Another new feature of Brazil’s oil and gas industry recently began operations in Salvador, Bahia. It is the Yemoja (Yemanjá, goddess of the sea, in the Yoruba language), manufactured by SGI. As one of the fastest computers installed in Latin America, it was purchased in 2015 as a result of a partnership between SENAI of Bahia and BG Brasil (Shell). There are projects underway aimed at innovations in oil and gas, but SENAI-CIMATEC is not disclosing information about them. With 405 teraflops of processing capacity, the Yemoja stood 95th in the TOP500 when it began operations, and is now at position 323 in the ranking. “A second supercomputer for studies in biomedicine, robotics, image processing and alternative energy will also be installed,” says engineer João Marcelo Silva Souza, leader of the high-performance processing field at SENAI-CIMATEC.
Another recent investment in high performance computing has benefited the Center for Research in Mathematical Sciences Applied to Industry (CeMEAI), one of the FAPESP-funded Research, Innovation and Dissemination Centers (RIDCs), which inaugurated the Euler computer cluster on the USP São Carlos Campus in July 2015. Purchased at a price of R$4.5 million, the system is composed of 104 networked computers, with enough capacity to perform 47 trillion operations per second. “It’s one of the fastest computers ever installed at a São Paulo university,” says José Alberto Cuminato, a professor at USP’s Institute of Mathematical Sciences and Computation (CMC) and CeMEAI director.
Named in honor of Swiss mathematician and physicist Leonhard Paul Euler (1707-1783), the system is used for fluid flow simulations, with applications in petroleum and aerospace engineering, in complex network engineering, such as the propagation of epidemics, and in nanostructure engineering. Because it is multi-user equipment funded by FAPESP, other institutions in the state of São Paulo are authorized to use it as well. Some of the main users of the Euler are researchers at the Technological Institute of Aeronautics (ITA) in São José dos Campos, the Federal University of São Carlos (UFSCar), the National Synchrotron Light Laboratory (LNLS) and the University of Texas.
One group of ICMC-USP researchers uses the cluster in partnership with Petrobras to simulate oil extraction, refining and combustion. In March 2017, the Euler is scheduled to be updated for the first time, at a cost of R$2 million. “It’s not enough to have the money needed to purchase a supercomputer. Operation of this type of equipment entails expenditures on maintenance and power. An institution needs to provide for all of this in its budget,” says Cuminato, reminding us that the Euler power bill is paid by USP.
To expand its processing capacity, in 2012, USP signed an agreement with Rice University that has a computer donated by IBM: the Blue Gene/Q, with a capacity of 189 teraflops. “USP contracts the services annually and uses approximately 30% of the supercomputer capacity on studies in fields such as engineering, physics, astronomy and atmospheric sciences,” says João Eduardo Ferreira, superintendent of information technology at USP. Nearly 41% of USP’s share in the Blue Gene system is used by researchers from the Physics Institute; 18% is used by the Polytechnic School (Poli); and 17% is used by the Institute of Astronomy, Geophysics and Atmospheric Sciences (IAG). In all, USP spends close to US$500,000 per year to obtain access to the Blue Gene/Q. Two computing clusters at USP, named Águia (45 teraflops) and Lince (70 teraflops), complement the work of the U.S. machine. Ferreira says USP’s partnership with Rice University took into consideration the sustainability factor of the supercomputer. The Blue Gene/Q is one of the world’s most economical systems and ranks highest on the Top Green 500, another list that identifies the most energy efficient systems.
Besides its high price tag and relatively short useful life, a supercomputer is expensive to maintain. Travassos at UFRJ says that the costs involved with power and maintenance were important factors in the decision to purchase the Lobo Carneiro supercomputer. “Even though it is very efficient, the electricity bill could be as high as R$60,000 a month if the system is fully used,” he says. Concern about the economics of computers is seen all over the world. “Supercomputers carry hefty maintenance and power costs,” says Pedro Dias of USP. The Santos Dumont, at the LNCC, did not begin full operations until June 2016. Between January and May, it operated only four hours per day and was not available to the scientific community, all due to a lack of funds. “The 2016 LNCC budget was cut in 2015. By June 2016, the issue was resolved, and now the Santos Dumont is used in 42 projects involving topics such as nanotechnology, renewable energy, civil engineering, the development of vaccines to fight the dengue and Zika viruses, software production and astronomy,” says Gadelha. The system uses 0.8 megawatt (MW) and the monthly electricity bill runs R$350,000 to R$400,000. The Santos Dumont and the Lobo Carneiro are the only Brazilian systems that are on this year’s Top Green 500.
Purchased for approximately R$45 million from the French company Atos/Bull, the Santos Dumont consists of three distinct computing systems with the capacity to reach 1.1 petaflops. This means the computer is capable of performing one quadrillion mathematical operations per second – a speed that is approximately one million times faster than an ordinary notebook computer. It is the centerpiece of the Brazilian National System for High-Performance Computing (SINAPAD), a network that consists of nine National High-Performance Processing Centers (CENAPADS), operated by universities and research institutes. COPPE’s NACAD is one of these interconnected centers that can be utilized remotely by researchers all over the country. Established in 1994, the São Paulo CENAPAD is located at the University of Campinas (Unicamp) and has already been used by researchers from 18 Brazilian states: 36% from São Paulo, 25% from Rio Grande do Sul and 13% from Minas Gerais. “Most users are working in mathematics, physics, chemistry, geosciences and biology,” says José Augusto Chinellato, a professor at the Gleb Wataghin Physics Institute of Unicamp and coordinator of the SP-CENAPAD. A total of 172 projects are being carried out by 502 researchers from institutions all over Brazil. “The SINAPAD has become a high-performance computing network for universities and research institutes,” Chinellato explains.
Pedro Dias notes that Brazil has made it a priority to disperse its resources for high-performance computing. “It opted for purchasing a large number of small machines,” he explains. The United States did just the opposite: there are large computational centers scattered throughout the country, meeting the demands of research and industry. Dias says that the lack of more robust systems explains why many Brazilian researchers seek out computers installed abroad, through scientific collaborations, or by contracting processing services from companies like Google and Amazon. According to the LNCC’s Gadelha, the model adopted in Brazil is a legacy dating back to the late 1980s, a time when Brazil did not have the technology capable of ensuring that researchers in distant areas of the country would be able to remotely use a supercomputer installed in the Southeast. “Consequently, institutions from various states demanded funds to acquire their own high performance computers,” he explains.Republish