EDUARDO CESARThe time it took a supermarket cashier in the city of Campinas to find the code of fruits and vegetables on a printed list drew the attention of IT professor Anderson de Rezende Rocha, from the State University of Campinas (Unicamp). Whereas products with barcodes on the packaging were rung up quickly, the identification of the vegetables slowed the line down. That was when he had the idea of developing a system to distinguish bulk produce sold by weight, which involves products that are difficult for a check-out electronic reader to identify because they have no code.
The solution found by Rocha along with researchers Daniel Hauagge, Jacques Wainer and Siome Goldenstein, also from Unicamp’s Computer Science Institute, was to develop a system with a camera on the check-out scale to analyze the images of the product to be classified. The invention takes into account several types of information, such as the color, shape, texture, silhouette, and appearance of fruit and vegetables. It then combines them in order to create a powerful indicator for each product. The software program that the researchers developed can tell the various produce items apart based on the combination of the characteristics of each.
Rocha explains that the camera only captures the product’s image. The information is extracted by a system using algorithms (mathematical calculations) to process the images and recognize patterns. Although it may sound complicated, the functioning of the invention, which Rocha developed during his doctorate, under the guidance of professor Siome Goldenstein and with a FAPESP grant, is simple. It involves two stages: training and testing. During the training, several images of the products sold in the supermarket are supplied to the system in such a way that it can identify the descriptive characteristics of each. This is done by identifying the specifications of each type of fruit or vegetable. “Then, each type of product is trained [compared] to another one, rather than doing them all in one go”.
The system uses a method that divides the problem of classifying many different products into smaller, more manageable problems. “This can be better understood if one considers a situation with three classes, such as three different types of fruit, for instance, oranges, apples and pineapples”, says Rocha. In this example, one can define two classes per time and say that one of them will be considered as a positive virtual class and the other as a negative one. Put more simply, it means one is apples and the other is oranges. This is done for the several combinations of products, taken two at a time: “orange vs. apple”, “orange vs. pineapple”, and “apple vs. pineapple”. Several other possibilities may exist. For example, the system might be trained to compare one type of product against all the others. In this situation, one would have “orange vs. the rest”, “apple vs. the rest” and “pineapple vs. the rest”. “The important thing here is to treat the problem by splitting it into smaller parts”, says Rocha.
Candidates on the scale
When the system starts operating, the testing phase begins. With each image captured and supplied for classification, the program extracts the same set of characteristics. They are compared to those previously stored in the training stage. With this, the system can provide the cashier with a list of probable candidates for a particular fruit or vegetable. After the employee confirms which one it is, it is only necessary to check the product’s price per kilogram and multiply by the weight. This way of solving the problem is the system’s chief innovation, which has led to a patent request being submitted to INPI, the National Institute of Industrial Property. According to Rocha, the currently available systems are different and less precise. For example, there is a system called VeggieVision in the United States . “This system extracts the background, identifies the size of the objects regardless of how many there are and compares this with the references”, explains Rocha. “It is based on properties of color, texture and density, which demands extra information from the scale”.
Comparing the ratio of right answers, the Brazilian system beats VeggieVision. “The ratio of correct answers produced by the American program, showing the four most likely products to the cashier, is 95%”, says Rocha. “Ours, however, showing the two most likely products, is 99%”. According to Rocha, a fuller comparison should also take other factors into account. Yet another VeggieVision disadvantage is that it incorporates into the data purchasing equipment special mechanisms to deal with variations of lighting and the suppression of the reflections caused by light on the scales and on the plastic bags. In a real setting, such mechanisms may make it more expensive for the supermarket to adopt the product.
The next step will be to develop a physical prototype. For the time being, what has been developed is a software program and algorithms that identify fruit and vegetables. To test this system’s efficiency, Rocha and his team used a digital camera to capture 2,633 images of 15 different species, including onions, oranges, limes, watermelons, pears, apples, cashews, kiwis and potatoes, displayed for sale in the Campinas Supply Center (Ceasa). “At present we are negotiating a partnership with an American company to give continuity to the project”, Rocha reveals.
The objective now is to improve the stages of separation among varieties in the same type of product, making the system able to tell two different types of bananas apart, such as the nanica (dwarf Cavendish banana) and the prata, for instance. Additionally, the researchers would like to incorporate a learning process into the operation, i.e., they want the system to learn each answer confirmed by the cashier, in order to further improve the quality of future classifications. “The last stage of the project will be to integrate our system with those already running in the supermarkets based on barcodes and connected to the invoice printers”, explains Rocha.
The project
Classifiers and learning in the processing of computing vision and image (nº 05/58103-3); Type Doctoral grant; PhD supervisor Siome Klein Goldenstein – Unicamp; PhD candidate Anderson de Rezende Rocha – Unicamp; Investment R$ 95,443.92 (FAPESP)
Scientific article
ROCHA, A. et al. Automatic fruit and vegetable classification from images. Computer and Electronics in Agriculture (Compag). v. 70, n. 1, p. 96-104. 2010.
