To extract the image of a Formula 1 racing car speeding at the Interlagos motor racing track, and to apply it in another scenario or to highlight the image of a cyclist cycling in a city and to superimpose it onto the arid and cold landscape of the Antarctic continent, are two examples of the special effects obtained by digital video editing through a piece of software that is in its final phase of development by the company SDC Engenharia, Sistemas e Eletrônica of São Paulo. “We have created an unprecedented computer tool, which will be valuable to the professionals that work with editing and post-editing of images”, says Robert Liang Koo, a director with SDC and the coordinator of the project entitled Assisted Segmentation of Images and Digital Videos, financed by FAPESP within the Small Business Innovation Research Program (PIPE).
A result of more than two years of extensive work, the software will be a simple and quick tool for helping in the process of the segmentation of digital images which will be very useful in recordings in the Digital Video Disc (DVD) system or for the future digital TV. This technology of segmentation consists in isolating the pixels (the smallest tonal point of a digital image system) which represents the object of interest from the others in the background of the image. With it, it will be possible to make a cut out of any object in any type of setting, without the need for the recordings to occur in the environment previously prepared, as is currently necessary. In other words, the software, dubbed the SDC Pronto Video (Ready Video), will allow for the segmentation of videos with much more complex images, independently from the colors and the properties of the background and of the image to be highlighted.
Image gluing is a resource often used by television programs throughout the world. For example, the weather forecasting of the National Journal of the TV network TV Globo, shows the announcer and in the background an animated image of the map of Brazil. However, for the scene to go on the air, the filming of the announcer and of the background map are done separately. First, the “weather man” is filmed in front of a blue colored wall, full of marks (nails or adhesive tape) that shows where, supposedly, the Brazilian States are located and the meteorological aspects to be reported – cold fronts, masses of dry air, warm air, rain etc. Afterwards, an animation is done in which the meteorological behavior in the Brazilian territory is carried out.
Blue scenario
In the final mounting, the image of the meteorologist is isolated, cut out from the blue background and glued over the animation. This special effect is only possible thanks to a sophisticated technique of editing of images known as chromo-key or blue background. In order to free the cutting from distortions, the recording is only carried out in a controlled environment. The scenario where the announcer is found is completely blue, and the illumination is orientated to accentuate the contrast between him and the background scenery.
Agility and rapidity
According to Koo, who is also a professor of Computer Science at the Catholic Pontificate University (PUC) of São Paulo, the developed software is vastly superior to similar tools available on the market. The segmentation of the images in a generic environment is so difficult that, in the majority of cases, it is done manually, with the user outlining the object, as if it were a mask, for a later re-cutting. This segmentation, done with an available tool in innumerable drawing software, such as Photoshop, is carried out according to he visual perception of the user and takes up a lot of time. “What we are now offering is a semi-automatic tool, extremely agile and very easy to be manipulated”, he explains.
Its operation is very simple. Initially the user must specify, in the first frame of the image, the objective that he intends to isolate. For this to happen, it is sufficient to position the cursor on top of the object and, by pressing the left hand button of the mouse, to make a small mark on the object. This is the internal marker, which takes on a red color. Next, he makes an external marker in blue in the areas of the object’s background, pressing the button on the right of the mouse and making another inscribed mark. “Two simple markers, one external and the other internal, suffice for the Ready Video to process the segmentation of the image”, explains Koo. It will find the image outline through the difference in the pixels pattern, which is expressed by he change of color, texture and sheen of the object to be cut out from its background.
Once the initial marking has been done, the Ready Video projects the outline of the object and of the background. Using calculations of the estimation of movement, these outlines are thrown onto the following film frames. Should small distortions during the cutting take occur – for example, if part of the object, such as the nose of a person, has not been highlighted -, the user interferes in the process, making the necessary corrections, erasing some markers or introducing new ones. In the case of a nose, it would be enough if there was an internal marker at this point and it would be recovered in the cut images. “This is the reason why the segmentation is semi-assisted and not totally automatic”, explains Koo.
The development of the technology was done with the assistance of the researchers Roberto Lotufo, of the Electrical Engineering and Computing College of the State University of Campinas (Unicamp); Junior Barrera, from the Mathematics and Statistics Institute of the University of São Paulo (USP); Alexandre Falcão, from the Computing Institute of Unicamp; and Rubens Machado, from the Renato Archer Research Center (Cenpra) of Campinas. In Lotufo’s opinion, the Ready Video will have, initially, two basic functions. “It will be used to carry out special effects on video, such as the withdrawal of backgrounds, and for the production of videos in the format MPEG-4, a new digital video format in which the various objects in the scene are codified separately”, he explains. “The new software of the SDC will help MPEG-4 producer in the separation of various elements present in the image.”
The product is expected to be on the market within the year. However, the prototype, concluded last year, can already be found fully able to be operated by the final user. “We are only making some minor improvements so that it become a competitive product on the international market”, explains the director of SDC. “Among other things, we are requesting a patent with the assistance of FAPESP and an improvement in the algorithm of the estimation of the movement of the propagation of the markers”, he emphasizes. This algorithm is the one that is responsible for the precision of the cutting of the object. The software will cost between US$ 50 and US$ 100 and its sale will be done only through the Internet. For anyone interested, they can make a download of the program directly from SDC’s (www.sdc.com.br), through a fee payment.
On sale through the Internet
Established in 1987, the SDC is a national company specializing in the integration and supply of equipment, software and solutions for industrial automation, laboratory automation, computerized vision, image processing, electronic commerce and telecommunications. In 2000, it grossed US$ 2 million, 10% of it referring to overseas sales. One of its commercial successes is the software SDC Morphology ToolBox, launched in 1999 and today exported to more than fifty countries.
“This software is a powerful collection of morphology tools for the segmentation of images and is used in the delineation of images of medical examinations”, explains Robert Koo. “The Morphology ToolBox was the base for the development of Ready Video.” Everything seems to indicate is that the new software has everything to be a commercially very successful product and turn itself into an important tool in the production of digital films.
The project
Assisted Segmentation of Images and Digital Videos (nº 97/13306-6); Modality Small Business Innovation Research Program (PIPE); Coordinator Robert Liang Koo – SDC; Investment R$ 91,000.00 and US$ 35,000.00
