Development of robust image-matching technology that can be used in various real-world situations
Shun'ichi Kaneko, Doctor of Engineering,
Professor of the Graduate School of Information Science and Technology, Hokkaido University,
Division of Systems Science and Informatics's Research Group of Informatics for System Creation
Robust image-matching technology that can be applied in industry
Dr. Kaneko's laboratory is one of the largest in the Graduate School of Information Science and Technology, and has undertaken many joint research projects with companies. Could you tell us about this?
Dr.Kaneko: Our laboratory conducts research and development on system information sciences, including an image-matching technology that can be used in robotics and mechatronics. Image matching is like a picture-matching game - it is a technology to find designated objects or compare and identify them with other images using a camera mounted on a robot. Image-matching technology is used in a wide range of fields, including assembly processes and visual inspections of production lines, video surveillance of roads and parking lots and general-purpose image measurement systems, and expansion of its application range and improvement of image-recognition techniques are required. Our laboratory aims to develop the technology to benefit the real world through industrial application.
Important factors in image matching are brightness and occlusion. As an example, while the method known as CC (correlation coefficient) is a common image-matching technology used in various fields, it may not be able to distinguish objects if there are changes in their status, position, posture or size, illumination variation, occlusion or other unfavorable conditions. With ISC (increment sign correlation), which is our own method, highly robust image matching can be achieved even with local brightness variation or occlusion (Fig. 1, Ex. 1).
Robustness is important because the robots of the future are expected to perform the jobs of humans over a range of places and settings of society. While robots that make precision products in unmanned factories are already in practical use, there will be an increasing incidence of humans and robots coexisting in the same space, such as robots for cleaning, nursing and disaster relief. It will then be necessary for robots to have the strength to function correctly by accurately understanding the presence of people and objects and appropriately identifying their targets under various conditions, such as well-lit and dark places or adverse environments full of dust or smoke.
Pursuing robustness to ensure operation in the real world rather than the realization of accuracy and reliability in the laboratory - that is truly a direction toward industrial application.
Expansion into industry, agriculture, services and various other fields
What is the current progress of the expansion to industrial application?
Dr.Kaneko: We have conducted a number of joint research projects with companies. Since a variety of methods are necessary depending on the type and purpose of the subject, some of these are studied concurrently. Offers for joint research are basically received after presenting the methods we have developed together with students in our laboratory for evaluation. There are many application fields, ranging from a vehicle vicinity monitoring system using an onboard camera to speed measurement for agricultural tractors, security and amenity surveys and measurement of mites.
One such method is known as orientation code matching (OCM). This can track an object correctly even if the brightness or direction changes in its transfer process. In Fig. 2, for example, the method can trace the moving object accurately even if it moves through an unlit section or changes direction. OCM is adopted for a vehicle vicinity monitoring system using the in-car image-processing LSI ViscontiTM technology developed by Toshiba (Ex. 2).
Another thing I would like to show you is M-ICP, which is a technology to collate and match three-dimensional image data. Figure 3 shows an attempt to recognize and overlap the shapes of two data sets prepared by measuring coffee makers with a three-dimensional sensor. There are some differences between the two data sets. While the conventionally used matching technology (ICP) cannot grasp the differences between the objects correctly or overlap them completely, our M-ICP technique can overlap them completely using an algorithm known as M-estimation. It recognizes the differences between the two images, and the computer itself surmises and supplements the different parts. This can be applied, for example, to quality inspection for manufactured products. It is expected to be applied to a system that can measure pressed products and check against the sample data to see if there are any flaws or defects.
Such a system cannot be realized using only image-matching technology. It must be developed in cooperation with other fields, such as camera and sensor technologies to identify people or things and technologies to move robots. To this end, our laboratory has an integrated research system with robotics, mechatronics, human sensing and other laboratories, and can conduct research by combining their respective realms of expertise. I think this is another advantage of this laboratory.
Establishment of a research system with emphasis on research papers, with the motto, "students are co-researchers"
What do you attach particular importance to in joint research with companies?
Dr.Kaneko: While research for industrial application is conducted by request from companies or industries in many cases, our laboratory places the highest priority on the development of excellent original technologies through collaboration between students and teachers and the presentation of such technologies in research papers. We maintain a stance of undertaking joint research after letting companies know about our achievements through seminars and presentations. This stems from our belief that the highest priority in research activities at the university should be given to the education and training of students.
So you place more importance on training students before responding to requests from companies or society.
Dr.Kaneko: Students are as much co-researchers as receivers of education. So, all joint research projects with companies have student participation, and the funds obtained from those projects are put toward the operation of the laboratory. This means that current students depend on technologies created by their predecessors, so they are devoted to research while thinking about what they should do and leave for younger students. Outstanding human resources are developed through research led by students, and the results are returned to the general public in the form of research papers. This then leads to joint research with companies, which in turn supports Japanese industry with world-class cutting-edge technologies. That, I think, is the role of us researchers and the task of a national university.
Our laboratory also places great importance on transparency. We intend to disclose a variety of information, including not only budgets and research content but also the results of joint research with companies. If a university laboratory maintains transparency and makes all information visible, cooperation within and outside the university will become smoother, and opportunities for mutual improvement will increase. I expect this to eventually lead to a higher level of technological development.
Explanation
Explanation 1: Brightness and occlusion
Among the unfavorable conditions in the actual application of image-matching technology, brightness and occlusion have the greatest impact. In CC, which is a common image-matching technology, pixels from two digital images are compared, and parts with a certain degree of brightness matching are identified as identical images. Thus, unfavorable conditions such as the use of a flashbulb or a change in the sun's position may cause variations in the brightness of images of the same object, and matching may not be performed. In Fig. 1-A (a doll), an experiment to make an image-processing robot search for and match the same part as the lower right image (the doll's head) was conducted. The face of the doll in the target image looks darker than the sample due to the use of a flashbulb. This difference in brightness could not be detected by CC or SSD, which indicated parts different from the sample. The increment sign correlation (ISC) developed by our graduate school can read the difference in brightness and specify an almost correct position. In Fig. 1-B (Japanese playing cards), ISC can identify the correct card even though there are parts covered with overlapping cards (occluded parts).
Explanation 2: Image matching by orientation code matching
When a snack container is close to the camera, some parts become shaded. While SSD cannot continue tracking the container as it fails to recognize the shaded parts, a method known as OCM (orientation code matching) can. This technique uses information on the direction of contrasts near the pixels, and can ensure robustness against local changes in brightness.