Technology and Research Products and Services
News
Home About Us Resources
Artificial Insect Vision
LadybugTM optic flow sensor

Mass: 4.5 grams, including optics, imaging, processing, and interface

Frame rate: Maximum 3000 fps

Currently being flown on test robotic aircraft

Status: Built in small quantities since August 2002
A significant part of our research involves designing visual microsensors which will provide robotic aircraft with the abiltiy to visually sense the environment. We use the visual systems of flying insects as a source of inspiration for our work. Why? Because these insects are able to navigate through complex, cluttered environments that are littered with obstacles, and yet do so with tiny brains and low resolution eyes. The resolution of a typical insect eye ranges from several hundred to several thousand pixels (depending on the insect and on how you define one "pixel"), which is significantly less than that of most conventional machine vision systems. However insect vision is anything but similar to human-built machine vision. Part of this difference is that insect visual systems are heavily populated with structures that respond to optic flow. More fundamentally, the difference is one of architecture.

A layman's comparison between conventional human vision and insect vision

In conventional machine vision, pixels essentially accumulate charge like jars accumulate rainwater, only to have the accumulated value digitized in a regular, brute force fashion. Grab yourself a million jars, and lay out a huge square of jars a thousand wide and a thousand long. If you are a climatologist, then this square may be spread out enough to span a continent. Every day take each jar, measure how much water has entered each jar, and then empty the jar's contents. This is essentially what happens in just about every imaging chip in your digital camera, camcorder, or cell-phone camera, except that photons are accumulated rather then raindrops. (If you want color, then just add colored filters on top of the jars so that each one collects only red, green, or blue raindrops...) All of this information is then processed using a powerful digital computer.

Insect vision, and animal vision for that matter, is quite different. Let us carry on with our metaphor of a pixel as a jar. Each jar has some amount of intelligence and is able to communicate with it's neighbors. This allows a neighborhood of jars to collectively widen or narrow their openings, so that jars in a region with much rainfall tend to have narrow openings and jars in dry regions have wider openings. The dry-region jars may also have "lids" to prevent water from evaporating in between rain storms. Individual jars can also figure out if, say, it is collecting twice or one half as much water as it's neighbor. Individual jars can also react alone to sudden changes in rainfall from one minute to the next. Each individual jar is thus able to output significantly more information than a jar in the dumb array above. The net result is that you can use far fewer jars, and depending on what specific information you need, still get the required information.

Electronically speaking...

In conventional machine vision, a CMOS or CCD imager samples the visual field at tens to thousands of times per second, generating kilobytes or megabytes of data for each frame. The resulting information is dumb data that must be heavily processed to produce any intelligent result. Such systems are generally heavy and power hungry, and don't handle all types of visual environments. For example, most of these systems cannot handle low contrast environments, and are not able to image an area that has a wide range of light levels. Moore's law has sufficiently advanced so that real-time machine vision is possible for more applications, however it is still not possible to implement anything that is practical and would fit on a micro air vehicle. This situation is even more acute when one is trying to measure optic flow: By definition, optic flow is both a spatial and temporal phenomena, which means that spatio-temporal features ultimately need to be extracted from the video stream. As a result, machine vision systems designed to measure optic flow have especially high computational requirements.

In insect vision, the compound eye elements grab an "image" that has a 2D topology somewhat similar to that in a digital camera. However the retina in an insect is anything but "dumb": Neural circuits in the retina adapt to the varying light levels, while other circuits adapt to changes in intensity. Further down the processing stream, other types of circuits detect edges, detect motion ultimately measure optic flow, and visually recognize objects of interest. To achieve this in a small package, only the more important information is passed on. All of this processing is performed using live neurons operating in primarily the analog domain.

Centeye's optic flow sensors

Centeye's approach to performing machine vision is effectively a hybrid between the two methods of processing. Our sensors contain one or more "vision chips", or smart imaging chips. A vision chip is an integrated circuit with both image acquisition and image processing on the same die. The vision chip performs front-end image processing on the image using proprietary mixed-signal (analog and digital) circuitry. Back-end processing is performed with an off-the-shelf microcontroller or DSP chip. The circuitry and algorithms are heavily inspired by the visual systems of insects. However these circuits are also "engineered" to give good results when implemented in silicon and software. It should be noted that although we take inspiration from biology, we do not blindly copy it without first extracting the important "gem" of knowledge.

Our approach allows a complete optic flow sensor to be integrated into a tiny package weighing just several grams, and yet be able to process imagery at several thousand frames per second and measure optic flow in real-world environments. The analog processing performed on the vision chip also allows operation in difficult visual environments. For example, our sensors are able to measure optic flow in low contrast (<2%) environments, and are also able to operate in light levels of under one lux. The high frame-rate also allows optic flows of even a hundred radians per second to be detected. Most important, our methods currently allow circuitry to be implemented in a package weighing less than five grams, allowing them to be used on small, robotic aircraft.

Most of the inner workings of current sensors are proprietary. However you can download papers that describe some of our earlier work in the downloads section. You can also visit the links page to view related work performed by others in academic institutions.
Acknowledgements:

We gratefully acknowledge the support of DARPA and the U.S. Naval Research Laboratory. Work performed 2000 and earlier was performed at the U.S. Naval Research Laboratory as part of Barrows' graduate work at the University of Maryland at College Park. Much of the work performed at Centeye, 2001 and later, was supported by DARPA, under DSO's Controlled Biological and Biomimetic Systems program.