Note: The video below does not describe our Stonyman chip.
The heart of Centeye’s visual microsensor technology is the “Vision Chip”. A vision chip is similar to a regular CCD or CMOS imaging chip in that it is placed at the focal plane of camera lens assembly to convert an optical image into electrical signals, i.e. to grab an image. However unlike a CCD or CMOS imager, which merely digitizes the image, a vision chip performs image processing on the raw image using a mixture of analog and digital circuitry. The output of a vision chip is thus a pre-processed version of the image. The figure below summarizes the difference between a machine vision system using a generic imager and one using a vision chip.
The nature of the “pre-processing” performed on a vision chip can vary widely depending on the specific implementation and application. Typically the circuits implemented are optimized for a relatively narrow class of applications and are not meant to support general image processing functions. As a tradeoff, these narrow applications are implemented extremely well. However with a clever use of circuits and processing algorithms, a wide variety of applications can be addressed. Sample functions that can be supported using a vision chip include contrast enhancement, motion detection, feature detection, and so forth. The overall goal of this processing could be to “condition” the analog image signal before digitizing it with an ADC, or to throw away redundant information to keep the useful information and thus reduce the load on post-processors. When executed properly, the use of a vision chip can greatly improve the performance specifications of a machine vision system, such as sensitivity and speed, while reducing basic physical requirements like size, mass, and power consumption.
A significant part of our research involves designing visual micro-sensors which will provide robotic aircraft with the abiltiy to visually sense the environment. We often use the visual systems of flying insects as a source of inspiration for our work. Why? Because these insects are able to navigate through complex, cluttered environments that are littered with obstacles, and yet do so with tiny brains and low resolution eyes. The resolution of a typical insect eye ranges from several hundred to several thousand pixels (depending on the insect and on how you define one “pixel”), which is significantly less than that of most conventional machine vision systems. However insect vision is anything but similar to human-built machine vision. Part of this difference is that insect visual systems are heavily populated with structures that respond to optic flow. More fundamentally, the difference is one of architecture.
A layman’s comparison between conventional human vision and insect vision:
In conventional machine vision, pixels essentially accumulate charge like jars accumulate rainwater, only to have the accumulated value digitized in a regular, brute force fashion. Take a million jars, and lay out a huge square of jars a thousand wide and a thousand long. This “square” may be laid out to span a field or spread out to span a continent, depending on what area you wish to monitor. Every day take each jar, measure how much water has entered each jar, and then empty the jar’s contents. Record these values in a table having a thousand rows and a thousand columns. This table is effectively an image, while each table entry is a pixel value. This is essentially what happens in just about every imaging chip in your digital camera, camcorder, or cell-phone camera, except that photons are accumulated rather then raindrops. (If you want color, then just add colored filters on top of the jars so that each one collects only red, green, or blue raindrops…) All of this information is then processed using a powerful digital computer.
Insect vision, and animal vision for that matter, is quite different. Let us carry on with our metaphor of a pixel as a jar. Each jar has some amount of intelligence and is able to communicate with it’s neighbors. This allows a neighborhood of jars to collectively widen or narrow their openings, so that jars in a region with much rainfall tend to have narrow openings and jars in dry regions have wider openings. The dry-region jars may also have “lids” to prevent water from evaporating in between rain storms. Individual jars can also figure out if, say, it is collecting twice or one half as much water as it’s neighbor. Individual jars can also react alone to sudden changes in rainfall from one minute to the next. Each individual jar is thus able to output significantly more information than a jar in the dumb array above. The net result is that you can use far fewer jars, and depending on what specific information you need, still get the required information.
In conventional machine vision, a CMOS or CCD imager samples the visual field at tens to thousands of times per second, generating kilobytes or megabytes of data for each frame. The resulting information is dumb data that must be heavily processed to produce any intelligent result. Such systems are generally heavy and power hungry, and don’t handle all types of visual environments. For example, most of these systems cannot handle low contrast environments, and are not able to image an area that has a wide range of light levels. Moore’s law has sufficiently advanced so that real-time machine vision is possible for more applications, however it is still not possible to implement anything that is practical and would fit on a micro air vehicle. This situation is even more acute when one is trying to measure optic flow: By definition, optic flow is both a spatial and temporal phenomena, which means that spatio-temporal features ultimately need to be extracted from the video stream. As a result, machine vision systems designed to measure optic flow have especially high computational requirements.
In insect vision, the compound eye elements grab an “image” that has a 2D topology somewhat similar to that in a digital camera. However the retina in an insect is anything but “dumb”: Neural circuits in the retina adapt to the varying light levels, while other circuits adapt to changes in intensity. Further down the processing stream, other types of circuits detect edges, detect motion ultimately measure optic flow, and visually recognize objects of interest. To achieve this in a small package, only the more important information is passed on. All of this processing is performed using live neurons operating in primarily the analog domain.
Centeye’s optical flow sensors
Centeye’s approach to performing machine vision is effectively a hybrid between the two methods of processing. Our sensors contain one or more “vision chips”, or smart imaging chips. A vision chip is an integrated circuit with both image acquisition and image processing on the same die. The vision chip performs front-end image processing on the image using proprietary mixed-signal (analog and digital) circuitry. Back-end processing is performed with an off-the-shelf micro-controller or DSP chip. The circuitry and algorithms are heavily inspired by the visual systems of insects. However these circuits are also “engineered” to give good results when implemented in silicon and software. It should be noted that although we take inspiration from biology, we do not blindly copy biology. We are more interested in utilizing the basic principles that allow biological systems to function.
Our approach allows a complete optic flow sensor to be integrated into a tiny package weighing as little as a few tenths of a gram, and yet be able to process imagery at several thousand frames per second and measure optic flow in real-world environments. The analog processing performed on the vision chip also allows operation in difficult visual environments. For example, some of our sensors are able to measure optic flow in low contrast (<2%) environments and are also able to operate in light levels of under one lux. The high frame-rate also allows optic flows of even a hundred radians per second to be detected. Most important, our methods currently allow circuitry to be implemented in a package weighing less than five grams, allowing them to be used on small, robotic aircraft.