New robotic vision system processes event-camera data up to 440 times per second

US researchers have developed an artificial intelligence tool that can process robotic vision data at up to 440 times per second, potentially allowing drones, autonomous vehicles, and other robots to respond faster to their surroundings.

Developed by engineers at the University of Pennsylvania, the system, called Fast Feature Field (F³), processes information from event cameras, which detect changes in light rather than capturing conventional images.

In tests, algorithms using F³ operated roughly two to five times faster than previous approaches. The system processed high-definition event-camera data 120 times per second and reached 440 times per second at lower VGA resolution.

The researchers say the technology could help robots recognize objects, estimate distances, and track movement more efficiently, particularly in challenging lighting conditions.

Processing millions of signals

Unlike conventional cameras that capture complete images at fixed intervals, event cameras continuously detect changes in light intensity at individual pixels.

This allows them to capture rapid movements without recording entire frames. However, interpreting their output remains difficult because the cameras generate millions of individual signals rather than recognizable images.

Existing approaches typically require specialized algorithms for different tasks, limiting their wider use in robotics.

The Penn researchers addressed this problem by developing a neural network that learns to identify useful patterns in event-camera data.

Instead of programming the system separately for object recognition, motion tracking, or distance estimation, the team trained F³ to predict future events based on previous signals.

“To accurately anticipate future events based on past ones, the network must learn to represent both the structure of a scene and the motion taking place within it,” said Richeek Das, the paper’s first author.

This approach allows the system to extract information about a scene and its movement without requiring a separate processing method for every application.

The researchers also found a way to reduce unnecessary calculations.

Although event cameras can generate tens of millions of signals every second, approximately 95% of possible positions in the data contain no events.

By processing only locations where changes occur, F³ avoids spending computing resources on empty areas. This optimization helped the researchers achieve the processing speeds needed for real-time robotic perception.

Testing across robotic platforms

The team evaluated F³ using event-camera data collected from cars, four-legged robots and flying drones.

Tests included object recognition, distance estimation, and motion tracking across environments ranging from nighttime urban streets to crowded indoor spaces.

According to the researchers, F³-based approaches achieved state-of-the-art performance on multiple perception tasks while operating substantially faster than earlier methods.

The system also allows existing computer-vision software to be adapted for event-camera data, potentially reducing the need to develop specialized algorithms for individual applications.

The researchers are now incorporating F³ into autonomous drones designed to navigate cluttered and poorly lit environments.

Eventually, they hope to integrate the technology directly into event-camera hardware, allowing raw signals to be converted into usable visual information without requiring separate processing software.

“Our goal is to make using an event camera as straightforward as using a conventional one,” said Pratik Chaudhari, associate professor at the University of Pennsylvania.

The researchers believe this approach could make event cameras more practical for autonomous systems that must interpret rapidly changing environments.

The findings were published in the journal Science Robotics.