MIT’s smart system helps robots understand what humans mean, not just what they say

MIT has developed an AI-powered system that can help robots understand human intentions, such as the implied desires of a person asking a robot to carry a cup of coffee on a cluttered table. The system, called FLEX (Flexible Language Guided Representation Editing for Adaptive Robot Behavior), is being presented at the annual Conference on Robot Learning this month. The researchers used simulated tasks with a robotic arm to test FLEX. In one example, the robot was asked to carry a cup of coffee over a cluttered table. The user said, “Don’t spill on the laptop!” FLEX identified the potential risk of spilling coffee on a laptop and used a large language model (LLM) to make an association between liquids and electronic devices. The robot then avoided the laptop and a nearby phone, even though the user had not mentioned the phone in the instruction.

This study was led by Assistant Professor of Aeronautics and Astronautics, Andreea Bobu. Co-authors include Postdoctoral Researcher Nathan Dennler of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), as well as Master’s student Audrey Lee SM ‘26 and NYU student Ilia Sucholutsky. Large language models (LLMs) have been trained on vast amounts of internet text and can associate objects and situations with related information. In the case of spilled coffee or water, for example, the LLMs associate these with potential damage to electronic devices. FLEX uses these associations to help robots better anticipate potential problems with objects that the user has not explicitly mentioned.

The system goes through three stages Add, Generalize, and Mask

In the Add stage, FLEX identifies objects and features that are to be avoided by the robot. For example, if a user said, “Don’t drop the bowling ball on my mug!” the mug would be added to the list of objects that the robot is to avoid. In the Generalize stage, FLEX utilizes the GPT-4o language model to generalize the objects and features that were added in the previous stage. For example, the mug would be classified as fragile, and thus all other fragile objects would also be avoided by the robot.

Finally, in the Mask stage, the robot’s standard computer vision system would be used to identify objects in the robot’s surroundings. Objects that are no longer present would be removed from the robot’s feature list, allowing the robot to update its decisions in real time based on changes to the environment. The entire process takes approximately five seconds, according to the researchers.

Simulations demonstrate broader reasoning abilities

The team tested FLEX in three activities serving coffee, moving a candle and carrying a bowling ball in simulated environments. The robot was able to follow instructions involving laptops, stacks of paper and mugs and extend these to include other objects in the same category (e.g. electronic devices, flammable materials and fragile items).

The MIT Generative AI Impact Consortium and the MIT Schwarzman College of Computing’s Social and Ethical Responsibilities of Computing program partially supported this research. Bobu said, “Household preferences are extremely heterogeneous. Thus, it is desirable for robots to learn people’s preferences with only a few sentences of instructions for various situations.” FLEX can help robots learn people’s preferences for various situations with only a few sentences of instructions for various situations.

Source: https://interestingengineering.com/ai-robotics/mit-ai-system-robots-human-instructions