Robot learns complex 12-step tasks in 5 minutes, replans when things go wrong

A robot learned to complete complex tasks involving up to 12 steps after receiving just five minutes of video demonstrations, while also adapting when people interrupted its actions or individual attempts failed.

Researchers at the University of Pennsylvania developed the system, called SymSkill, to help robots combine simple skills into longer tasks without requiring demonstrations of every possible sequence. The approach was tested both in simulation and on a physical robotic arm.

Instead of relying on increasingly large training datasets, SymSkill combines symbolic planning with a physics-based approach called dynamical systems. Together, they allow the robot to decide what actions are needed and determine how to physically carry them out.

The researchers demonstrated the system on tasks such as moving a banana from a covered pan onto a plate. The robot had to remove the lid, pick up the banana and place it on the plate, while adjusting its actions if a person interfered or an attempt failed.

Five minutes teaches sequences

SymSkill separates robot learning into two connected levels. At the higher level, symbolic planning represents objects, actions and their relationships, allowing the robot to determine the sequence needed to reach a goal.

The second level controls movement. A dynamical systems approach determines how the robotic arm should execute individual actions and allows it to react when its environment changes, such as when a person gets in its way.

Crucially, the system learned its basic capabilities from only five minutes of demonstrations. Once individual skills had been learned, SymSkill could recombine them into longer sequences rather than requiring researchers to demonstrate each complete task separately.

“What makes this exciting is that the robot doesn’t have to experience every possible version of a task before it can perform it,” said lead author Yifei Simon Shao, a doctoral student in Nadia Figueroa’s lab at Penn Engineering.

“Once it has learned all basic skills and understands how they relate to the world around it, it can start putting those skills together in ways it hasn’t seen before. That’s an important step toward robots that can actually operate in the unpredictable and highly variable environments of everyday life.”

Robots recover from failures

The ability to respond when a task goes wrong is important for robots operating around people. A machine following only a fixed sequence could fail if an object is moved or someone blocks its path.

SymSkill instead allows the robot to re-plan and continue. If a person interferes or one step fails, the system can adjust its plan and retry rather than requiring the entire task to be restarted.

The researchers demonstrated tasks containing as many as 12 steps without providing additional demonstrations for those longer sequences.

“We can make robots more capable without simply throwing more data at the problem,” said Figueroa, an assistant professor in mechanical engineering and applied mechanics at Penn Engineering.

Her lab is also investigating how previously learned robot movements can be reused to reach new destinations. A separate approach called Gaussian Graph turns demonstrated movements into a map of possible paths through a workspace, allowing a robot to create routes it was never explicitly shown.

Researchers now plan to combine these approaches with language models so robots can better interpret human instructions. They are also exploring their use with dual-arm and humanoid robots and machines that can move through an environment while manipulating objects.