NVIDIA-trained robotic hand at ETH Zurich crawls, types and picks up objects

Engineers at ETH Zurich have taught a robotic hand to crawl across the ground using nothing but its own fingers. The team, from the university’s Soft Robotics Lab, started with an off-the-shelf WUJI hand built for grasping objects and turned it into a self-propelled walker instead.

Researchers Amirhossein Kazemipour, Hehui Zheng and Robert Katzschmann fitted the five-fingered hand with a Raspberry Pi, an inertial measurement unit and a battery on its back. The added hardware let the hand operate untethered, bringing its total weight to about 1.8 pounds (818 grams). The 20 powered joints, four per finger, stayed factory standard, with the thumb positioned off to one side.

Turning fingers into legs

Walking on fingers instead of legs creates an unusual problem. Lifting any single finger to take a step removes one of the hand’s support points, so the remaining fingers must instantly compensate to keep it balanced.

To solve that, the team trained the hand inside NVIDIA’s Isaac Lab simulation platform using reinforcement learning. Researchers ran thousands of virtual trials at once inside a simulator tuned to match the real hardware closely. A neural network tracked joint angles, tilt, spin rate and the hand’s most recent command, then calculated target positions for all 20 joints. Rewarding the system for moving each fingertip toward a reference point, rather than copying reward structures built for four-legged robots, sped up training considerably.

Crawling across terrain

Once trained, the policy decided independently which finger to lift, how far to extend it and how to keep the rest of the hand steady. Researchers then tested the hand outdoors across 14 surfaces, including rubber mats, carpet, hardwood, gravel, grass, diamond plate and stone along Zurich’s Limmat River.

Operators supplied steering commands through a controller, and the hand’s own geometry nudged it back on course whenever it started drifting, an unplanned quirk the team came to appreciate. A separate recovery policy also let the hand right itself after a fall in 21 of 25 attempts, with most recoveries taking less than 20 seconds.

Multitasking while mobile

The same fingers that carried the hand across rough ground could still perform other tasks once it arrived. Without any camera, the hand typed on a keyboard well enough to play Sokoban, executing 29 of 32 commands correctly with a quarter-second delay. With an overhead camera added, it picked up a small cube and pushed it toward marked targets, landing within roughly 0.7 inches (17 millimeters) on average.

Kazemipour said the approach could let a larger robot unclip the hand, send it to reach an object the main body cannot access, then have it walk back on its own. No extra wheels or legs would be needed, since the hand relies entirely on the fingers it already has.

Source: https://interestingengineering.com/ai-robotics/eth-zurich-walking-robotic-hand-wuji