Award-winning lip-reading AI aims to help deaf users by solving recognizable words
Researchers at the Korea Advanced Institute of Science and Technology (KAIST) built a lip-reading AI...

Researchers at the Korea Advanced Institute of Science and Technology (KAIST) built a lip-reading AI that decodes clear words first and uses them to resolve ambiguous ones, in a paper that won an Outstanding Paper Award at a 2026 language-processing conference. It reached a 19.4 percent word error rate on the LRS3 benchmark, which the authors call the best among methods using only LRS3 as labeled training data.
Lip reading is hard because English’s 40 sounds map to only 10 to 13 distinguishable mouth shapes, the paper said, so /p/, /b/ and /m/ look nearly identical. AI models write sentences from first word to last, so an early wrong guess can affect later words, KAIST said.
Word error rate counts wrong, missing or added words against a correct transcript, so lower is better. KAIST said the technology could turn noisy speech into text or help deaf and hard-of-hearing people follow conversations. The paper tests only recorded benchmark videos.
Filling in the clear words first
The system uses a diffusion large language model, a text generator that starts with every word position blank and fills them in over repeated passes instead of writing left to right. According to the paper, the model commits early to positions it is confident about, then uses these words as context on both sides to refine uncertain ones. In the paper’s example, confident words such as “bags” and “trip” help correct “back” to “pack.”
The team, led by Professor Yong Man Ro, trained the model in two stages, KAIST said. It first learns to match lip movements in video to text, then learns to model sentence length. When decoding, it estimates several plausible lengths from the video’s duration, generates a candidate sentence for each and picks the best scorer on length likelihood and model confidence.
The team trained on LRS3, 433 hours of labeled TED and TEDx video, the paper said. Using the same visual encoder, error fell from 25.8 percent for a baseline the authors reproduced themselves to 19.4 percent.
Where the results stop
The authors’ claimed record applies only to methods using LRS3 as their only labeled training data. The paper’s table lists systems trained on far more labeled video with lower error, including 12.8 percent for a 2024 model trained on 100,000 hours. On WildVSR, an out-of-domain test set, the same model scored 39.9 percent.
Tests covered English only, and results haven’t been checked across speaker or demographic subgroups. Testing sentence lengths adds decoding time: 0.87 times the video’s duration on one RTX 3090 graphics card, versus 0.14 without it, the paper says. They also flag privacy risks, since speech could be inferred from silent video without consent.
KAIST said the team plans to reduce the model’s size and computing cost, aiming for real-time use on wearables and robots without an internet connection. Ro said the work would lay the foundation for physical AI (machines that sense and act in the real world) that people can reliably use on varied devices in daily life.
The study appeared in the conference’s October proceedings in Budapest.
Get the latest news delivered to your inbox.
Related articles





