Researchers develop AI that gives virtual characters human-like gestures as they speak
Characters may be able to pronounce each word correctly but may appear to lack realism...

Characters may be able to pronounce each word correctly but may appear to lack realism in their body movements, such as hand, facial and bodily movements. This issue has been tackled by researchers from Tohoku University and the University of Hong Kong with their creation of a new system named DynaConTalk*.*This converts spoken language into smooth 3D movements.
The system works by creating realistic body movements. It creates fluid movements with parts of the body instead of just hand movements. This allows for smoother movements, such as changing posture, finger and facial movements, to create a much more realistic character.
However, getting these movements to be realistic can be difficult. Big body movements and small facial and hand movements occur at different speeds and if the animation system isn’t in sync you may get repetitive movements, stiff postures or sudden changes in movements. This issue has been addressed by dividing movements into slow and fast elements.
This ensures that movements are in sync as big movements won’t clash with smaller ones. The system takes into account the rhythm, vocal qualities and words being spoken to decide what movements to produce.
Making AI-generated movements more natural
Additionally, movements are kept smooth by using previous movements to guide the next set of movements. This means there aren’t any sudden changes in movements as the character is continuing to speak. You can also specify the initial posture of the body or provide reference movements.
This level of control could be useful for artists and developers who want the character to be a certain style or move in specific ways. This system will manage the link between the spoken words and the movements.
The system was tested with two public data sets (BEAT2 and SHOW) and a human evaluation was organized with 30 participants. Participants were shown videos created by three different systems with 15 examples of speech.
The system scored the highest average across all four categories. The movements created were more similar to the distribution of human movements recorded in the evaluations. This was based on naturalness, relevance to the spoken words, synchronization and variety.
AI could transform animation and virtual learning
The results could be interesting for developers creating video games, films, virtual presenters and online learning. An instructor could have hand and facial movements that match the rhythm and meaning of a spoken explanation. Similar technology could be used to make virtual characters appear more realistic during online presentations.
However, there are still issues with getting the system to be realistic. There are not many examples of large scale body movements within current data sets. The spoken words alone don’t give enough information for the animation system to know exactly where to move the person. Whilst a sentence may indicate someone’s emotions or intentions it doesn’t show if they should be moving forwards, turning around or going somewhere else.
The team wants to look into larger movement data sets and adding extra information about the path of the movements. They also want to expand this to activities such as walking.
This research will also be presented at SIGGRAPH Asia 2026 which will be held in Kuala Lumpur, Malaysia between the 1st and 4th December. The findings were published in ACM Transactions on Graphics.
Get the latest news delivered to your inbox.
Related articles





