LLM Knows Body Language, Too: Translating Speech Voices into Human Gestures (2024)

Abstract: In response to the escalating demand for digital human representations, progress has been made in the generation of realistic human gestures from given speeches. Despite the remarkable achievements of recent research, the generation process frequently includes unintended, meaningless, or non-realistic gestures. To address this challenge, we propose a gesture translation paradigm, GesTran, which leverages large language models (LLMs) to deepen the understanding of the connection between speech and gesture and sequentially generates human gestures by interpreting gestures as a unique form of body language. The primary stage of the proposed framework employs a transformer-based auto-encoder network to encode human gestures into discrete symbols. Following this, the subsequent stage utilizes a pre-trained LLM to decipher the relationship between speech and gesture, translating the speech into gesture by interpreting the gesture as unique language tokens within the LLM. Our method has demonstrated state-of-the-art performance improvement through extensive and impartial experiments conducted on public TED and TED-Expressive datasets.

## introduction the synthesis of human motions and gestures that correspond with concurrent speech, a process known as co-speech gesture generation, is instrumental in conveying messages during human communication and augmenting selfexpression @xcite . therefore, generating realistic and controllable gesture motions that are both plausible and synchronous with the corresponding speech input can significantly bolster the acceptance of social robots by human users @xcite . this holds substantial promise for various applications, including but not limited to, education, * corresponding author training, and medical contexts. additionally, the potential for this pursuit extends to the development of digital humans in emerging virtual environments, non-player game characters, robotic assistants @xcite , and embodied artificial intelligence. in practical scenarios, a single speech input could correspond to a variety of gestures or motions. for example, a speech input corresponding to a beat motion could be performed using the left hand, right hand, or both hands; all these variations are plausible and would be considered appropriate by human users @xcite . however, prior methodologies @xcite often frame co-speech gesture generation as a regression problem, resulting in a model that is more likely to learn an average of all plausible gestures rather than distinct ones, thereby generating excessively smoothed and unrealistic average gestures. consequently, these methods tend to yield more restrained gesture motions, which are less engaging from a human perception standpoint. it remains unclear how existing methods could capture such one-to-many variability. furthermore, such methods tend to exhibit instability in practical applications and are susceptible to regression towards nonstandard poses beyond the gesture subspace, such as freezing or meaningless swaying @xcite . a primary inherent issue contributing to the aforementioned problems is that previous methods fail to adequately model the semantic relation between speech and gesture, specifically, their diverse temporal correspondence. to enable the generative model to more comprehensively understand and encapsulate this relationship, we propose treating the gesture as a distinct form of body language that can be seamlessly translated into and out of speech. recent research suggests that large language models (llms) @xcite @xcite can process multimodal inputs, such as images and videos, through a lightweight adapter. consequently, we anticipate that llms, with a suitable adapter, can also comprehend gesture sequences. the integration of gesture and speech (audio and its corresponding text) data, encoded within a unified vocabulary, makes the relationship between motion and language more discernible. this would enable the gesture generator, which is fine-tuned from llms, to produce gestures with diverse patterns and flexible sequences. in this paper, we propose a new llm-driven co-speech gesture translation method, namely ges-tran, which emulates the procedure of bilingual translation in humans and has the capability to comprehend and translate human gestures that are concomitantly associated with speech and its corresponding text. in order to equip gestran with the capability to comprehend and generate gestures akin to humans, an initial step involves training a gesture-specific vector quantized variational autoencoder (vq-vae) @xcite model. the objective of this step is to compile a "gesture lexicon" @xcite , analogous to a natural language vocabulary, which subsequently allows for the conversion of unprocessed gesture data into a series of gesture tokens. these tokens are subsequently processed by a pre-trained language model, which has been trained to understand the inherent grammar and syntax of the gesture language, as well as its correlation with the corresponding audio and text of human speeches. to efficiently amalgamate speech and gesture, we finetune the pre-trained language model on a multimodal co-speech gesture dataset, which is instrumental in learning the correlation and conversion between speech and gesture. in this way, we can easily translate the speech to desirable body language, i.e., human gesture, in the unified llm. extensive experimental results demonstrate that ges-tran attains a performance that surpasses current benchmarks in the task of co-speech generation. we summarize our contributions as follows: regarding human gesture as a specific body language, our method can better comprehend the correlation between speech and gesture and effectively translate them in the unified pre-trained llm. previous studies aim to augment model capacity through a range of architectural choices, including convolutional neural networks (cnn) @xcite , recurrent neural networks (rnn) @xcite , transformer models @xcite , generative adversarial networks (gans) @xcite , and diffusion models @xcite @xcite . large language model. large-scale language models (llms) @xcite @xcite , facilitated by extensive datasets and large model sizes, have showcased remarkable comprehension and generation capabilities, significantly advancing the field of natural language processing. bert @xcite , for instance, pre-trains deep bidirectional language representations capable of supporting downstream tasks. t5 @xcite introduced a unified framework that transforms all textbased language tasks into a text-to-text format. recent studies have demonstrated that fine-tuning pretrained models using input-output pairs comprising instructions and corresponding responses can further enhance their performance. @xcite introduces an instruction-tuning technique that outperforms non-tuned models on unseen tasks. llama @xcite is a collection of open-source and efficient foundation large language models ranging from 7b to 65b parameters. moreover, the emergence of multimodal models, which process text along with other modalities such as images, audio, and videos, has garnered considerable attention. despite the success of language models in various vision-language tasks, the development of multi-modal language models capable of interpreting human gestures remains relatively limited. ## method inspired by motiongpt @xcite @xcite , we introduce llms into the task of co-speech gesture generation. capitalizing on the exceptional ability of llms to comprehend and translate multilingual data, we propose a co-speech gesture translator (gestran) governed by multimodal conditions, namely speeches (audio and corresponding text) and human gestures captured in video frames. we intend to frame human gestures as a particular form of body language, thereby enabling the large language model to translate desired human gestures in accordance with corresponding prompts and control conditions. the overall framework of our ges-tran is shown in figure 1 . specifically, we first quantize raw gesture data into discrete tokens using vq-vae @xcite . ## conclusion in this paper, we introduce an innovative gesture translation technique, termed gestran, that capitalizes on the capabilities of large language models (llms) to enhance the comprehension of the intricate relationship between verbal and non-verbal communication. this is accomplished by sequentially generating human gestures, thereby interpreting them as a distinct mode of body language. the initial phase of the proposed architecture incorporates a transformer-based auto-encoder network to transcribe human gestures into discrete symbolic representations. subsequently, the succeeding phase exploits a pre-trained llm, aiming to decipher the interplay between speech and gesture. this is achieved by transforming the verbal input into corresponding gestures, thereby interpreting the gestures as unique language tokens within the llm's context. through a series of rigorous experiments, conducted on two universally acknowledged datasets, consistent evidence of the superior performance of our proposed approach was observed across almost all evaluative metrics. con-sequently, this strongly corroborates the efficacy of the method introduced in this study.