T 3 M : Text Guided 3 D Human Motion Synthesis from Speech (2024)

Abstract: Speech-driven 3D motion synthesis seeks to create lifelike animations based on human speech, with potential uses in virtual reality, gaming, and the film production. Existing approaches reply solely on speech audio for motion generation, leading to inaccurate and inflexible synthesis results. To mitigate this problem, we introduce a novel text-guided 3D human motion synthesis method, termed T3M. Unlike traditional approaches, T3M allows precise control over motion synthesis via textual input, enhancing the degree of diversity and user customization. The experiment results demonstrate that T3M can greatly outperform the state-of-the-art methods in both quantitative metrics and qualitative evaluations. We have publicly released our code at https://github.com/Gloria2tt/naacl2024.git

## introduction speech-driven 3d motion synthesis, known as speech-to-motion, is a technique aimed at generating realistic and expressive motion animations from human speech. despite its promising applications in virtual reality (vr) @xcite , gaming @xcite , and film production @xcite , speech-to-motion also encounters significant challenges, involving various modalities and intricate mappings. speech signals tend to be high-dimensional, noisy, and subject to variability, while motion data often exhibit sparsity, discreteness, and adherence to physical laws. additionally, the connection between speech and motion is not deterministic; instead, it relies on factors such as the environment, emotions, and individual personalities. moreover, in traditional speech-to-motion systems, speech audio serves as the sole input for generating various motions for the face, body, and hands. however, this approach may lead to imprecise and undesired motion synthesis due to limitations in the expressive capabilities of the audio signal. identical audio signals could also stem from entirely unrelated contexts. for instance, as depicted in figure 1 , when examining the same audio segment, an introverted speaker tends to use minimal body and hand motions compared to an extroverted speaker, who exhibits a more extensive range of movements. capturing such contextual information solely from audio input proves nearly impossible. this limitation in precise control poses potential difficulties for emerging industries like ai-driven film or animation production, where generated motions may need additional refinement to match user preferences more accurately. to address this issue, we introduce a novel textguided 3d human motion synthesis from speech method, termed t3m. the t3m framework enables accurate control of body-hand motion generation via provided text prompts. this improvement is especially valuable for addressing the rigidity often observed in the motions generated due to the relationship between speech and co-speech gesture is one-to-many in nature. even the same speech in different situation can be result in different motion style. the controllability afforded by t3m facilitates the creation of more nuanced and realistic motion sequences, enhancing overall realism and expressiveness. our t3m contains three major blocks, a vq-vae network to generate an intermediate codebook for action-to-action mapping, an audio feature extraction network to extract acoustic information of audio, and a multimodal fusion block to implement audio and text interaction. specifically, we train a vq-vae network with a two-layer codebook, which contains hand and body information respectively. considering that human motions are related to the speaker's emotion, intonation, and rhythm, we utilize a pre-trained encodec (défossez et al., 2022) model to extract acoustics features from the original audio. to align the sequenc lenth of audio feature and the stored body-hand motion parameter, we use an audio encoder to downsample the features. furthermore, we propose a multimodal fusion encoder structure, which inserts a cross-attention layer to the transformer decoder acoustics architecture for better textual information fusion. another significant challenge arises in the generation of training datasets for t3m. most existing training datasets for speech-to-motion are in the form of speech-motion pairs, lacking corresponding textual information. one simple approach to address this gap is to utilize a video large language model (vllm) like video-llama @xcite for labeling datasets. however, current vllms can only provide a coarse-grained description of the video input. additionally, since speech-motion pairs in the training dataset are often extracted from particular segments of lengthy videos, employing vllm for text generation may lead to highly similar text descriptions being produced across various video clips. to enhance the diversity of textual descriptions within the training dataset, we adopt the video-language contrastive learning framework, videoclip @xcite . this framework enforces the alignment of video and text in a joint embedding, enabling the processing of video frames and utilizing the resultant video features to replace textual features for t3m training. our research primarily centers on text-guided speech to body and hand motions generation. for 3d face reconstruction, we utilize cutting-edge methods in the field, such as those demonstrated in @xcite . overall, our contributions can be described as follows: ## method in this section, we describe the detailed design of t3m, which can generate holistic body motion, including body poses, hand gestures, and facial expressions, based on provided text descriptions. ## experiment we first describe experiment setup in section 4.1, and provide quantitative and qualitative evaluate in section 4.2 and section 4.3. we then evaluate t3m over more data examples in section 4.4, and present the ablation study in section 4.5. we include a video demo in the supplementary materials (data). ## conclusion in this paper, we proposed t3m, a novel textguided 3d human motion synthesis method from speech. t3m can generate realistic and expressive holistic motions by leveraging both speech and textual inputs. we use a pre-trained encodec model to extract audio features from speech and a multimodal fusion model to fuse the audio and text features. to enhance the text diversity during training, we employed videoclip, a video-language contrastive learning framework, to process the video frames and use the output video features to replace the textual features. by training on the show dataset, a 3d holistic dataset, t3m enables users to precisely control the holistic motion generated from speech by utilizing textual inputs.