No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures (2020)

Abstract: We study relationships between spoken language and co-speech gestures in context of two key challenges. First, distributions of text and gestures are inherently skewed making it important to model the long tail. Second, gesture predictions are made at a subword level, making it important to learn relationships between language and acoustic cues. We introduce AISLe, which combines adversarial learning with importance sampling to strike a balance between precision and coverage. We propose the use of a multimodal multiscale attention block to perform subword alignment without the need of explicit alignment between language and acoustic cues. Finally, to empirically study the importance of language in this task, we extend the dataset proposed in Ahuja et al. (2020) with automatically extracted transcripts for audio signals. We substantiate the effectiveness of our approach through large-scale quantitative and user studies, which show that our proposed methodology significantly outperforms previous state-of-the-art approaches for gesture generation. Link to code, data and videos: https://github.com/chahuja/aisle

## introduction spoken language has gained more traction in the past decade due to improvements in natural language understanding and speech recognition. with an eye on the future, technologies such as intelligent personal assistants (e.g. alexa, siri, cortana) are likely to also include embodiment to take advantage of the non-verbal communication that people naturally use in face-to-face interactions. as a stepping stone in this direction, it is important to study the relationship between spoken language (which also includes acoustic information) and free form gestures (which go beyond just a pre-defined dictionary of gesture animations). in other words, how can we automatically generate human body pose (gestures) from language and acoustic inputs? an important technical challenge in such a natural language processing task, is modeling the long tail of the language-gesture distribution (see figure 1 ). if not addressed directly, computational models will likely focus on the common gestures (e.g beat gestures) as a way to improve precision at the cost of reduced coverage for less frequent words and gestures @xcite . hence, when learning these models, we need to not only be accurate for gesture generation, but also handle coverage of both linguistic and visual distributions @xcite . in other words, we need models that can balance precision and coverage. another technical challenge comes from the differences in granularity between language and gestures. gestures can be triggered at the sub-word level; for example, by a change of intonation in acoustics. thus, it is important to have sub-word level alignment between language and acoustics to generate the freeform gestures. in this paper, we study the link between spoken language and free form gestures. as a first contribution, we propose adversarial importance sampled learning(or aisle), an approach whose main novelty is to bring adversarial learning and importance sampling together to improve coverage of the generated distribution without compromising on the precision at no extra computational cost. as a second contribution, we introduce the use of neural cross-attention architecture @xcite for gesture generation conditioned on spoken language. this idea allows transformer blocks to help with subword alignment between language and acoustic signals. a third contribution is the extension of dataset proposed in @xcite with automatically extracted transcripts for audio signals corresponding to 250+ hours of freeform gesture information and 25 speakers. our experiments study the effectiveness of our proposed method with a focus on precision-coverage trade-off. these quantitative experiments are complimented with important subjective human studies as the englobing judges of the generation quality. moving forward, neural networks were employed to predict a sequence of frames for gestures @xcite , head motions @xcite and body motions @xcite @xcite conditioned on a speech input while @xcite uses only a text input. unlike these approaches, @xcite rely on both speech and language for gesture generation. but their choice of early fusion to com-bine the modalities ignores multi-scale correlations @xcite between speech and language. while publicly datasets of co-speech gestures are available, they are either small @xcite @xcite or do not contain language information @xcite @xcite , which motivates for a dataset that resolves these shortcomings. ## problem statement the goal of this cross-modal translation task is to generate a series of freeform gestures that are aligned with the spoken sentence (see figure 2 ). by free form gestures, we refer to a sequence of joint positions (a.k.a. poses) of the upper human body including neck, torso, arms, hands and fingers. on our way to achieving this goal we work towards solving two challenges: (1) generating gestures from the long-tail of the language-gesture distribution while maintaining high precision of these generated gestures and, (2) sub-word level alignment of language, acoustic cues and gestures to account for the differences in frame rates between among these modalities. formally, we are given a sentence of k language tokens x @xmath0, x w 1 , . . . x w k-1 which has a dynamic frame rate -i.e. each token has a variable time duration dependent on its context-as compared to the fixed frame rate of a sequence of speech features, x @xmath1, x a 1 , . . . x a t -1 . we want to predict a sequence of t gesture poses y @xmath2, y p 1 , . . . y p t -1 that co-occur with x a and x w . here y p @xmath3. this problem can be formalized as learning a true conditional probability distribution p data (y|x) of output @xmath4, given input @xmath5, x w } consisting of text and speech. we write this in form of a generator function g θ with trainable parameters θ as: all our experiments are in an adversarial set-up to alleviate the challenge of overly smooth generation @xcite caused by the reconstruction loss the generated pose sequence ŷp is fed as a signal for the adversarial discriminator d η , which tries to classify the true pose y p from the generated pose ŷp . this is jointly trained with the generator, which learns to fool the discriminator by generating realistic poses. this adversarial loss @xcite is written as: ## model in this section, we present our adversarial importance sampled learning (or aisle) paradigm which is designed to improve coverage while learning accurate relationships between spoken language and gestures. this contribution is described in section 4.1. our second contribution is the application of a transformer architecture to the problem of sub-word alignment between language and acoustic features. this model multimodal multi-scale transformer (mms-transformer) is presented in section 4.2. the remaining components of our full model; pose decoder g dec , language encoder g w enc and acoustic encoder g a enc are presented in section 4.3. the key contributions are illustrated in figure 2 and can be summarized by optimizing the overall loss function l(y, x) with aisle in algorithm 1. ## results and discussions first, we study the effect of different components of our model on coverage and precision. we follow this up with the quantitative effects of dataset subsampling. finally, we conclude with a discussion on the need of a precision-coverage trade-off for cospeech gesture generation. all models are trained separately for each of 25 speakers in pats dataset and we report scores averaged over all speakers for comparison. comparison with previous baselines: we focus first on the human perceptual study in ## conclusions in this paper, we studied the relationship between spoken language and free-form gestures. first, we introduced adversarial importance sampled learning, which combines adversarial learning with importance sampling to strike a balance between precision and coverage at no extra computational cost. second, this work also introduced the use of transformers for gesture generation conditioned on spoken language. third, we extended the pats dataset in @xcite by extracting transcripts for audio signals to study the effect of language in co-speech gesture generation. we substantiated the effectiveness of our approach through largescale quantitative and user studies and show significant improvements over previous state-of-the-art approaches on both precision and coverage.