The ReprGesture entry to the GENEA Challenge
This paper describes the ReprGesture entry to the Generation and Evaluation of Non-verbal Behaviour for Embodied Agents (GENEA) challenge 2022.
The GENEA challenge provides the processed datasets and performs crowdsourced evaluations to compare the performance of different gesture generation systems.
In this paper, we explore an automatic gesture generation system based on multimodal representation learning.
We use WavLM features for audio, FastText features for text and position and rotation matrix features for gesture.
Each modality is projected to two distinct subspaces: modality-invariant and modality-specific.
To learn intermodality-invariant commonalities and capture the characters of modality-specific representations, gradient reversal layer based adversarial classifier and modality reconstruction decoders are used during training.
The gesture decoder generates proper gestures using all representations and features related to the rhythm in the audio.
Our code, pre-trained models and demo are available at https://github.com/YoungSeng/ReprGesture.
ACM Reference Format:
1 INTRODUCTION
Nonverbal behavior plays a key role in conveying messages in human communication [13], including facial expressions, hand gestures and body gestures.
Co-speech gestures introduce better self-expression.
In the virtual world, it helps to present a rather realistic digital avatar.
Gesture generation studies how to generate human-like, natural, speech-oriented gestures.
There are many different techniques for gesture generation.
In this paper, we focus on the task of speech-driven gesture generation.
Representative speech-driven gesture generation are either rule-based or data-driven [19].
Many data-driven works for gesture generation are based on multimodal fusion and representation learning.
Taras et al.
map speech acoustic and semantic features into continuous 3D gestures [12].
Youngwoo et al.
propose an endto-end model to generate co-speech gestures using text, audio, and speaker identity [19].
Jing et al.
sample gesture
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page.
Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
{+© 
 Copyright held by the owner/author(s). Manuscript submitted to ACM+} in a variational autoencoder (VAE) latent space and infer rhythmic motion from speech prosody to address the nondeterministic mapping from speech to gesture [18].
Taras et al.
propose a speech-driven gesture-production method based on representation learning [11].
Xian et al.
propose the hierarchical audio features extractor and pose inferrer to learn discriminative representations [17].
Jing et al.
present a co-speech gesture generation model whose latent space is split into shared code and motion-specific code [15].
However, gesture generation is a challenging task because of cross-modality learning issue and the weak correlation between speech and gestures.
The inherent heterogeneity of the representations creates a gap among different modalities.
It is necessary to address the weak correlation among different modalities and provide a holistic view of the multimodal data during gesture generation.
Inspired by [19] and [6], we propose a gesture generation system based on multimodal representation learning.
In particular, we first extract features of audio, text and gestures.
Then, a system consisting of four components is proposed: (1) Each modality is projected to two distinct representations: modality-invariant and modality-specific.
A gradient reversal layer based adversarial classifier is used to reduce the discrepancy between the modality-invariant representations of each modality.
(3) Modality decoders are used to reconstruct each modality, allowing modalityspecific representations to capture the details of their respective modality.
The gesture decoder takes six modality representations (two per modality) and rhythm-related features in audio as its input and generates proper gestures.
The main contributions of our work are: (1) A multimodal representation learning approach is proposed for gesture generation, which ensures comprehensive decoupling of multimodal data.
(2) To solve the problem of heterogeneity of different modalities in feature fusion, each modality is projected to two subspaces (modality-invariant and modalityspecific) to get multimodal representations using domain learning and modality reconstruction.
(3) Ablation studies demonstrate the role of different components in the system.
The task of the GENEA 2022 challenge is to generate corresponding gestures from the given audio and text.
A complete task description can be accessed in [21].
We submitted our system to the GENEA 2022 challenge to be evaluated with other gesture generation systems in a large user study.
2 METHOD
I am human.
FastText
WavLM
Encode Motion
U t
U a
U g
E c ( u m ; θ c )
E p ( u t ; ) p t 
E p ( u a ; ) pa 
E p ( u g ; ) pg 
p t
h pa h pg h c g h c a h ct h
Gradient Reversal Layer do m a i n c p m m  h h
Modality Decoder r ec on ct h
pg h
Gesture Decoder task
Modality Representation Feature Extraction
Domain Learning
Modality Reconstruction
Gesture Generation ˆ m u u t u a u g
[Figure]
2.1 The architecture of the proposed system
As shown in Figure 1, the system generates a sequence of human gestures from a sequence of u 𝑚 ( 𝑚 ∈ { 𝑡,𝑎,𝑔 }) that contain the features of text, audio and seed gestures.
The architecture of the proposed model consists of five modules: feature extraction, modality representation, modality reconstruction, domain learning and gesture generation.
The following describes each of these modules in detail.
2.1.1 Feature extraction.
For each of the modality, the pipeline of extracting features is as follows:
• Text: We first use FastText [1] to get the word embeddings.
Padding tokens are inserted to make the words temporally match the gestures by following [19].
One-dimensional (1D) convolutional layers are then adopted to generate 32-D text feature sequence U 𝑡 (‘ 𝑡 ’ for ‘text’) from the 300-D word embeddings.
• Audio: All audio recordings are downsampled to 16kHz, and features are generated from the pre-trained models of WavLM Large [3].
We further adjust sizes, strides and padding in the 1D convolutional layers to reduce the dimension of features from 1024 to 128 forming the final audio feature sequence U 𝑎 (‘ 𝑎 ’ for ‘audio’).
Gesture:
Due to the poor quality of hand motion-capture, we only use 18 joints corresponding to the upper body without hands or fingers.
Root normalization is used to make objects face the same direction.
We apply standard normalization (zero mean and unit variant) to all joints.
Seed gestures for the first few frames are utilized for better continuity between consecutive syntheses, as in [19].
On top of these, position and 3 × 3 rotation matrix features are computed, and the size of final gesture sequence U 𝑔
(‘ 𝑔 ’ for ‘gesture’) feature is 216.
2.1.2 Modality representation.
First, for each modality 𝑚 ∈ { 𝑡,𝑎,𝑔 } , we use a linear layer with leaky ReLU activation and layer normalization to map its feature sequence U 𝑚 into a new feature sequence u 𝑚 ∈ R 𝑇 × 𝑑 ℎ with the same feature dimension 𝑑 ℎ .
Then, we project each sequence u 𝑚 to two distinct representations: modality-invariant h 𝑐𝑚 and modality-specific h 𝑝𝑚 .
Afterwards, h 𝑐𝑚 learns a shared representation in a common subspace with distributional similarity constraints [5].
h 𝑝𝑚 captures the unique characteristics of that modality.
We derive the representations using the simple feed-forward neural encoding functions:
[Equation]
Encoder 𝐸 𝑐 shares the parameters 𝜃 𝑐 across all three modalities, whereas 𝐸 𝑝 assigns separate parameters 𝜃 𝑝𝑚 for each modality.
2.1.3 Representation learning.
Domain learning can improve a model’s ability to extract domain-invariant features [2].
We use an adversarial classifier to minimize domain loss that reduces the discrepancy among shared representations of each modality.
The domain loss can be formulated as:
[Equation]
where 𝐷 𝑟𝑒𝑝𝑟 represents feed-forward neural discriminator, 𝑑 𝑚 represents the result after gradient reversal of h 𝑝𝑚 .
The modality reconstruction loss L recon is computed on the reconstructed modality and the original input u 𝑚 .
The L recon is used to ensure the hidden representations to capture the details of their respective modality.
Specifically, a modality decoder 𝐷 is proposed to reconstruct u 𝑚 :
[Equation]
where 𝜃 𝑑 are the parameters of the modality decoder.
The modality reconstruction loss can then be computed as:
[Equation]
where ∥ · ∥ 2 2 is the squared 𝐿 2 -norm.
Gesture generation.
[Equation]
[Figure] [Equation]
[Figure]
[Equation]
[Figure]
We use generative adversarial network (GAN) based gesture decoder for generating gestures.
Gestures are directly related to rhythm and beat, thus we concatenate audio rhythm related features (pitch, energy and volume) and the output of six stacked modality representations together and send them to Transformer encoders with multi-head self-attention as the generator, as shown in Figure 2.
The generator part is trained using L 𝑔𝑒𝑠𝑡𝑢𝑟𝑒 consisting of the Huber loss and the MSE loss, and the discriminator part is trained with L 𝐺𝐴𝑁 .
[Equation]
[Equation] (6) where 𝐷 𝑔𝑒𝑠𝑡𝑢𝑟𝑒 represents gesture discriminator using multilayered bidirectional gated recurrent unit (GRU) [4] that outputs binary output for each time step, 𝑡 is the length of the gesture sequence, 𝑔 𝑖 represents the 𝑖 th human gesture, ˆ 𝑔 𝑖 represents the 𝑖 th generated gesture.
The loss of the proposed system can be computed as:
[Equation]
2.2 Data processing and experiment setup
2.2.1 Data and data processing.
In the challenge, the Talking With Hands 16.2M [14] is used as the standard dataset.
Each video is separated into two independent sides with one speaker each.
The audio and text in the dataset have been aligned.
For more details please refer to the challenge paper [21].
We note that the data in the training, validation and test sets are extremely unbalanced, so we only use the data from the speaker with identity "1" for training.
And we believe that if speech and gesture data are trained on the same person, the gesture behavior would match the speech.
2.2.2 Experiment setup.
The proposed system is trained on training data only, using the ADAM [9] optimizer (learning rate is e-4, 𝛽 1 = 0.5, 𝛽= 0.98) with a batch size of 128 for 100 steps.
We set 𝛼 = 300, 𝛽 = 50 for Equation (5) and 𝛾 = 5 ,𝛿 = 0 .
1 ,𝜖 = 0 .
1 (we ReprGesture entry GENEA Challenge noticed in our experiments that too large 𝛿 and 𝜖 will lead to non-convergence) for Equation (7).
There is a warm-up period of 10 epochs in which the L 𝐺𝐴𝑁 is not used ( 𝛾 = 0).
The feature dimension 𝑑 ℎ of sequence u 𝑚 is 48.
During training, each training sample having 100 frames is sampled with a stride of 10 from the valid motion sections; the initial 10 frames are used as seed gesture poses and the model is trained to generate the remaining 90 poses (3 seconds).
3 EVALUATION 3.1 Evaluation setup
The GENEA Challenge 2022 evaluation is divided into two tiers, and we participated in the upper-body motion tier.
The challenge organizers conducted a detailed evaluation comparing all submitted systems[21].
The challenge evaluates human-likeness to assess motion quality, and appropriateness to assess how well the gestures match the speech.
The evaluation is based on the HEMVIP methodology [8] and Mean Opinion Score (MOS)
There are in total 11 systems participated in the upper-body tier.
The following abbreviations are used to represent each model in the evaluation:
• UNA: Ground truth (‘U’ for the upper-body tier, ‘NA’ for ‘natural’).
• UBT: The official text-based baseline [20], which takes transcribed speech text with word-level timing information as the input modality (‘B’ for ‘baseline’, ‘T’ for ‘text’).
• UBA:
The official audio-based baseline [10], which takes speech audio into account when generating output (‘A’ for ‘audio’).
• USJ–USQ: 8 participants’ submissions to the upper-body tier (ours is USN).
For more details about the evaluation studies, please refer to the challenge paper [21].
3.2 Subjective evaluation results and discussion 3.2.1 Human-likeness Evaluation.
[Table]
t i n g
[Table]
[Figure]
[Figure]
S
i g n i ﬁ c a n t p r e f e r e n ce f
o r c
o
nd
i t
i o n y ...
USQ UNA USJ USO USN USK USM UBT UBA USP USL
USQ
UNA
USJ
USO
USN
USK
USM
UBT
UBA
USP
USL
In this evaluations, study participants are asked to rate “How human-like does the gesture motion appear?”
on a scale from 0 (worst) to 100 (best).
Bar plots and significance comparisons are shown in Figure 3.
Our system (USN) receives a median score of 44 and a mean score of 44.2, and is ranked fourth among the participating systems.
[Figure]
r
[Figure]
h
[Figure]
o t i o n
[Figure]
In this evaluation, participants are asked to choose the character on the left, on the right, or indicate that the two are equally well matched to response “Please indicate which character’s motion best matches the speech, both in terms of rhythm and intonation and in terms of meaning.”
Bar plots are shown in Figure 4.
Our system (USN) receives a “Percent matched” 54.6, which identifies how often participants preferred matched over mismatched motion in terms of appropriateness.
Our system is rated seventh in appropriateness among the participants’ submissions.
It should be noted that the difference of our system to the five higher-ranked systems (USL, UBA, USO, USK and USJ) is not significant.
Furthermore, if we only consider the ratio of matched motion, i.e., the blue bar in Figure 4, our system is ranked fifth among the participating systems.
3.3 Ablation studies
Moreover, we conduct ablation studies to address the performance effects from different components in the system.
The GENEA challenge computes some objective metrics of motion quality by GENEA numerical evaluations 1 .
For calculation and meaning of these objective evaluation metrics, please refer to the challenge paper [21].
A perfect natural system should have average jerk and acceleration very similar to natural motion.
The closer the Canonical correlation analysis (CCA) to 1, the better.
Lower Hellinger distance and Fréchet gesture distance (FGD) are better.
To compute the
FGD, we train an autoencoder using the training set of the challenge.
The results of our ablations studes are summarized in Table 1.
Supported by the results, when we do not use WavLM to extract audio features, but use 1D convolution instead, the Hellinger distance average and FGD on feature space present ICMI November 7– [Table]
India
[Table]
the worst performance. When the model is trained without the GAN loss, the average jerk and average acceleration are better [Table] but the global CCA and CCA for each sequence are decreased.
When the reconstruction loss is removed, the average jerk and average acceleration are worst.
The generated gesture movements are few and of small range.
When the model is trained using Central Moment Discrepancy (CMD) loss [6] instead of domain loss, the best FGD on feature space and the worst FGD on raw data space are obtained.
When the modality representations are removed (w/o
Repr), we feed the modality sequence u 𝑡 , u 𝑎 and u 𝑔 directly to the gesture decoder and only use the L 𝑡𝑎𝑠𝑘 loss, the performances of all metrics have deteriorated except for FGD on feature space.
4 CONCLUSIONS AND DISCUSSION
In this paper, we propose a gesture generation system based on multimodal representation learning, where the considered modalities include text, audio and gesture.
Each modality is projected into two different subspaces: modality-invariant and modality-specific.
To learn the commonality among different modalities, an adversarial classifier based on gradient reversal layer is used.
To capture the features of modality-specific representations, we adopt a modality reconstruction decoder.
The gesture decoder utilizes all representations and audio rhythmic features to generate appropriate gestures.
In subjective evaluation, our system is ranked fourth among the participating systems in human-likeness evaluation, and ranked seventh in appropriateness evaluation.
Whereas, for appropriateness, the differences between our system and the five higher-ranked systems are not significant.
For appropriateness evaluation, whether there is a relationship between subjective evaluation and segmentation duration deserves to be investigated.
The segments are around 8 to 10 seconds during evaluation[21].
We believe that a longer period of time (e.g. 20-30 seconds) might produce more pronounced and convincing appropriateness results.
There is room for improvement in this research.
First, we only use data from one person to learn gesture due to unbalanced dataset issue.
Such one-to-one mapping could produce boring and homogeneous gestures during inference.
Second, the finger motions are not considered because of the low motion-capture quality.
Such finger motions could be involved in the future if some data cleanup procedures could be conducted.
Third, besides text and audio, more modalities (e.g. emotions, facial expressions and semantic meaning of gestures [16]) could be taken into consideration to generate more appropriate gestures. ACKNOWLEDGMENTS
This work is supported by Shenzhen Science and Technology Innovation Committee (WDZC20200818121348001),
National Natural Science Foundation of China (62076144) and Shenzhen Key Laboratory of next generation interactive media innovative technology (ZDSYS20210623092001004).