Audio-Driven Co-Speech Gesture Image Generation
Anonymous Author(s)
AffiliationAddress email
Abstract
Co-speech gesture is crucial for human-machine interaction and digital entertainment.
While previous works mostly map speech audio to human skeletons( e.g., 2D keypoints), directly generating speakers’ gestures in the image domainremains unsolved.
In this work, we formally define and study this challengingproblem of audio-driven co-speech gesture image generation , i.e., using a unifiedframework to generate speaker image sequence driven by speech audio.
Our keyinsight is that the co-speech gestures can be decomposed into common motionpatterns and subtle rhythmic dynamics.
To this end, we propose a novel framework,
A udio-drive N G esture I mage g E neration ( ANGIE ), to effectively capture thereusable co-speech gesture patterns as well as fine-grained rhythmic movements.
To achieve high-fidelity image sequence generation, we leverage an unsupervisedmotion representation instead of a structural human body prior ( e.g., 2D skeletons).
Specifically, 1) we propose a vector quantized motion extractor ( VQ-Motion
Extractor ) to summarize common co-speech gesture patterns from implicit motionrepresentation to codebooks.
Moreover, a co-speech gesture GPT with motionrefinement ( Co-Speech GPT ) is devised to complement the subtle prosodic motiondetails.
Extensive experiments demonstrate that our framework renders realisticand vivid co-speech gesture images.
All the code, data and models will be released.
Introduction
During daily conversation among humans, speakers naturally emit co-speech gestures to complementthe verbal channels and express their thoughts [17, 34, 53].
Such non-verbal behaviors ease speechcomprehension [10, 55] and bridge the communicator’s gap for better credibility [7, 51].
Therefore,equipping the social robot with conversation skills constitutes a crucial step to human-machineinteraction.
To achieve it, researchers delve into the task of co-speech gesture generation [21, 37, 59],where audio-coherent human gesture sequences are synthesized in the form of structural humanrepresentation ( e.g., skeletons).
However, such representation contains no appearance information ofthe target speaker, which is crucial for human perception.
As demonstrated in audio-driven talkinghead synthesis [62, 63], generating real-world subjects in the image domain is highly desirable.
Tothis end, we explore the problem of audio-driven co-speech gesture image generation , i.e., using aunified framework to generate speaker image sequence driven by speech audio (illustrated in Fig. 1).
Conventional methods require exhaustive human efforts to pre-define the speech-gesture pairs andconnection rules for coherent result [11, 12].
With the development of deep learning, neural networksare leveraged to learn the mapping from encoded audio feature to human skeletons in a data-drivenmanner [21, 37, 59].
Notably, one category of approaches relies on small-scale MoCap datasetsin co-speech setting [16, 18, 45], which contributes to specific models with limited capacity androbustness.
To capture more general speech-gesture correlations, another category of methods buildslarge training corpus by exploiting off-the-shelf pose estimators [9, 15] to label enormous online videos as pseudo ground truth [21, 60].
However, the inaccurate pose annotations induce error
[Figure]
accumulation in the training phase, which makes the generated results unnatural.
Besides, mostprevious works ignore the problem of co-speech gesture image generation.
Only few works [21, 37]animate in the image domain as an independent post-processing step , which borrows from the existingpose-to-image generators [5, 13] to train on the target person’s images.
How to design a unifiedframework to generate speaker image sequence driven by speech audio remains unsolved.
To effectively learn the mapping from audio to co-speech gesture images, we pinpoint two importantobservations from current studies: 1) hand-crafted structural human priors like 2D/3D skeletons wouldeliminate articulated human body region information.
Such a zeroth-order motion representation failsto formulate first-order motion like local affine transformation in image animation [41].
Besides, theerror in structural prior labeling impairs cross-modal audio-to-gesture learning [32, 33]. 2) Motivatedby previous linguistic studies [27, 44], the co-speech gestures could be decomposed into commonmotion patterns and rhythmic dynamics, where the former ones refer to large-scale motion templates( e.g., periodically put hands up and down), while the latter ones play a refinement role to complementsubtle prosodic movements and synchronize with speech audio ( e.g., finger flickers).
We take inspiration from the above observations and propose a novel framework A udio-drive N
G esture I mage g E neration ( ANGIE ) to generate co-speech gesture images.
The key insight is tosummarize common co-speech gesture patterns from motion representation to quantized codebooksand further refine subtle rhythmic details by motion residuals for fine-grained results .
In particular,two modules are designed, namely VQ-Motion Extractor and Co-Speech GPT .
In VQ-Motion
Extractor, we utilize an unsupervised motion representation to depict the articulated human bodyand first-order gestures [42].
The codebooks are established to quantize the reusable commonco-speech gesture patterns from unsupervised motion representation.
To guarantee the validity ofgesture patterns, we propose a cholesky decomposition based quantization scheme to relax the motioncomponent constraint.
The position-irrelevant motion pattern is extracted as final quantization targetto represent the relative motion.
In this way, the quantized codebooks naturally contain rich commongesture pattern information.
With the quantized motion code sequence, in Co-Speech GPT we usea GPT-like [38] structure to predict discrete motion patterns from speech audio.
Finally, a motionrefinement network is used to complement subtle rhythmic details for fine-grained results.
To summarize, our main contributions are three-fold: 1)
We explore a challenging problem of audiodriven co-speech gesture image generation.
To the best of our knowledge, we are the first to generateco-speech gesture in image domain with a unified framework without any structural human bodyprior .
We propose the VQ-Motion Extractor to quantize the motion representation into commongesture patterns and the Co-Speech GPT to refine subtle rhythmic movement details.
The codebooksnaturally contain reusable motion pattern information.
3) Extensive experiments demonstrate that theproposed framework ANGIE renders realistic and vivid co-speech gesture image generation results.
Related Work
Co-Speech Gesture Generation.
Synthesizing co-speech gesture has gained research interest invision [3, 21, 29], graphics [4, 57, 59] and robotics [23, 25, 60] domains.
Recent researches resort todeep neural networks to learn the speech-gesture mapping in a data-driven manner, with major focuseson below perspectives: 1) Dataset.
One strand of methods use small-scale MoCap datasets to learnspecific models [16, 18, 30, 32, 39, 45, 48, 52], while another strand of works exploit off-the-shelfestimator to label enormous videos as structural prior [1–3, 21, 33, 37, 58–60].
The dataset scalev.s.
pose annotation accuracy often acts as a trade-off in this task:
A large amount of speech-gesturepairs facilitate the training of more general models with better capacity and robustness, yet erroraccumulation in annotations induces unnatural results.
2) Framework architecture.
CNN-based [21], RNN-based [59] and Transformer-based {+[
 ]+} frameworks show promising results.
To further improve
[Figure]
shift-translation µ and covariance C to relative motion pattern representation of ∆ µ and ∆ L , which are further quantized by codebooks to extract the common motion patterns.
Given the driving audio and starting gesture codes, the Co-Speech GPT predicts the future motion fields.
A Motion
Refinement network further learns motion residuals to complement the subtle rhythmic dynamics.
the diversity of generated gestures and grasp the fine-grained cross-modal associations, componentslike adversarial loss [21], VAE sampling [30, 56] and hierarchical encoder-decoder design [33] areproposed.
3) Input modality.
Some approaches treat single modality of speech audio [2, 19, 21–23 , 30, 37] or text transcription [3, 6, 25, 60] as input to drive the co-speech gesture, while someothers use both modalities as stimuli for generation [1, 33, 59].
To ease the learning of implicitcross-modal mapping from speech to gesture and create more stable results, recent works involveauxiliary input modality such as speaker style [2], pose mode [56] and motion template [37].
In this work, we propose a novel vector quantization (VQ) network with cholesky decompositionscheme to extract the valid motion patterns. Besides, we improve the quantization scheme to encodethe relative motion representation that is position (absolute location) irrelevant.
A motion refinementnetwork is further devised to complement subtle rhythmic dynamics.
Notably, our approach gives anidea on how to deal with the constraints in vector quantization and how to complement sequentialresults with missing details. Such design could prospectively provide insights for relevant domainslike constrained vector quantization problem and video generation tasks.
Video/Audio-Driven Image Generation.
Traditional video-driven approaches for image generationcan be categorized into supervised and unsupervised, where the supervised methods typically involvestructural human body prior such as landmarks [8, 61] and 3D parametric models [20, 47], while theunsupervised approaches design self-supervised tasks to animate unlabeled images [40–42, 54].
Tofacilitate broader applications, researchers explore audio-driven image generation, where one of themost relevant tasks is talking face generation [14, 36].
Different from the strong correlations betweenaudio and mouth shape, the mapping from audio to complicated co-speech motion is multi-modal andharder to learn.
Most co-speech gesture studies synthesize human skeletons as final results ( e.g., 2Dkeypoints), while only few works [21, 37] generate co-speech images in a post-processing manner.
Our Approach
We present ANGIE that generates audio-driven co-speech gesture in image domain, where thespeakers’ image sequence is driven by speech audio as shown in Fig.
The whole pipeline isillustrated in Fig.
To make the content self-contained and narration clearer, we first introduce thepreliminaries and problem setting in Sec. 3.1.
Then, we present the VQ-Motion Extractor whichextracts common co-speech gesture patterns as quantized codebooks in Sec. 3.2.
Finally, we elaboratethe Co-Speech GPT to complement subtle rhythmic dynamics for fine-grained results in Sec. 3.3.
3.1 Preliminaries and Problem Setting
Unsupervised Motion Representation.
To achieve high-fidelity image animation, we take inspirationfrom MRAA [42] that uses an unsupervised motion representation to drive articulated objects.
MRAAfirst estimates a coarse motion representation from the source and driving frames, then predicts densepixel-wise flow for image generation.
Specifically, an encoder-decoder keypoint predictor produces
K different heatmaps H 1 , H 2 , . . .
, H K , where K is region number and H k denotes the k -th imageregion.
Afterwards, each heatmap is normalized by softmax operation, i.e , (cid:80) z ∈Z H k
( z ) = 1 , wherez is the image pixel location and Z is the set of all pixels.
The key insight behind MRAA is torepresent each region’s motion by affine transformation with a shift-translation component.
Theshift-translation component µ k ∈ R 2 and the distribution C k of the k -th part can be calculated as:
[Equation]
where T is matrix transpose, C k ∈ R 2 × 2 measures the covariance of heatmap value.
It naturallydepicts the size and shape of an articulated region.
To represent the affine transformation A k ∈ R 2 × 2of the k -th region, we apply singular value decomposition (SVD) to C k and derive A k as:
[Equation]
where unitary matrices U k , V k and diagonal matrix Σ k are the SVD result of covariance matrix C k .
The representation M extracted by motion estimator is the concatenation of [ µ ; C ; A ] ∈ R K × (2+4+4)for K distinct regions, an image generation module with dense pixel-wise flow predictor synthesizesthe final generation results.
In this work, the motion representation M and image generation module
G generally follow MRAA.
We suggest the readers referring to [42] for more details.
Problem Setting for Co-Speech Gesture Image Generation.
We collect training data of massivelyavailable speaking videos with clear co-speech gestures for natural self-reconstruction supervision.
Specifically, given an ( N + 1) -frame video clip V =
{ I (0) , . .
, I ( N ) } , the goal of our frameworkat the training stage is to predict the motion representation (cid:99) M (1: N ) based on the first image frame
I (0) and video’s accompanying audio sequence a = { a (1) , . . .
, a ( N ) } .
Further, the image generationmodule G reconstructs the video frames (cid:98) I (1: N ) .
At the inference stage, an arbitrary reference imagewith speech audio clip is provided to generate subsequent image frames.
According to the observationin Sec. 1, we decompose the co-speech gestures into common motion patterns and subtle rhythmicdynamics.
The overall training setting can be formulated as:
[Equation]
where M pattern denotes the gesture pattern (Sec. 3.2) and M rhythmic is rhythmic movement (Sec. 3.3).
3.2 Vector Quantized Motion Pattern Extractor
To decompose the co-speech gestures, we propose to firstly extract the common motion patterns.
However, three problems remain: 1)
The gesture sequences are different from each other.
While somemotion sequences share the same action pattern, the dynamic details may vary a lot.
How to extractthe major motion pattern despite the influence from minor prosodic movements?
The covariancematrix C is symmetric positive definite (Eq. 1), which further constrains the range of affine matrix
A .
How to preserve such characteristic for valid gesture patterns?
3) Since the unsupervised motionrepresentation is extracted in the image pixel space, it is affected by the absolute location of eacharticulated region.
How to represent the position-irrelevant motion pattern information?
Vector Quantized Motion Pattern Learning.
Our solution to the first problem is to quantize thecommon motion pattern into a codebook.
Since the gesture pattern is finite, it could be summarizedto discrete codebook entries.
Besides, each codebook entry refers to a certain type of gesture pattern,which matches our goal to extract the common and reusable co-speech gesture patterns.
A naive way is to quantize the motion representation M separately as shift-translation µ , covariancematrix C and affine transformation A .
Specifically, for a T -frame co-speech gesture sequence
I (1: T ) , we transform it into [ µ ; C ; A ] (1: T ) ∈ R T × K ×
(2+4+4) , where [ µ ; C ; A ] =
M denotes motionrepresentation of K regions as in Sec. 3.1.
We first build three codebooks D µ = { d µ,m } Mm =1 ,
D C = { d C,m } Mm =1 and D A = { d A,m } Mm =1 for each motion component respectively, where M iscodebook size.
d µ,m , d C,m and d A,m ∈ R ℓ are the m -th entry of ℓ -channel for each codebook.
Then,three separate encoders E µ , E C and E A are utilized to encode the corresponding context informationinto latent features of e µ = { e µ,i } T ′ i
=1 , e C = { e C,i } T ′ i
=1 and e A
= { e A,i } T ′ i
=1 ∈ R T ′ × ℓ , where T ′ isthe temporal dimension and ℓ is the channel dimension.
Notably, we denote the i -th temporal featureof each motion component as e µ,i , e C,i and e A,i .
The feature encoding process can be formulated as:
[Equation]
Following the pipeline of VQ-VAE [50], we individually quantize e µ , e C and e A by substitutingeach temporal feature e µ,i , e C,i and e A,i to the nearest codebook entry d µ,m , d C,m and d A,m as:
[Equation]
where e q µ = { e q µ,i } T ′ i
=1 , e q C = { e q C,i } T ′ i
=
and e q A = { e q A,i } T ′ i
=1 ∈ R T ′ × ℓ are the quantized codesequence of length T ′ for each motion component.
The i -th quantized code of each motion componentis denoted as e q µ,i , e q C,i and e q A,i , respectively.
Finally, three separate decoders D µ , D C and D A areleveraged to reconstruct the motion representations of each component as:
[Equation]
Such discrete representation also ease the audio-to-gesture learning (Sec. 3.3): Previous methodspredict continuous output as a harder regression problem.
While we only need to predict featuresnearer to the correct codebook entry, which in essence resembles an easier classification problem.
Quantization Design for Valid Motion Representation.
To extract valid gesture patterns, we haveto preserve certain characteristics of motion representation.
Especially, the covariance matrix Cshould be symmetric positive definite (Eq. 1), and the affine transformation A is determined by Cthrough SVD (Eq. 2).
Therefore, instead of naively quantize each component in Eq.
5, we propose toonly quantize the shift-translation µ and covariance matrix C , while derive the affine transformation
A with SVD.
The only constraint is to guarantee that the covariance matrix C is symmetric positivedefinite.
To satisfy such requirement, we use the unique cholesky decomposition theorem [49]:
Theorem 1.
For any real symmetric positive definite matrix C ∈ S n ++ , there exists a unique lowertriangular matrix L with positive diagonal entries, such that C = LL T .
In this way, we turn to quantize the lower triangular matrix L = (cid:18) l
[Equation]
l [Equation] (cid:19) , where the constraint is much simpler as l 1 , l 3 > 0 .
The updated quantization scheme with cholesky decomposition is:
[Equation]
where e L , e q L , D L and d L denote the encoded feature, quantized feature, codebook and codebookentry for factorial covariance L , respectively.
A simple transformation of l 1 , 3 = ReLU ( l 1 , 3 ) + ϵguarantees the diagonal entries to be positive, where ϵ is a small positive number.
The motioncomponent C and A can be further obtained by LL T and SVD calculation, respectively.
Position-Irrelevant Motion Pattern.
Another problem arises when we inspect the value of themotion representation: Since M is extracted in the image pixel space, the object location will affectthe element in M .
For example, if a person poses the same gesture at different image regions, themotion component differs yet the underlying motion pattern remains the same.
Thus we focus on aimage location invariant motion pattern representation.
In particular, due to the linear additiveness,the relative shift-translation µ between adjacent frames can be represented as ∆ µ j = µ j − µ j − 1 , andthe relative change of the lower triangular matrix is ∆ L j =
L j − L j − 1 for any j = 2 , . .
. , N .
Notethat with the uniqueness of cholesky decomposition, ( L + ∆ L ) corresponds to the sole covariancematrix C
=
( L + ∆ L )( L + ∆ L ) T , which further determines affine matrix A by SVD.
In this way,the term ∆ L is sufficient to represent any relative affine transformation between two frames.
We accordingly update the quantization scheme with position-irrelevant motion pattern representation as:
[Equation]
where e { ∆ µ, ∆ L } , e q { ∆ µ, ∆ L } , D { ∆ µ, ∆ L } and d { ∆ µ, ∆ L } are the encoded feature, quantized feature,codebook and entry for the relative shift-translation ∆ µ and factorial covariance ∆ L , respectively.
Overall Quantized Motion Pattern Learning.
With the position-irrelevant motion pattern, thecodebook naturally contains reusable common co-speech gesture patterns M pattern .
The encoders
E ∆ µ , E ∆ L and the decoders D ∆ µ , D ∆ µ are jointly learned with the codebooks D ∆ µ and D ∆ L via:
[Equation]
where sg denotes the stop gradient operation, β 1 and β 2 are two weight balancing coefficients.
3.3 Co-Speech Gesture GPT with Motion Refinement
Co-Speech Gesture GPT Network.
With the position-irrelevant motion pattern of valid quantizationdesign, each co-speech gesture clip can be transformed into discrete representation.
We then learn a cospeech gesture GPT network to map from speech audio a (1: T ) to quantized code sequences e q ∆ µ, (1: T ′ )and e q ∆ L, (1: T ′ ) .
Specifically, we extract audio features a onset (1: T ′ ) with onset strength information, whichis more suitable for cross-modal pattern learning [43, 46].
Then, a feature embedding layer withpositional embedding is leveraged to obtain the tokens for audio onset features, quantized relativeshift-translation and qunatized relative factorial covariance.
Further, we encode cross-attentioninformation with a series of transformer layers.
Finally, followed by a linear transformation withsoftmax activation, the M -dimensional output denotes the probability of each quantization code atthat time step.
The whole co-speech gesture GPT is trained with cross-entropy loss L CE .
Such designenables us to predict and sample future quantization code e q ∆ µ and e q ∆ L with speech audio.
Motion Refinement by Learning Residuals.
Now that we can reconstruct the relative shifttranslation (cid:99) ∆ µ and factorial covariance change (cid:100)
∆ L by VQ-VAE decoding.
Given µ 1 and L 1 extractedfrom the initial image frame I (1) , the absolute shift-translation and factorial covariance for the j -thframe can be calculated as (cid:99) µ j = µ 1 + (cid:80) ji =2 (cid:100) ∆ µ i and (cid:99)
L j
=
L 1 + (cid:80) ji =2 (cid:100)
∆ L i , respectively.
However,since the quantized codebook is designed to only represent the large-scale common motion patterninformation, while fine-grained rhythmic details are omitted.
Therefore, we propose to refine theco-speech movements by learning residual terms.
Concretely, we extract the audio mfcc featuresa mfcc (1: T ) to encode more contextual audio cues for prosodic dynamics learning.
Then a bi-directional
LSTM is used to predict the per-frame motion representation residuals R to the main motion patternresult (cid:98) µ (1: T ) and (cid:98) L (1: T ) , i.e., (cid:99) M rhythmic (1: T ) =
(cid:104) R ( (cid:98) µ (1: T ) ; a mfcc (1: T ) ); R ( (cid:98) L (1: T ) ; a mfcc (1: T ) )
(cid:105) .
By adding residualterms, the overall co-speech gesture GPT with motion refinement learning can be formulated as:
L Residual = ||M (1: T ) − (cid:99)
M (1: T ) ( a ) || , where (cid:99) M (1: T ) ( a ) = (cid:99)
M pattern (1: T ) ( a onset ) +
(cid:99) M rhythmic (1: T ) ( a mfcc ) .
In this way, we capture both major gesture patterns and subtle rhythmic dynamics for vivid results.
Experiments
4.1 Experimental Settings
Dataset and Preprocessing.
Pose, Audio, Transcript, Style (PATS) is a large-scale dataset of 25speakers with aligned pose, audio and transcripts [1, 2, 21].
The training corpus contains 251 hoursof data with around 84,000 intervals of mean length 10.7s.
Notably, the PATS dataset contains threemodalities of audio log-mel spectrograms, speech transcripts and per-frame skeletons labeled with
OpenPose [9].
To bypass the error accumulation in pose annotation and facilitate co-speech gestureimage generation task, we extend PATS with more features: 1) preprocessed image frames and 2)onset strength audio features which are more suitable for co-speech gesture pattern learning.
[Table]
We conduct the experiments on four speakers’ co-speech video subsets, including Oliver, Seth,
Kubinec and Jon.
Concretely, 2D skeletons of the image frames are obtained by OpenPose [9] forbaseline methods training.
The frames are cropped to make the speaker locate at the image center.
Since the time span of a meaningful co-speech gesture unit sequence ranges from 4s to 14s [27, 44],we trim invalid videos and finally obtain 1306, 990, 1294 and 1284 clips for four subsets, respectively.
The overall mean clip length is 9.8s.
We randomly split the segments into 90% for training and 10%for evaluation.
The image frames are sampled at 25 fps and further resized to 256 × 256 .
Comparison Methods.
We compare with recent SOTA works: 1) Speech to Gesture (S2G) [21], a
GAN-based pipeline that maps audio to 2D keypoints with a U-Net; 2) Hierarchical Audio to Gesture(HA2G) [33] which captures the hierarchical associations between multi-level audio features and treelike human skeletons; 3) Speech Drives Template (SDT) [37] which relieves the one-to-many mappingambiguity by a set of continuous gesture template vectors; 4) Trimodal Context (TriCon) [59], arepresentative framework that considers the trimodal context of audio, text and speaker identity.
Notethat all methods could drive 2D human skeletons with speech audio.
We train baselines on the PATSimage dataset and tune the hyper-parameters by grid search for the best evaluation result.
In particular,we also show direct evaluations on the Ground Truth (GT) skeletons for clearer comparison.
Implementation Details.
We sample T = 96 frame clips with stride 32 for training.
For the VQMotion Extractor: the co-speech gesture pattern codebook size M for both relative shift-translation∆ µ and factorial covariance change ∆ L are set to 512 .
The encoders E ∆ µ , E ∆ L and the decoders
D ∆ µ , D ∆ µ are based on 1D-convolution structure.
The channel dimension ℓ of each codebook entryd ∆ µ , d ∆ L as well as the encoded latent features e ∆ µ , e ∆ L are 512 , while the temporal dimension
T ′ is set as T/ 8 = 12 to encode contextual features with downsampling rate of 8 .
The ϵ is set as1 × 10 − 5 to guarantee the positiveness of diagonal entries in factorial covariance L .
The commit losstrade-offs in L VQ are empirically set as β 1 = β 2 = 0 .
We optimize the gesture pattern VQ-VAEwith Adam optimizer [28] of learning rate 3 × 10 − 5 . 2)
For the Co-Speech GPT: the Transformerchannel dimension is 768 , and the attention layer is implemented in 12 heads with dropout probabilityof 0 .
The onset strength audio features a onset ∈ R 426 are extracted by Librosa, while the audio mfccfeatures a mfcc ∈ R 28 × 12 are computed with the window size of 10 ms.
During the GPT training, thee q ∆ µ, (1:11) , e q ∆ L, (1:11) are used as input while e q ∆ µ, (2:12) , e q ∆ L, (2:12) serve as supervision labels.
Forthe motion representation M and image generator G : we implement as MRAA [42] to use K = 20regions.
The motion estimator is pretrained for knowledge distillation.
The overall framework isimplemented in PyTorch [35] and trained on one 16G Tesla V100 GPU for three days.
4.2 Quantitative Evaluation
Evaluation Metrics.
We adopt 1) Fréchet Gesture Distance (FGD) [59] to evaluate the distancebetween the real and synthetic gesture distribution.
Following Yoon et al.
, we train an auto-encoderon the PATS image dataset and use the encoder to compute fréchet distance between the real andsynthetic gesture in feature space.
We also use the 2) Beat Consistency Score (BC) and 3) Diversity(Div.)
[31, 33] to account for the speech-motion alignment and the diversity among generated gestures.
Specifically, BC is computed as the average temporal distance between each audio beat and its closestgesture beat, and Diversity indicates the difference of multiple audios’ corresponding gestures inthe latent space, where the features are extracted by the pretrained encoder.
Note that since all themetrics are skeleton-based, we downgrade our results from image to 2D keypoints
[Table]
Evaluation Results.
The results are reported in Table 1.
It can be seen that the proposed ANGIEachieves the best evaluation results on most metrics.
Since our method summarizes reusable cospeech gesture patterns into quantized codebooks and complements subtle rhythmic dynamics, wecan cover richer gesture patterns and create diverse results.
Note that HA2G [33] tends to generateover-expressive gestures with multi-level audio features, which makes their results on BC even betterthan the ground truth in some cases.
Despite of this, we perform comparable to ground truth on BCmetric with stable motion results, showing that we can generate audio-aligned gestures.
Besides, wecan find that both SDT [37] and ours perform better on FGD and Diversity metrics than other methodsdue to the explicit modeling of co-speech gesture patterns.
However, since the gesture pattern is finiteand discrete, our quantized codebook design is more suitable than continuous representation.
4.3 Qualitative Analysis
User Study.
We further conduct a user study to reflect the quality of audio-driven gestures.
Concretely,we sample 25 audio clips from the PATS image test set for all methods to generate results, then involve18 participants for user study.
The Mean Opinion Scores protocol is adopted, which requires theparticipants to rate three aspects: (1) Realness ; (2) Synchrony ; (3) Diversity .
The rating scale is 1 to 5,with 1 being the poorest and 5 being the best.
The results are reported in Table 2, where our methodperforms the best on all three aspects.
Notably, with the help of motion pattern codebook, we achievecomparable diversity result to the ground truth, demonstrating that the quantization design helps tocapture common gesture patterns.
Besides, we measure the participants’ scoring disagreement with
Fleiss’s-Kappa statistic.
The value is 0 . 806 , which indicates “almost perfect agreement”.
Video Generation Results.
With the Co-Speech GPT and motion refinement network, we couldpredict the future quantization code and complement rhythmic motion details based on audio features,which enables us to generate video results.
As shown in Fig.
3, the synthesized image sequenceresults from left to right contain diverse and meaningful gestures that are aligned with speech audio.
Codebook Analysis for Co-Speech Gesture Pattern.
We analyze the meaning of the quantizedcodebooks by asking the question: Does each codebook entry really correspond to a certain type ofmotion pattern?
To answer this, we validate two things in Fig.
4: 1) Different entries represent diversegesture patterns, hence driving the same image with different quantized codes leads to differentgestures (left).
Each codebook entry denotes a fixed motion pattern, hence driving different imageswith the same quantized code show same motions (right).
Please refer to demo video for more results.
[Table]
[Figure]
4.4 Ablation Study
In this section, we present ablation study of two modules in our framework.
Note that except forthe skeleton-based metrics, we also use video reconstruction accuracy as a proxy for image quality,including L 1 and perceptual loss [26] between the reconstructed and GT image; Average Euclidean
Distance (AED) evaluates identity preservation by pretrained re-identification networks [24, 42].
VQ-Motion Extractor.
We conduct ablation experiments on five settings: 1) w/o Quantization,where we directly infer motion representation with audio; 2) Quantize µ , L ; 3) Quantize ∆ µ , L ; 4)
Quantize µ , ∆ L , where we quantize the absolute or relative difference of the motion components; 5)w/o Cholesky Decomposition, which means the covariance matrix is not guaranteed to be symmetricpositive definite.
The results are shown in Table 3, which proves that the quantization designwith position-irrelevant relative motion pattern could improve the performance.
We also find thatquantizing ∆ µ is more effective than ∆ L , since the shift translation is more correlated with regionposition.
Note that when motion component is invalid, the pipeline fails to generate image due to thenumerical instability in calculating affine matrix.
Hence the results for the setting 5 are not reported.
Motion Refinement Module.
To verify the efficacy of motion residuals, we eliminate the motionrefinement for ablation.
The results in Table 3 suggest that this module improves beat consistencyby capturing subtle rhythmic movements, while the improvement on FGD majorly derives from thequantization design.
Such module complements the motion pattern for fine-grained results.
Discussion
Conclusion.
In this paper, we propose a novel framework ANGIE to generate audio-driven cospeech gesture in the image domain.
To summarize valid common co-speech gesture patterns, wepropose VQ-Motion Extractor with cholesky decomposition based quantization scheme and positionirrelevant design to represent relative motion patterns.
Then we propose Co-Speech GPT to refinesubtle rhythmic movement details for fine-grained results.
Extensive experiments demonstrate thatour framework renders realistic and vivid co-speech gesture image generation results.
Ethical Consideration.
Generating co-speech gesture images facilitates applications such as digitalhuman.
However, it could be misused for malicious purposes like forgery generation.
We believe thatthe proper use of this technique will enhance the machine learning research and digital entertainment.
Limitation and Future Work.
As an early work towards audio-driven co-speech image generationwithout structural prior, we notice that our method fails for some challenging cases.
For example, ifthe source image is in a large pose, it is hard to generalize well in such an out-of-domain data.
Wewill explore how to develop a model with higher generalization ability in future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] The limitations are discussed in
Section 5.
More details and analysis are provided in the supplemental document.(c)
Did you discuss any potential negative societal impacts of your work?
[Yes]
The potential negative societal impacts are discussed in Section 5.
More ethical considerations with potential measures are provided in the supplemental document.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Theassumptions of the unique cholesky decomposition theorem are the real symmetricpositive definite characteristic of the matrix, which are stated in the Theorem 1.
(b) Did you include complete proofs of all theoretical results?
[Yes] The proofs are
included in the supplemental document.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)?
[N/A]
We
include very detailed implementation instructions in Section 4.1 to help reproduce the
main experimental results.
Besides, although the code and data are not included, as
promised we will make the code, models and data publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] They are specified in the Implementation Details.
Please see
Section 4.1.
(c) Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)?
[Yes] Please see the supplemental document for
evaluation results with error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g.,
type of GPUs, internal cluster, or cloud provider)?
[Yes] They are specified in the
Implementation Details.
Please see Section 4.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] The citations are all
included in the reference, and the detailed licenses are included in the supplemental
document.
(b) Did you mention the license of the assets?
[Yes] The citations are all included in the
reference, and the detailed licenses are included in the supplemental document.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
The citations are all included in the reference, and the detailed licenses are included in
the supplemental document.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] Our work and data are based on publicly available dataset and
used for the academic use exclusively.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] The discussions are elaborated in the
supplemental document, where all the data and models are for the academic use
only.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[Yes] The detailed user study instructions with details are provided in the
supplemental document.
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[Yes] Since the user study merely requires the
participants to watch and evaluate the co-speech gesture generation results, there is no
potential participant risk.
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[Yes] We pay the user study participants for their
efforts.
The details are provided in the supplemental document.