Vector Quantized Diffusion Model with CodeUnet for Text-to-Sign Pose Sequences Generation
Anonymous Author(s)
AffiliationAddress email
Abstract
Sign Language Production (SLP) aims to translate spoken languages into signsequences automatically.
The core process of SLP is to transform sign glosssequences into their corresponding sign pose sequences (G2P).
Most existing G2Pmodels usually perform this conditional long-range generation in an autoregressivemanner, which inevitably leads to an accumulation of errors.
To address this issue,we propose a vector quantized diffusion method for conditional pose sequencesgeneration, called PoseVQ-Diffusion, which is an iterative non-autoregressivemethod.
Specifically, we first introduce a vector quantized variational autoencoder(Pose-VQVAE) model to represent a pose sequence as a sequence of latent codes.
Then we model the latent discrete space by an extension of the recently developeddiffusion architecture.
To better leverage the spatial-temporal information, weintroduce a novel architecture, namely CodeUnet, to generate higher quality posesequence in the discrete space.
Moreover, taking advantage of the learned codes,we develop a novel sequential k-nearest-neighbours method to predict the variablelengths of pose sequences for corresponding gloss sequences.
Consequently,compared with the autoregressive G2P models, our model has a faster samplingspeed and produces significantly better results.
Compared with previous nonautoregressive G2P methods, PoseVQ-Diffusion improves the predicted results withiterative refinements, thus achieving state-of-the-art results on the SLP evaluationbenchmark.
Introduction
Sign Language Production (SLP), as an essential task for the Deaf community, aims to providecontinuously sign videos for spoken language sentences.
Since sign languages are distinct linguisticsystems [1] which differ from natural languages, sign languages have different word orders from theircorresponding natural languages.
Therefore, directly learning the alignment mapping between themis challenging.
To tackle this issue, previous works first translate spoken languages into glosses 1 , thengenerate the sign pose sequences based on the gloss sequences (G2P) [2, 3] and finally optionallyuse the sign pose sequence to generate the photo-realistic sign video [4].
Accordingly, G2P is theheart procedure of this task, and it is the focus of this paper.
Existing approaches for G2P can be categorized into autoregressive [2, 3] and non-autoregressive [5]methods depending on their decoding strategies.
Autoregressive models [2, 3] generate the next poseframe depending on previous frames relying on the teacher forcing strategy [6].
In inference, therecurrent decoding is likely to lead to prediction error propagation over time due to the exposurebias [7].
To break the bottleneck of autoregression, non-autoregressive methods are proposed to induce the decoder to generate all target predictions simultaneously [8, 9].
Huang at al.
[5] proposeda non-autoregressive G2P model to generate sign pose sequence parallelly in a one-shot decodingscheme, and used an External Aligner (EA) for sequence alignment learning.
Motivated by the recent developed Discrete Denoising Diffusion Probabilistic Model (D3PMs) [10,11, 12] which achieved impressive results for language generation and vector quantized imagegeneration.
We propose a Pose Vector Quantized Diffusion (PoseVQ-Diffusion) model to learn theconditional pose sequence generation in the latent discrete space instead of the continuous coordinatespace.
It is also a non-autoregressive method that performs parallel refinement on the generatedresults with iterations and therefore shows expressive generative capacity.
We will elaborate our approach in three steps.
Firstly, we utilize a vector quantized variationalautoencoder (VQ-VAE) to represent the pose sequence as sequential latent codes.
Different fromimage VQ-VAE [13, 14], we devise a specific architecture, Pose-VQVAE, to learn the meaningfulcodebook by reconstructing the pose sequence.
Specifically, we divide a sign skeleton into three localpoint patches representing pose, right hand and left hand separately.
Furthermore, a Tokenizer with avector quantized variational autoencoder is designed to learn discrete point tokens containing localsemantic information.
Next, we extend the standard vector quantized diffusion methods [11, 12] to model the sequentialalignments between sign glosses and quantized codes.
The discrete diffusion model samples the datadistribution by reversing a forward diffusion process that gradually corrupts the input via
a fixed
Markov chain.
Its corruption process by adding noise data ( e.g. , [MASK] token) draws our attentionto the mask-based generative model, Mask-Predict [9], which is proved to be a variant of diffusionmodel [11].
In this paper, we explore two variants of the diffusion model for our quantized posesequence generation.
Expanding further, to better leverage the spatial and temporal information of thequantized pose sequences, we introduce a new architecture CodeUnet.
In contrast to Unet [15] whichis a “fully convolution network” for image data, CodeUnet is a “fully transformer network” designedfor discrete tokens.
As a result of iterative refinements and better spatial-temporal modelling, ourmodel achieves a higher quality of the conditional pose sequence generation.
Lastly, the length prediction of the non-autoregressive G2P models is challenging since the corresponding lengths of different sign glosses are different and variable.
In this paper, we propose a novelclustering method for this typical sequential data that local adjacent frames should belong to a cluster.
Specifically, taking advantage of the meaningful learned codes in the first stage, we firstly apply thek-nearest-neighbor based density peaks clustering algorithm [16, 17] to locate the peaks with higherlocal density.
Secondly, we design a heuristic algorithm to find the boundary between two peaksaccording to their semantic distance with the two peak codes.
Finally, we leverage the length of eachgloss as the additional supervised information and predict the length of the gloss sequence in theinference.
Our model significantly improves the generation quality on the challenging RWTH-PHOENIXWEATHER-2014T [18] dataset.
The evaluation of conditional sequential generation is evaluatedusing a back-translated model.
Extensive experiments show that our model increases the WER scorefrom 82 .
01% [5] to 78 .
21% on generated pose sequence to gloss sequence, and BLEU score from6 .
66 [5] to 7 .
42 on generated pose sequence to spoken language.
Related Works
Sign Language Production.
Most sign language works focus on sign language recognition (SLR)and translation (SLT) [18, 19, 20, 21, 22, 23], aiming to translate the video-based sign languageinto text-based sequences.
And few attempts have been made for the more challenging task of signlanguage production (SLP) [24, 25].
Stoll et al.
proposed the first deep SLP model, which adopts thethree-step pipeline.
In the core process for G2P, they learn the mapping between the sign glosses andthe skeleton poses via a look-up table.
After that, B. Saunders et al.
[3] proposed the progressivetransformer to learn the mapping with an encoder-decoder architecture and generate the sign posein an autoregressive manner in the inference.
Further, B. Saunders et al.
[4] proposed a Mixture
Density Network (MDN) to generate the pose sequences condition on the sign glosses and utilize a
GAN-based method [26] to produce the photo-realistic sign language video.
B. Saunders et al.
[27]translates the spoken language to sign language representation with an autoregressive Transformernetwork, and uses the gloss information to provide the additional supervision.
Then they propose a
SP L
T r an s f o r m e r
Points Embs.
Encoder
Decoder
L i nea r
T r an s f o r m e r T ok e n i ze r
Different from these autoregressive methods, Huang et al.
[5] proposed a non-autoregressive modelto parallelly generate the sign pose sequence avoiding the error accumulation problem.
They applythe monotonic alignment search [28] to generate the alignment lengths of each gloss.
Our modelalso explores a non-autoregressive method with a diffusion strategy, and the adopted diffusion modelarchitecture provides us with a chance to refine the results with multiple iterations.
Denoising Diffusion Probabilistic Models.
Diffusion generative models have achieved outstandingresults on continuous data, such as image generation [29, 30, 31, 32, 33] and speech synthesis [34, 35,36].
However, most previous works focus on Gaussian diffusion processes that operate in continuousstate spaces.
The discrete diffusion model is first introduced in [37], and it is applied to text generationin Argmax Flow [10].
To improve and extend the discrete diffusion model, D3PM [11] use a structuredcategorical corruption process to shape data generation and embed structure in the forward process.
VQ-Diffusion [12] apply the discrete diffusion model to conditional vector quantized image synthesis [Figure] with a mask-and-replace diffusion strategy. [Figure]
The Proposed Method
[Figure]
The overall objective of this work is to extend the discrete diffusion model for conditional sign posesequence generation.
The proposed PoseVQ-Diffusion model consists of three key components, PoseVQVAE to learn the latent codes, a discrete diffusion model with CodeUnet to model the discretecodes generation, and a sequential-KNN algorithm on the length prediction for this non-autoregressivemethod.
3.1 Pose VQ-VAE
In this section, we introduce how to tokenize the points of a sign pose skeleton into a set of discretetokens.
A naive approach is to treat per point as one token.
However, such a points-wise reconstructionmodel tends to tremendous computational cost due to the quadratic complexity of self-attentionin Transformers.
On the other hand, since the details of hand points are essential for sign poseunderstanding, treating all the points into one token leads to remarkable inferior reconstructionperformance.
To achieve a better trade-off between quality and speed, we propose a simple yetefficient implementation that groups the points of a sign skeleton into three local patches, representingpose, right hand and left hand separately.
Figure 1 illustrates the framework of our proposed PoseVQVAE model with the following submodules.
Encoder.
Given a sign pose sequence of N frames s = ( s 1 , s 2 , ..., s n , ..., s N ) ∈ R N × J × K , where{ x jn } Jj =1 presents a single sign skeleton containing J joints and K denotes the feature dimensionfor human joint data.
We separate these points into three local pathes, s p ∈ R N ×
( J p × K ) , s r ∈
R N × ( J r × K ) , s l ∈ R N ×
( J l × K ) for pose, right hand and left hand respectively, where J =
J p + J r + J l .
In the encoder module E ( e | s ) , we first transform these three points sequences into feature sequencesby simple three linear layers and concatenate them together.
Then we apply a spatial-temporal
Transformer network to learn the long-range interactions within the sequential point features.
Finally,we arrive at the encoded features { e n ∈ R 3 × h } N n =1 .
Point Tokenizer.
Similar to image VQ-VAE [14], we take the encoded features as inputs and convertthem into discrete tokens.
Specifically, we perform the nearest neighbors method Q ( z | e ) to quantizethe point feature to the quantized features { z n ∈ R 3 ×
M M 1 M M 103 M M M M ···
Gloss Sequence: MORGEN MAL SONNE SPEZIELL
Transformer Encoder
CodeUnet
N x
N / 2 x
N / 4 x
Transformer-based Encoder downsample
Transformer-based Encoder-Decoder upsample N / 4 x
N / 2 x
N x
M M M M M
M M 1 M M 103 M M M M
M 42 M 23 M
M 23 16 19 M 103 M 98 48 M
M 99 M 23 M 12 72 16 26103 180 98 4827 99 47 23 90 ···
N xcopy and add
Linear and softmax
Length Predictor
Decoder.
The decoder D (˜ s | z ) receives the quantized features as inputs and also applies spatialtemporal Transformer to get the output features { o n ∈ R 3 × h } N n =1 .
Finally, we separate the outputfeature for three sub-skeleton and utilize a structured prediction layer (SPL) [38] P (˜ s | o ) to reconstructthe corresponding sub-skeleton ˜ s p ∈ R N × ( J r × K ) , ˜ s l ∈ R N × ( J r × K ) , ˜ s r ∈ R N × ( J r × K ) .
We adoptthe SPL to rebuild the skeleton from feature because it explicitly model the spatial structure of thehuman skeleton and the spatial dependencies between joints.
The hierarchy chains of the pose, righthand and left hand skeleton are given in supplemental material.
Training.
The encoder E ( e | s ) , tokenizer Q ( z | e ) and decoder D (˜ s | z ) can be trained end-to-end viathe following loss function:
[Equation]
where sg [ · ] stands for stop-gradient operation.
In practice, we replace the forth term with exponentialmoving averages (EMA) to update the codebook.
3.2 Discrete Diffusion Model with CodeUnet
To allow conditional sampling, a discrete diffusion model is trained on the latent codes obtained fromthe Pose-VQVAE model.
Figure 2 shows the architecture of our proposed PoseVQ-Diffusion, whichaims to model the latent space in a non-autoregressive manner.
We will subsequently introduce thediffusion process, reverse denoising process and the parametered model CodeUnet.
Diffusion Process.
Given a sequence of latent codes x 0 ∈ R N × 3 obtained from the vector quantizedmodel, where x ( i,j )∈ { 1 , 2 , ..., V } at location ( i, j ) represents the index within the codebook.
Thediffusion process aims to corrupt the original data x 0 via a fixed Markov chain p ( x t | x t − 1 ) by addingsmall amount noise continuously.
After a fixed T timesteps, it produces a sequence of increasinglynoisy data x 1 , .., x T with the same dimensions as x 0 , and x T becomes a pure noise sample.
For the scalar discrete variables with V categories x ( i,j ) t ∈ [1 , V ] , the forward transition probabilitiesfrom x t − 1 to x t can be represented by matrices [ Q t ] mn = q ( x t = m | x t − 1 = n ) ∈
R V × V .
Note thatwe omit the superscripts ( i, j ) to avoid confusion.
Then the forward diffusion process can be writtenas:
[Equation]
where x t ∈ R V × 1 is the one-hot version of x t and Q t x t − 1 is the categorical distribution for x t .
Anice property of the above Markov diffusion process is that we can sample x t as any timestep directlyfrom x 0 as:
[Equation]
D3PM [11] formulate the transition matrix Q t ∈ R V × V by introducing a small number of uniformnoises to the categorical distribution.
As formulated as the first matrix in Eq.
(4) with α ∈ [0 , 1] andβ t =
(1 − α t ) /V .
It can be interpreted as each token having a probability of α t + β t to remain theprevious value and a probability of β t to be the value from the whole V categories.
Based on D3PM,
VQ-Diffusion [12] propose a mask-and-replace diffusion strategy that not only replaces the previousvalue but also insert [MASK] token to explicitly figure out the tokens that have been replaced.
Weextend this mask-and-replace strategy to our quantized pose sequence modelling.
Since the length ofpose sequences may be different in a minibatch, we have to add two special tokens, [MASK] and[PAD] tokens, so each token has V + 2 states.
The mask-and-replace diffusion process can be definedas follows: each token has a probability of α t to be unchanged, V β t to be uniformly resampled andγ t = 1 − α t − V β t to be replaced with [MASK] token.
Note that [MASK] and [PAD] tokens alwayskeep its own state.
The difference is that [PAD] is used to represent the padding part in the initialsequence x 0 , and [MASK] is used to replace the original token of the code sequence in the diffusionprocess.
Moreover, in the revised denoising process, the [MASK] positions are required to predictbut [PAD] positions need to be ignored.
The transition matrix Q t ∈ R ( V +2) × ( V +2) is formulated asthe second matrix of the following:
[Equation]
{+β t 0 
+}
β t α t + β t · · · β t 0...
β t β t · · · α t + β t 0 0 γ t γ t · · · γ t 1 0 · · · 0  .
Finally, the categorical distribution of x t can be derived as following using reparameterization trick:when x 0 ̸ = V + 2 ,
[Equation]
and x t
[Equation]
t
[Equation]
when x 0 = V + 2 ,
[Equation]
where ¯ α t = (cid:81) ti =
1 α
i , ¯ γ t = 1 − (cid:81) ti =1 (1 − γ i ) , and ¯ β t
=
(1 − ¯ α t − ¯ γ t ) /V .
Therefore, we can directlysample x t within the computation cost O ( V ) .
Reverse Denoising Process.
The reverse denoising process aims to recreate the real sample from afull noise input by gradually sampling from q ( x t − 1 | x t ) .
However, it is intractable to estimate theconditional probability q ( x t − 1 | x t ) since it needs to use the whole dataset.
Fortunately, the conditionalprobability is tractable when conditioned on x 0 using Bayes’ rule:
[Equation]
thus we train a denoising model p θ ( x t − 1 | x t , c ) to approximate the tractable distributionq ( x t − 1 | x t , x 0 ) , where c is the conditional feature of gloss sequence.
And the model is trainedto minimize the variational lower bound [37]:
[Equation]
Reparameterization Trick on Model Learning.
Compared with directly predicting p θ ( x t − 1 | x t , c ) ,recent works [30, 10, 12] find that predict the data x 0 gives better quality at every reverse step.
Thus, we let our denoising model to predict the distribution p θ (˜ x 0 | x t , c ) .
With a reparameterization trick,the conditional reverse distribution can be formulated as:
[Equation]
Under this x 0 -parameterization trick, we introduce an auxiliary denoising objective to encouragegood predictions of the data x 0 at each time step [11].
The final loss function is combined with thenegative variational lower bound and the auxiliary loss:
[Equation]
where λ is a coefficient for the auxiliary loss term.
CodeUnet for Model Learning.
Most image diffusion models [29, 30, 39] adopt the Unet [15] astheir architectures since it is effective for data with spatial structure.
However, directly applying the
Unet in discrete sequence generation, e.g. , text generation [11] and quantized image synthesis
[12],will bring information leakage problem, since the convolution layer over adjacent tokens may provideshortcuts for the mask-based prediction [40].
Therefore, Austin et al.
[11] and Gu et al
[12] use thetoken-wise Transformer framework to learn the distribution p θ (˜ x 0 | x t , c ) .
In this work, to incorporatethe advantages of Unet and Transformer networks, we propose a novel architecture CodeUnet tolearn the spatial-temporal interaction for our quantized pose sequence generation.
As shown in Figure 2, the CodeUnet consists of a contracting path (left side), an expansive path (rightside) and a middle module.
The middle module is an encoder-decoder Transformer framework.
Theencoder consists of 6 Transformer blocks.
It takes the gloss sentence as input and obtains a conditionalfeature sequence.
The decoder has two blocks.
Each block has a self-attention, a cross-attention, afeed-forward network and an Adaptive Layer Normalization [41, 12](AdaLN).
The AdaLN operatoris devised to incorporate timestep t information as AdaLN ( h, t ) =
α t LayerNorm ( h ) + b t , whereh is the intermediate activations, α t and β t are obtained from a linear projection of the timestepembedding.
Both contracting path and the expansive path are hierarchical structures and each level has two
Transformer encoder blocks.
For downsampling in contracting path, given the feature of quantizedpose sequence, e.g. , h ∈ R N × 3 × d model , where d model is the feature dimension, we first sample uniformlywith stride 2 in the temporal dimension and remain constant in the spatial dimension.
Then we set thedownsampled feature as query Q ∈ R N/ 2 × 3 × d model , and keep key K and value V unchanged for thefollowing attention network.
In the upsampling of expansive path, we directly repeat the feature 2times as a query, but the key and value remains for the following attention network:
[Equation]
where · // · denotes floor division.
Finally, a linear layer and a softmax layer are applied to make theprediction.
3.3 Length Prediction with Sequential-KNN
In this section, inspired by [17] which merges tokens with similar semantic meanings from differentlocations, we propose a novel clustering algorithm to get the lengths for corresponding glosses.
Specifically, given a token sequence which is obtained from the Pose-VQVAE model, we computethe local density ρ of each token according to its k-nearest-neighbors:
[Equation] [Equation]
where i, j is the position in the sequence, l is a predefined hyperparameter indicating that we onlyconsider the local region since the adjacent tokens are more likely to belong to a gloss.
z i and z j arethe latent feature for i th and j th tokens.
KNN ( x i ) represents the k-nearest neighbors for i th token.
We assign { p 1 , ..., p M } positions with a higher local density as the peaks, where M is the lengthof the gloss sequence.
Then between two adjacent peaks, for example p 1 and p 2 , we sequentiallyiterate from p 1 to p 2 , and find the first position that is farther from z p 1 and closer to z p 2 , which is the boundary we determined.
After finding these boundaries, we get the lengths of the contiguouspose sequence for its corresponding glosses.
As shown in Figure 2, we define the obtained lengthsas { L 1 , .., L M } , and the Transformer encoder for gloss sequence is trained under the supervisedinformation of lengths.
For each gloss word, we predict a number from [1 , P ] , where P is themaximum length of the target pose sequence.
Mathematically, we formulate the classification loss oflength prediction as:
[Equation]
In the training of the discrete diffusion mode, L len is trained together with a coefficient δ .
In theinference, we predict the length of glosses, and their summation is the length of target pose sequence.
In summary, we arrive at our proposed two-stage approach, PoseVQ-Diffusion, with the first-stage
Pose-VQVAE model and the second-stage discrete diffusion model with a length predictor.
Thewhole training and inference algorithm is shown in supplementary material.
Experiments
4.1 Experiment Setups and Implementation Details
Dataset.
We evaluate our G2P model on RWTH-PHOENIX-Weather 2014T (RPWT) dataset [18].
Itis the only publicly available SLP dataset with parallel sign language videos, gloss annotations, andspoken language translations.
This corpus contains 7096 training samples (with 1066 different signglosses in gloss annotations and 2887 words in German spoken language translations), 519 validationsamples, and 642 test samples.
Evaluation Criteria.
Following the widely-used setting in SLP [3], we adopt the back-translationmethod for evaluation.
Specifically, we utilize the state-of-the-art SLT [19] model to translate thegenerated sign pose sequence back to gloss sequence and spoken language, where its input is modifiedas pose sequence.
Specifically, we compute BLEU [42] and Word Error Rate (WER) between theback-translated spoken language translations and gloss recognition results with ground truth spokenlanguage and gloss sequence.
Data Processing.
Since the RWTH-PHOENIX-Weather 2014T (RPWT) dataset doesn’t contain poseinformation, we generate the pose sequence as the ground truth.
Following B. Saunders et al.
[3],we extract 2D joint points from sign video using OpenPose [43] and lift the 2D joints to 3D witha skeletal model estimation improvement method [44].
Finally, similar to [24], we apply skeletonnormalization to remove the skeleton size difference between different signers.
Model Setting.
The Pose-VQVAE consists of an Encoder, a Tokenizer, and a Decoder.
The Encodercontains a linear layer to transform pose points to hidden feature with dimension set as 256, a 3-layer
Transformer module with divided space-time attention [45].
The Tokenizer maintains a codebookwith a size set as 2048.
The Decoder contains the same 3-layer Transformer module as the encoderand an SPL layer to predict the structural sign skeleton.
For the discrete diffusion model, we setthe timestep T as 100 .
All Transformer blocks of CodeUnet have d model =
and N depth =2 .
Thesize of local region l in Equation 11, is set as 16 which is the average length of a gloss.
And thenumber of nearest neighbors k is set as 16 .
We optimize our network using AdamW [46] withβ 1 = 0 .
9 and β 2 = 0 .
The learning rate is set to 0 .
after 5000 iterations of warmup.
We trainthe model on 8 NVIDIA Tesla V100 GPUs.
We include all hyperparameters setting and the details ofimplementation in the supplementary material.
4.2 Comparisons with State-of-the-Art Methods
Competing Methods.
We compare our PoseVQ-Diffusion with previous state-of-the-art G2Pmodels.
Progressive Transformer (PTR)
[3] is the first SLP model to tackle the G2P problem in anautoregressive manner.
Since they use the ground-truth first sign pose frame and timing information,their reported results are not comparable to ours.
Thus we adopt the results reported by Huang etal.
NAT-EA [5] propose a non-autoregressive method to directly predict the target pose sequence
[Table]
with the External Aligner (EA) to learn alignments between glosses and pose sequence.
NAT-AT isthe NAT model without EA that uses the decoder-to-encoder attention to learn the alignments.
Quantitative Comparison.
The comparison between our PoseVQ-Diffusion and the competingmethods is shown in Tabel 1.
The row of PoseVQ-AR refs to the vector quantized model withan autoregressive decoder.
The row of PoseVQ-MP refs to the vector quantized model with the
Mask-Predict [9] strategy, which is also a variant of discrete diffusion model [11].
PoseVQ-Diffusionrefs to the vector quantized model with mask-and-replace diffusion strategy.
As indicated in Table 1,both diffusion-based models outperform the state-of-the-art G2P models with relative improvementson WER score by 4 .
→ 78 . 21 ) and on BLEU-4 by 11 .
4% ( 6 . 66 → 7 . 42 ).
This showsthe effectiveness of the iterative mask-based non-autoregressive method on the vector quantizedpose sequence.
In addition, the Mask-Predict strategy is a mask only strategy that is similar to
PoseVQ-Diffusion with ¯ γ T = 1 .
Therefore, PoseVQ-Diffusion achieves better performance than
PoseVQ-MP.
This reflects the mask-and-replace strategy is superior to the mask only strategy.
4.3 Model Analysis and Discussions
We also investigate the effects of different components and design choices of our proposed model.
Analysis of The Design of Pose-VQVAE.
As shown in the first three rows of Table 2a, we studythe design of our Pose-VQVAE model.
Pose-VQVAE-joint means we compress all points into onetoken, Pose-VQVAE-separate means the points are separated into three local patches according to thestructure of a sign skeleton.
Empirically, Pose-VQVAE-separate achieves much better reconstruction(MSE) performance.
This indicates that compressing all skeleton points into one token embedding isnot advisable, leading to information loss.
The second row of Figure 3 shows the sample of sign posesequences reconstructed by Pose-VQVAE-separate.
CodeUnet vs. Transformer.
For a fair comparison, we replace our CodeUnet with Transformernetwork with keeping other settings the same.
As shown in the last three rows of Table 2a, thediffusion-based model with our CodeUnet achieves better performance on the back-translate evaluation.
This phenomenon suggests that the hierarchical structure of CodeUnet makes it particularlyeffective for data with spatial structure.
Moreover, in our experiments with the same batch size,
CodeUnet coverages faster than Transformer.
Having said that, due to sign pose sequences beingtemporally redundant, the compression of CodeUnet in the time dimension makes it more efficient intraining.
Number of Timesteps.
We compare the performance of the model with different numbers of trainingsteps.
As shown in the left two columns of Table 2b, we find that the results get better when thetraining step size is increased from 10 to 100 .
As it increased further, the results seemed to saturate.
Therefore, we set the training step to 100 to trade off performance and speed.
Length Candidates.
Length prediction is essential for a non-autoregressive generation.
Our approachproposes a Sequential-KNN algorithm to learn the lengths for corresponding glosses and then treatthe length prediction as a classification problem.
As shown in the right two columns in Table 2b,we study the performance with different length candidates and compare it with the reference (gold)target sequence length.
The results show that multiple candidates can increase performance, but toomany candidates can even degrade performance.
GT
Pose
[Figure]
PTR
Sign Gloss: BESONDERS OST DEUTSCH LAND MEHR REGEN ODER SCHNEE
Sign Gloss: DONNERSTAG SUEDOST
WEITER WECHSELHAFT NORDWEST MEHR FREUNDLICH SONNE
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Conclusion
In this paper, we presented a novel paradigm for conditional sign pose sequence generation through aniterative non-autoregressive method.
Specifically, we first devise a specific architecture Pose-VQVAEto learn discrete codes by reconstruction.
Then we extend the discrete diffusion model to model thesequential alignments between sign glosses and quantized codes.
And a “fully transformer” network
CodeUnet is proposed for the spatial-temporal information in discrete space.
Finally, we propose asequential-KNN algorithm to learn the length of corresponding glosses and then predict the length asa classification task.
Compared with previous state-of-the-art autoregressive and non-autoregressivemethods, extensive experiments demonstrate the effectiveness of our proposed PoseVQ-Diffusionframework.
Broader Impact and Limitations
We develop a general paradigm for conditional pose generation in this paper.
We do not foresee anynegative ethical/societal impacts at this moment.
Although the proposed PoseVQ-Diffusion proveseffective in conditional sign pose sequence generation, we notice several limitations of our approach.(i)
Although our discrete diffusion model-based model has a faster sampling speed in longer sequencegeneration than traditional autoregressive models, it is much slower than one-shot non-autoregressivemodels.
(ii)
Our proposed two-stage based models are not end-to-end and thus more difficult to trainthan previous methods.
We, therefore, plan to resolve the aforementioned issues and further mitigatethe training and inference speed between the one-shot non-autoregressive method.
Checklist
[Table] • Do the main claims made in the abstract and introduction accurately reflect the paper’s [Table]
contributions and scope?
[Yes]
[Table] • Have you read the ethics review guidelines and ensured that your paper conforms to them? [Table]
[Yes]
[Table]
• Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 6•
Did you describe the limitations of your work?
[Yes] See Section 6•
Did you state the full set of assumptions of all theoretical results?
[Yes]
[Table]
• Did you include complete proofs of all theoretical results?
[Yes] More proofs are insupplemental material.
[Table]
• Did you include the code, data, and instructions needed to reproduce the main experimental
[Table]
results (either in the supplemental material or as a URL)?
[Yes] The code and instructions [Table] are in the the supplemental material
[Table]
• Did you specify all the training details (e.g., data splits, hyperparameters, how they were [Table] chosen)?
[Yes] More training details are in the supplemental material [Table] • Did you report error bars (e.g., with respect to the random seed after running experiments
[Table]
multiple times)?
[No]
[Table]
• Did you include the amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?
[Yes] See Section 4.
[Table]
• If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
• Did you mention the license of the assets?
[No] [Table] • Did you include any new assets either in the supplemental material or as a URL?
[No] [Table] • Did you discuss whether
and how consent was obtained from people whose data you’re [Table] using/curating?
[No] [Table] • Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[No]
[Table]
• Did you include the full text of instructions given to participants and screenshots, if applicable?
[No]
[Table]
• Did you describe any potential participant risks, with links to Institutional Review Board
[Table]
(IRB) approvals, if applicable?
[No]
• Did you describe any potential participant risks, with links to Institutional Review Board
(IRB) approvals, if applicable?
[Yes]