T 2 S - GPT : Dynamic Vector Quantization for Autoregressive Sign Language Production from Text (2024)

Abstract: In this work, we propose a two-stage sign language production (SLP) paradigm that first encodes sign language sequences into discrete codes and then autoregressively generates sign language from text based on the learned codebook. However, existing vector quantization (VQ) methods are fixed-length encodings, overlooking the uneven information density in sign language, which leads to under-encoding of important regions and over-encoding of unimportant regions. To address this issue, we propose a novel dynamic vector quantization (DVA-VAE) model that can dynamically adjust the encoding length based on the information density in sign language to achieve accurate and compact encoding. Then, a GPT-like model learns to generate code sequences and their corresponding durations from spoken language text. Extensive experiments conducted on the PHOENIX14T dataset demonstrate the effectiveness of our proposed method. To promote sign language research, we propose a new large German sign language dataset, PHOENIX-News, which contains 486 hours of sign language videos, audio, and transcription texts. Experimental analysis on PHOENIX-News shows that the performance of our model can be further improved by increasing the size of the training data. Our project homepage is https://t2sgpt-demo.yinaoxiong.cn.

## introduction sign language is a visual language with complex grammatical structures and is the primary means of communication for nearly 70 million deaf people worldwide 1 . research on sign language production @xcite @xcite @xcite and sign language translation @xcite @xcite @xcite has attracted widespread attention. sign language production (slp) is a challenging problem that aims to automatically translate spoken language descriptions into corresponding continuous sign sequences. slp can help deaf people better access information and communicate with others, thereby facilitating their lives, which has important social significance. slp models are expected to learn precise mapping from the spoken language space to the sign language space. early work used 2d or 3d skeleton poses to represent sign language @xcite @xcite , while recent work has suggested using 3d human models, such as smpl-x @xcite , to represent sign language, as it introduces human priors and can better animate @xcite . to learn the mapping between these two different modal spaces, some work uses autoregressive models @xcite , non-autoregressive models @xcite @xcite , or diffusion models @xcite to learn the direct mapping from spoken language text to sign language skeleton poses. @xcite proposed to learn the discrete representation of sign language through vq-vae @xcite and then learn the mapping from text to discrete representation through a discrete diffusion model. however, we found that existing sign language discrete representation methods are fixed-length encodings, as shown in figure 1 , which overlooks the uneven information density in sign language. in addition, many existing works rely on expert-annotated intermediate representations, i.e. glosses, which limit the scalability of the model. in this work, we are inspired by recent advances from learning the discrete representation for generation @xcite @xcite @xcite . specifically, we investigate a twostage framework based on dynamic vector quantized variational autoencoders (dvq-vae) and generative pre-trained transformer (gpt) @xcite ) for text-to-sign language production. in the first stage, as shown in figure 1 , dvq-vae will learn the weights of each frame and the boundaries of the basic semantic units. then, the weighted latent vectors are mapped to discrete code indices. further quantitative analysis of the uneven information density in sign language is provided in section 3. to encourage models to perform variable-length encoding and compress sequence lengths, we propose a novel budget loss. additionally, to preserve the semantic information of the reconstructed sign language sequences, we also introduce a translation auxiliary loss. in the second stage, a gpt-like model is learned to to generate code index sequences from spoken language text. furthermore, since the duration of quantized code in a sequence can also vary dynamically, we further propose a duration transformer to predict the duration of the next code based on the previous code's duration and the current code. the experimental results on the widely used slp dataset @xcite demonstrate that our proposed method achieves superior back translation performance compared to previous approaches. furthermore, throughout the entire development process of image generation and text generation, the scale of the dataset has played a crucial role. a large amount of highquality corpus is also very important for slp tasks. in this paper, we present the largest known german sign language dataset, phoenix-news, which consists of 486 hours of sign language videos, audio, and transcription texts. the native expression, clear hand details, and extensive coverage of our large-scale dataset make it suitable for a variety of sign language research tasks, such as sign language translation and sign language production. based on this dataset, we further explore the impact of training data size on slp tasks. empirical analy-sis shows that the performance of our model can be further improved by increasing the size of the training data. our main contributions are summarized as follows: ## method our overall two-stage framework is depicted in figure 3 , which consists of two stages: dvq-vae and t2m-gpt. in the following, we will first briefly revisit the formulation of vq and then describe our proposed method in detail. ## experiments 6.1 experimental setup we will introduce our experimental setup in this section, including the dataset and evaluation metrics. we will provide the implementation details of the model in the appendix c. dataset and evaluation metrics. we evaluate our proposed t2s-gpt model on the phoenix14t @xcite ) dataset, which is the most commonly used dataset for slp tasks and has been used as a benchmark in many previous slp works @xcite @xcite . the phoenix14t dataset contains 7,096 training samples (with 2,887 words in german spoken language translations), 519 validation samples, and 642 test samples. we use the pose parameter θ in the smpl-x model to represent the sign language pose, and the rotation 6d representation @xcite is used to represent the rotation in the pose. follow-ing the most widely used setting in slp @xcite , we use the back translation metric to evaluate the generation quality. since previous works did not publicly release the weights of their slt models used to calculate the back translation metric, following the previous setting, we train an slt model using the code from @xcite . we provide the details of the sign language representation in the appendix b. we provide the training details of the slt model in the appendix d. ## conclusion in this work, we propose a two-stage text-to-sign model t2s-gpt, which consists of a dynamic vector quantization vae (dvq-vae) and a gptlike autoregressive generation model. our method achieves better performance than the previous stateof-the-art text-to-sign model. in addition, we have collected a new large-scale document-level sign language dataset phoenix-news, and the experimental results show that a larger dataset can still bring additional improvements to our method.