MS 2 SL : Multimodal Spoken Data-Driven Continuous Sign Language Production (2024)

Abstract: Sign language understanding has made significant strides; however, there is still no viable solution for generating sign sequences directlyfrom entire spoken content, e.g., text or speech. In this paper, we propose a unified framework for continuous sign language production, easing communication between sign and non-sign language users. In particular, a sequence diffusion model, utilizing embeddings extracted from text or speech, is crafted to generate sign predictions step by step. Moreover, by creating a joint embedding space for text, audio, and sign, we bind these modalities and leverage the semantic consistency among them to provide informative feedback for the model training. This embedding-consistency learning strategy minimizes the reliance on sign triplets and ensures continuous model refinement, evenwith a missing audio modality. Experiments on How2Sign and PHOENIX14T datasets demonstrate that our model achieves competitive performance in sign language production.

## introduction sign language, a visual language, combines both manual (hand gestures) and non-manual cues for communication. it is specifically designed for the deaf and hearing-impaired community @xcite @xcite . according to the world federation of the deaf, there are 70 million deaf people and more than 200 kinds of sign languages in the world @xcite . improvements in sign language production (slp) can bridge the communication gap between the deaf and hearing @xcite @xcite @xcite . the challenges primarily arise from phonological difference and data scarcity. phonological † corresponding author. difference: signs are composed of various manual and non-manual features @xcite , such as hand gestures, facial expressions and limb movements @xcite @xcite . the differences in phonological structure and means of expression create challenges in modeling the two languages. data scarcity: multimodal high-quality sign language datasets are relatively scarce, and some datasets tend to be specific to a particular language or domain, e.g., american sign @xcite , german weather @xcite . furthermore, hearing impairments hinder pronunciation @xcite , making it strenuous to collect sign video with aligned audio and usually resulting in the lack of auditory information. previous researches @xcite @xcite primarily focused on sign language recognition, which identifies sign fragments as the corresponding sign language lexicons (e.g., gloss). several work @xcite @xcite @xcite manage the transition from gloss to sign sequences, yet the grammar of gloss can be perplexing for those without sign language training. saunders et al. @xcite can transcribe discrete words or phrases into continuous sign language se-quences. however, directly producing continuous signs from entire spoken sentences still remains more exploration and efforts. to promote barrier-free communication between signers and speakers, we introduce a multimodal spoken data-driven continuous sign language production (ms2sl) framework (fig. 1 ). ms2sl can animate sign keypoint sequences from either speech audio or text. in addition, to alleviate data demands, we adopt an embedding-consistency learning (ecl) strategy, which is inherently based on the reciprocity among modalities, to bolster the model training. specifically, ms2sl initially employs pre-training models like clip (text) @xcite and hubert (audio) @xcite to extract features from input. subsequently, we utilize these features, serving as control conditions for the diffusion, to generate sign sequences. the attention mechanism @xcite is employed to model the relationships among conditions, denoising steps, and sign movements. besides that, ecl does not require the three modalities to coexist in the dataset. by learning a joint embedding space, inspired by imagebind @xcite , ecl tightly binds the properties of different modalities and generates feedback signals to boost the training process. first, we utilize contrastive learning to bind audio and text in the embedding space. then, we leverage the semantic consistency between co-occurring data to infer and reconstruct the embedding of missing modalitiy. the reconstruction error between the generated signs and groundtruth can be used to iteratively update ms2sl until convergence. ecl can foster cross-learning between different generation streams, allowing training even in the absence of certain modality. furthermore, the inclusion of audio data not only enriches sample diversity and enhances multimodal comprehension but also assists in accurately capturing the expression and semantic content of sign language. we validate the effectiveness of our method across two prevalent datasets how2sign @xcite and @xcite . experimental results demonstrate that ms2sl achieves sota performance, both in terms of semantic consistency and sign accuracy. in conclusion, our primary contributions are outlined as follows: ## method assuming the triplets (a, t , s) represent the audio, text, and sign space respectively, our goal is to learn the mapping from text or audio to sign within a unified framework (fig. 2 ). given a training dataset @xmath0, t, s) ∈ a ×t ×s}, ms2sl can realize text-to-sign @xmath1: @xmath2: @xmath3, where g is the sign sequence diffusion generator. we initially employ pretrained models clip @xcite and hubert @xcite to extract features from text t and audio a. next, we employ three encoders e a , e t , e s to encode these features, acquiring their embeddings e a , e t , and e s . subsequently, drawing on the operating mechanism of diffusion models, we employ a diffusion step encoder e h and a sign noise encoder e n to encode step h and noise n to e h and e n , respectively. finally, we utilize the generator g to produce the sign sequences: the paucity of co-occurring triplet data renders the direct training of ms2sl a formidable task. to overcome this challenge, we develop a joint embedding space that facilitates the natural alignment of multimodal data. furthermore, we employ ecl strategy to exploit the reciprocity among modalities within the embedding space, effectively furnishing feedback signals to boost the training. ## experiments we evaluate the effectiveness of ms2sl under textto-sign and audio-to-sign settings. ## conclusion we explore a unified framework that combines diffusion and pretrained models to generate sign language from spoken depictions. we surpass other competitors and solidify this classic framework as a highly competitive method for slp. ms2sl effectively handles diverse modalities of data for analysis and decoupling. despite its advancements, our model struggles with maintaining contextual flow in generation, and ms2sl cannot handle lengthy data, which is a future focus. our research pioneers direct sign language generation from speech, offering some insights to advance the community.