Word-Conditioned 3 D A merican S ign L anguage Motion Generation (2024)

Abstract: Sign words are the building blocks of any sign language. In this work, we present wSignGen, a word-conditioned 3D American Sign Language (ASL) generation model dedicated to synthesizing realistic and grammatically accurate motion sequences for sign words. Our approach leverages a transformer-based diffusion model, trained on a curated dataset of 3D motion meshes from word-level ASL videos. By integrating CLIP, wSignGen offers two advantages: image-based generation, which is particularly useful for children learning sign language but not yet able to read, and the ability to generalize to unseen synonyms. Experiments demonstrate that wSignGen significantly outperforms the baseline model in the task of sign word generation. Moreover, human evaluation experiments show that wSignGen can generate high-quality, grammatically correct ASL signs effectively conveyed through 3D avatars.

## introduction sign languages are natural languages with their own grammatical structures and lexicons, primarily used by deaf and hard-of-hearing (dhh) communities @xcite . while most current sign language research focuses on sign-to-text translation @xcite @xcite , we address the reverse problem with wsigngen 1 -a model dedicated to sign language synthesis and production, converting text into corresponding signing motions. many sign synthesis works in the literature (more on the sign synthesis approaches are discussed in section 2) aim to directly learn the mappings between continuous visual signs and written language, via neural machine translation (nmt). but we propose a different approach where the 1 the code and data are available on the project page. nmt would occur between the textual words and the corresponding asl gloss foot_0 . the resulting gloss sentences can then readily be furnished with asl word-sign motions, the focus of this work. current continuous sign language production methods only perform close-set generation and cannot handle words that were not previously seen; whereas, we empirically observe that new asl signers can construct a large number signed phrases based only on a fixed number of signs, further motivating the importance of word sign synthesis. but word sign synthesis can be a highly challenging task as asl consists of manual (hands-only) and nonmanual (non-hands, such as mouthing cues, facial expressions, etc) cues. in this work, we focus on the manual cues which are expressed with four grammatical parameters: handshape, palm orientation, location, and movement @xcite . also, different signers may perform the same signs with different movement amplitudes and forces, and these variations should be considered in a signing avatar. clip @xcite pre-trains an image encoder and a text encoder to predict which images correspond to specific text pairs in the dataset. instead of using one-hot encoding for the sign word, we leverage the clip text encoder as a conditional input. this approach offers two key advantages: (1) image-based generation, where the image embedding connects to the most semantically similar word, which can be particularly useful for children who cannot read yet but are learning sign language; and (2) generalization to unseen synonyms. since many words with similar meanings (e.g., 'table' in the dataset and 'desk' not in the dataset) share the same sign motion, clip's semantically aligned text embeddings are well-suited to handle such synonyms. figure 1 : top row: given a text word in dataset, wsigngen can generate a 3d signing avatar corresponding to that word. also, wsigngen supports unseen but semantically close words. bottom row: given a semantically meaningful image, the closest associated word is identified via clip and wsigngen generates the corresponding sign. our contributions are threefold: first, we introduce a novel approach for word-conditioned 3d asl motion generation, synthesizing more accurate, realistic, and expressive sign language motions. second, we present wsigngen, a model that combines clip with a diffusion-based approach using a classifier-free strategy, offering two advantages: image-based generation and the ability to generalize to unseen synonyms. third, extensive experiments show that wsigngen consistently outperforms baseline models on standard generative metrics and receives highly positive feedback from human evaluators. ## related work sign language synthesis sign language synthesis (/production) aims to convert text input into grammatically correct sign sequence motions. researchers have explored various seq2seq models @xcite @xcite @xcite , and multi-channel approaches @xcite , to synthesize pixel-level videos @xcite or produce sign language skeletal sequences @xcite . other works have relied solely on the regression of 2d/3d joint positions @xcite with openpose @xcite or post-regression smplx @xcite fitting to elevate 2d joints to 3d features @xcite . reconstructing 3d avatars @xcite @xcite further advances recent studies. we go a step further by utilizing a diffusion model to directly synthesize 3d features, thereby preserving lifelike diversity and authenticity in our avatars. ## experiments dataset we curated a 3d smplx-based dataset from the sign recognition video dataset, wlasl @xcite . we sorted the dataset by sample size and then selected the top 103 words we further selected a subset of 30 words denoted as asl3d s for our scalability evaluation; next, we used the hand4whole model @xcite to extract smplx features, creating our smplxbased asl3d dataset. ## conclusion we introduce the wsigngen model, representing a significant milestone in the realm of wordconditioned 3d sign language motion generation. by leveraging the clip model to align word semantics and a diffusion model to learn clean signing motions, wsigngen demonstrates notable advancements over existing methodologies. extensive experiments showcase our work has excellent scalability and can generalize to unseen words.