Unsupervised Sign Language Translation and Generation (2024)

Abstract: Motivated by the success of unsupervised neural machine translation (UNMT), we introduce an unsupervised sign language translation and generation network (USLNet), which learns from abundant single-modality (text and video) data without parallel sign language data. USLNet comprises two main components: single-modality reconstruction modules (text and video) that rebuild the input from its noisy version in the same modality and cross-modality back-translation modules (text-video-text and video-text-video) that reconstruct the input from its noisy version in the different modality using back-translation procedure. Unlike the single-modality back-translation procedure in text-based UNMT, USLNet faces the cross-modality discrepancy in feature representation, in which the length and the feature dimension mismatch between text and video sequences. We propose a sliding window method to address the issues of aligning variable-length text with video sequences. To our knowledge, USLNet is the first unsupervised sign language translation and generation model capable of generating both natural language text and sign language video in a unified manner. Experimental results on the BBC-Oxford Sign Language dataset and Open-Domain American Sign Language dataset reveal that USLNet achieves competitive results compared to supervised baseline models, indicating its effectiveness in sign language translation and generation.

## introduction sign language translation and generation (sltg) have emerged as essential tasks in facilitating communication between the deaf and hearing communities @xcite . sign language translation involves the conversion of sign language videos into natural language, while sign language generation involves the generation of sign language videos from natural language. sign language translation and generation have achieved great progress in recent years. however, training an sltg model requires a large parallel video-text corpus, which is known to be ineffective when the training data is insufficient @xcite . furthermore, manual and professional sign language annotations are expensive and timeconsuming. inspired by the successes of unsupervised machine translation (unmt) @xcite and unsupervised image-toimage translation @xcite , we propose an unsupervised sltg network (uslnet) that does not rely on any parallel video-text corpus. similar to unmt, uslnet consists of the following components: the text reconstruction module ( §2.1) and the sign video reconstruction module ( §2.2) that rebuild the input from its noisy version in the same modality, and cross-modality back-translation modules ( §2.3) that reconstruct the input from its noisy version in the different modality using a backtranslation procedure. unlike the single-modal back-translation in textbased unmt, uslnet faces the challenge of crossmodal discrepancy. sign and spoken languages exhibit distinct characteristics in terms of modality, structure, and expression. sign language relies on visual gestures, facial expressions, and body movements to convey meaning, while spoken language depends on sequences of phonemes, words, and grammar rules @xcite . the cross-modal discrepancy in feature representation presents unique challenges for uslnet. to address the cross-modal discrepancy in feature representation, a common practice is to use a linear projection to map the representations from the single-modal representation to a shared multimodal embedding space @xcite . in this work, we propose a sliding window method to address the issues of aligning the text with video sequences. to the best of our knowledge, uslnet is the first unsupervised sltg model capable of generating both text and sign language video in a unified manner. experimental results on the bbc-oxford sign language dataset (bobsl) @xcite and open-domain american sign language dataset (openasl) @xcite reveal that uslnet achieves competitive results compared to the supervised baseline model @xcite indicating its effectiveness in sign language translation and generation. our contributions are summarized below: ## methodology the proposed framework in this study consists of four primary components: a text encoder, a text decoder, a video encoder, and a video decoder. as illustrated in figure 2 , the uslnet framework encompasses four procedures: a text reconstruction procedure (gray line), a sign video reconstruction procedure (blue line), a text-video-text back-translation (t2v2t-bt) procedure which initially translates input text into pseudo video (red line) and subsequently backtranslates pseudo video into text (yellow line), and a video-text-video back-translation (v2t2v-bt) procedure which firstly translates input video into pseudo text (yellow line) and then back-translates pseudo text into video (red line). the latter two modules are considered cross-modality backtranslation modules due to their utilization of the back-translation procedure. in this section, we will first describe each module and then introduce the training procedure. ## conclusion in this paper, we present an unsupervised sign language translation and generation network, uslnet. it is the first bi-directional (translation/generation) sign language approach trained in unsupervised manner. experimental results on the large-scale sign dataset such as bobsl and openasl reveal that uslnet achieves competitive performance compared to the supervised approach.