Cross-modality Data Augmentation for End-to-End Sign Language Translation (2023)

Abstract: End-to-end sign language translation (SLT) aims to directly convert sign language videos into spoken language texts without intermediate representations. It has been challenging due to the data scarcity of labeled data and the modality gap between sign videos and texts. To tackle these challenges, we propose a novel Cross-modality Data Augmentation (XmDA) framework to transfer the powerful gloss-to-text translation capabilities to end-to-end sign language translation (i.e., video-to-text). Specifically, XmDA consists of two key components: cross-modality mix-up and cross-modality knowledge distillation. The former one explicitly encourages the alignment between sign video features and gloss embeddings to bridge the modality gap. The latter one utilizes the generation knowledge from gloss-to-text teacher models to guide the spoken language text generation. Experimental results on two widely used SLT datasets, i.e., PHOENIX-2014T and CSL-Daily, demonstrate that the proposed XmDA framework significantly and consistently outperforms the baseline models. Extensive analyses confirm our claim that XmDA enhances end-to-end sign language translation by reducing the representation distance between sign videos and glosses, as well as improving the translation of low-frequency words and long sentences.

## introduction sign language is an essential communication tool used in deaf communities. sign language translation (slt) has made significant progress in recent years @xcite @xcite @xcite , with the goal of converting sign language videos into spoken language texts. the conventional approach to slt uses a cascaded system in † corresponding authors: xing wang and hui xiong. end-to-end slt models offer an intuitive solution to circumvent the challenges posed by cascaded systems. these models directly translate sign language videos into spoken texts without requiring an intermediary sign gloss representation @xcite @xcite . however, the end-to-end approach faces significant obstacles in practice due to inadequate documentation and resources for sign languages. this shortage of resources results in a high cost of annotated data @xcite , which hinders the development of end-to-end models. there have been a few recent attempts to improve the end-to-end slt performance, including back-translation @xcite and pre-training @xcite , to mitigate the issue of data scarcity. along this research line, in this work, we propose a novel cross-modality data augmentation (xmda) approach to improve the end-to-end slt performance. the main idea of xmda is to leverage the powerful gloss-to-text translation capabilities @xcite ., text-to-text) to end-to-end sign language translation (cross mode, i.e., video-totext). specifically, xmda integrates two techniques, namely cross-modality mix-up and crossmodality knowledge distillation (kd) ( § 2.2). the cross-modality mix-up technique combines sign language video features with gloss embeddings extracted from the gloss-to-text teacher model to generate mixed-modal augmented samples ( § 2.3). concurrently, the cross-modality kd utilizes diversified spoken language texts generated by the powerful gloss-to-text teacher models to soften the target labels ( § 2.4), thereby further diversifying and enhancing the augmented samples. we evaluate the effectiveness of xmda on two widely used sign language translation datasets: phoenix-2014t and csl-daily. experimental results show that xmda outperforms the compared methods in terms of bleu, rouge, and chrf scores ( § 3.2). through comprehensive analyses, we observed two key findings. firstly, crossmodality mix-up successfully bridges the modality gap between sign and spoken languages by improving the alignment between sign video features and gloss embeddings ( § 4.1). secondly, the crossmodality kd technique enhances the generation of spoken texts by improving the prediction of lowfrequency words and the generation of long sentences ( § 4.2). the contributions of our work are summarized as follows: ## methodology we integrate the proposed cross-modal data augmentation (xmda) technique into the sign language transformers, which is widely used in sign language translation tasks. the proposed framework is illustrated in figure 1 . in this section, we will first revisit the sign language transformers structure @xcite . then we provide a more comprehensive explanation of the proposed approach, including two essential components: cross-modality mixup and cross-modality kd. to ensure clarity, we define end-to-end slt and the notation used throughout this paper. ## analysis in this section, we delve into the mechanisms behind the improved end-to-end slt performance achieved by the xmda method. we investigate our approach's impact on input space and output translations, focusing on the effectiveness of the cross-modality mix-up and cross-modality kd techniques and their influence on word frequency and sentence length. ## conclusion in this paper, we propose a novel cross-modality data augmentation (xmda) approach to tackle the modality gap and data scarcity challenge in end-to-end sign language translation (slt). xmda integrates two techniques, namely cross-modality mix-up and cross-modality knowledge distillation (kd). cross-modality mix-up bridges the modality gap by producing mixed-modal input representations, and cross-modality kd augments the spoken output by transferring the generation knowledge from powerful gloss-to-text teacher models. with the xmda, we achieve significant improvements on two widely used sign language translation datasets: phoenix-2014t and csl-daily. comprehensive analyses suggest that our approach improves the alignment between sign video features and gloss embeddings and the prediction of low-frequency words and the generation of long sentences, thus outperforming the compared methods in terms of bleu, rouge and chrf scores.