Signer Diversity-driven Data Augmentation for Signer-Independent Sign Language Translation (2024)

Abstract: The primary objective of sign language translation (SLT) is to transform sign language videos into natural sentences.A crucial challenge in this field is developing signer-independent SLT systems which requires models to generalize effectively to signers not encountered during training.This challenge is exacerbated by the limited diversity of signers in existing SLT datasets, which often results in suboptimal generalization capabilities of current models.Achieving robustness to unseen signers is essential for signer-independent SLT.However, most existing method relies on signer identity labels, which is often impractical and costly in real-world applications.To address this issue, we propose the Signer Diversity-driven Data Augmentation (SDDA) method that can achieve good generalization without relying on signer identity labels. SDDA comprises two data augmentation schemes. The first is data augmentation based on adversarial training, which aims to utilize the gradients of the model to generate adversarial examples. The second is data augmentation based on diffusion model, which focuses on using the advanced diffusion-based text guided image editing method to modify the appearances of the signer in images. The combination of the two strategies significantly enriches the diversity of signers in the training process.Moreover, we introduce a consistency loss and a discrimination loss to enhance the learning of signer-independent features.Our experimental results demonstrate our model significantly enhances the performance of SLT in the signer-independent setting, achieving state-of-the-art results without relying on signer identity labels.

## introduction sign languages are an indispensable communication medium for individuals who are deaf or * equal contribution † corresponding author hearing-impaired, utilizing the combination of handshapes, facial expressions, and body movements to convey information @xcite . converting sign language into spoken language sentences, known as sign language translation (slt), is an essential bridge that connects the deaf community with the hearing world @xcite , thus receiving increasing attention and leading to significant advancements by the research community in recent years @xcite @xcite @xcite . to this end, we propose the signer diversitydriven data augmentation (sdda) method to improve the generalization of the model to unseen signers without relying on signer identity labels. sdda consists of two main components: data augmentation based on adversarial training (daat) and data augmentation based on diffusion model (dadm). firstly, daat utilizes the gradients of the model to generate adversarial examples to enhance the robustness of the model to changes in the signers. different from vanilla adversarial training methods @xcite ) that perturb the whole image indiscriminately, we only perturb the sign language non-critical regions of the image by introducing a keypoint masking. since gestures and expressions contain rich information of sign language, we regard these parts as critical regions and other parts as non-critical regions. in this way, we can improve the model's robustness to signers without losing the semantics of the sign language. secondly, motivated by the recent advancements in diffusion-based image generation models, dadm applies a variety of elaborate prompts to guide the diffusion-based text-guided image editing model @xcite in modifying the appearance of the signer for each video frame, thereby significantly increasing the diversity of signers. when combined, daat and dadm provide comprehensive augmentation for signers. daat ensures that the model is robust to small changes and noise occurring in the signer, while dadm significantly increases the diversity of signer appearances that the model is exposed to during training. to effectively learn signer-independent representations, we introduce a consistency loss and a discrimination loss into our model training. the former minimizes the kl divergence between the output distributions of the original and augmented samples to ensure that the augmented data does not deviate semantically from the original data. the latter trains a discriminator to distinguish the features from the original and augmented samples, ensuring that the model's feature extraction is robust to variations in signer appearance. we conduct a variety of experiments on the phoenix- @xcite benchmark to verify the effectiveness of our model. experimental results indicate that sdda effectively enhances the performance of slt in the signerindependent setting without relying on signer identity labels, achieving state-of-the-art results. ## approach in the signer-independent setting, the slt model is trained on the signer group g and subsequently tested on another signer group g ′ , where g∩g ′ = ∅. however, the limited availability of signers in current slt datasets restricts the ability of the slt model to generalize effectively to unseen signers. ## conclusion in this work, we propose sdda, a signer diversitydriven augmentation for signer-independent slt. sdda comprises two data augmentation methods. the first is data augmentation based on adversarial training, which focuses on using the gradient of the model to generate adversarial samples. the second is data augmentation based on the diffusion model, which focuses on using the advanced diffusion based text guide image editing method to edit the signers in the picture, alleviating the problem of scarcity of signer diversity. by employing the two data augmentation methods, each frame in the sign language video can be transformed into a signer-diverse image. to learn signer-independent features, we introduce a consistency loss and a discrimination lass to align the features of the original and augmented videos. through our method, the model learns more robust visual features and consistent global contextual representations for different signers, thus improving the generalization ability of the model to unseen signers. experimental results on the benchmark phoenix-2014t affirm the effectiveness of sdda.