K now C omp Submission for WMT 23 Sign Language Translation Task (2023)

Abstract: Sign Language Translation (SLT) is a complex task that involves accurately interpreting sign language gestures and translating them into spoken or written language and vice versa. Its primary objective is to facilitate communication between individuals with hearing difficulties using deep learning systems. Existing approaches leverage gloss annotations of sign language gestures to assist the model in capturing the movement and differentiating various gestures. However, constructing a large-scale gloss-annotated dataset is both expensive and impractical to cover multiple languages, and pre-trained generative models cannot be efficiently used due to the lack of textual source context in SLT. To address these challenges, we propose a gloss-free framework for the WMT23 SLT task. Our system primarily consists of a visual extractor for extracting video embeddings and a generator responsible for producing the translated text. We also employ an embedding alignment block that is trained to align the embedding space of the visual extractor with that of the generator. Despite undergoing extensive training and validation, our system consistently falls short of meeting the baseline performance. Further analysis shows that our model’s poor projection rate prevents it from learning diverse visual embeddings. Our codes and model checkpoints are available at https://github.com/HKUST-KnowComp/SLT.

## introduction machine translation has significantly improved thanks to the development of pre-trained language models @xcite . while translation within a single modality has been extensively studied, translation involving multiple modalities remains challenging and less explored @xcite . sign language translation (slt), which translates sign gestures into spoken language, remains an exceedingly complex task due to two fundamental challenges. firstly, sign languages are visual-gestural languages that rely on manual signs, facial expressions, and body movements to convey information. this fundamental distinction sets them apart from written languages, which consist of word characters and symbols. consequently, translation models must be able to accurately interpret visual signals and gestures and develop a deep understanding of the semantics involved in producing prompt translations. however, the multimodal nature of sign languages poses a significant challenge for models, requiring them to learn and generalize these complex interactions effectively. moreover, sign languages are typically represented as exceedingly lengthy sequences of frames, surpassing the number of tokens in a standard sentence @xcite . this requires translation models to grasp the prolonged dependencies within the video to accurately capture the information conveyed through these visual signals. to tackle these challenges, methods have been proposed that utilize pre-training a visual backbone based on gloss annotations @xcite . these approaches have demonstrated exceptional performance in various multimodal translation tasks. nevertheless, the acquisition of extensive gloss annotations comes with significant cost and practical constraints, making it impractical to cover a wide range of multilingual translation directions @xcite . in this paper, we propose a gloss-free framework for the slt task. our approach combines a pretrained visual backbone model @xcite , which has been trained to recognize sign gestures, with a gpt2-based language model @xcite to generate the translated sentence. to align the embedding space between both models, we utilize an embedding alignment block inspired by clipcap @xcite . the final translation is produced using converted visual embeddings and text embeddings (section 3). despite pretrained i3d feature extractor embedding alignment block # of prefix pretrained german gpt2 schweizer unternehmen und die folgender steuerreform von präsident trump. (swiss companies and the following tax reform by president trump.) 𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸 𝐹𝐹 𝑝𝑝 1 , 𝐹𝐹𝑝𝑝 2 … , 𝐹𝐹𝑝𝑝𝐹𝐹 = 𝑉𝑉 𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐹𝐹𝐴𝐴𝐴𝐴𝐹𝐹𝐸𝐸 𝑉𝑉 = {𝑇𝑇 1 , 𝑇𝑇 2 … , 𝑇𝑇𝐴𝐴} 𝐺𝐺𝐴𝐴𝐹𝐹𝐴𝐴𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸𝐸({𝑇𝑇 1 , 𝑇𝑇 2 … , 𝑇𝑇𝐴𝐴}) = {𝑆𝑆 1 , 𝑆𝑆 2 … , 𝑆𝑆𝑆𝑆} {𝑆𝑆 1 , 𝑆𝑆 2 … , 𝑆𝑆𝑆𝑆} figure 1: an overview of our framework. we first downsample video data and feed them into the visual feature extractor to obtain the visual embeddings. the embeddings are then passed into the alignment block to project them into embedding inputs of the german-gpt2. they are used as the prefix of the gpt2 model to generate the final translation results. ## method this section introduces our proposed framework, which is depicted in figure 1 . while previous systems @xcite @xcite primarily employ an encoder-decoder paradigm and train their models from scratch to address this task, we distinguish ourselves by being the first to utilize a pre-trained language model for this task, as these language models possess strong natural language understanding and generation ability @xcite @xcite @xcite . specifically, we leverage the pre-trained i3d model provided by @xcite as our visual extractor backbone and employ a german-gpt2 model @xcite as the generator's backbone. ## conclusions in conclusion, this paper presents the knowcomp system for the wmt23-slt sign language translation shared task. our system utilizes two pretrained backbone models for visual feature extraction and translation text generation. however, this architecture fails, resulting in unsatisfactory perfor-mance across all evaluation datasets. our system's performance is significantly below the baseline's performance. we have identified a critical weakness in our model through further analysis, including embedding t-sne plots and case studies. the embedding alignment block unexpectedly densifies all visual embeddings together, leading to the generator generating repeated tokens. to enhance our model's performance in future work, an appropriate data augmentation technique @xcite can be implemented to help the alignment block distinguish different input features more efficiently. also, future works can focus on whether further increasing the model capacity could help to mitigate the issue shown in the analysis section considering the advancing computation resources.