G loss2 T ext: Sign Language Gloss translation using LLM s and Semantically Aware Label Smoothing (2024)

Abstract: Sign language translation from video to spoken text presents unique challenges owing to the distinct grammar, expression nuances, and high variation of visual appearance across different speakers and contexts. Gloss annotations serve as an intermediary to guide the translation process. In our work, we focus on Gloss2Text translation stage and propose several advances by leveraging pre-trained large language models (LLMs), data augmentation, and novel label-smoothing loss function exploiting gloss translation ambiguities improving significantly the performance of state-of-the-art approaches. Through extensive experiments and ablation studies on the PHOENIX Weather 2014T dataset, our approach surpasses state-of-the-art performance in Gloss2Text translation, indicating its efficacy in addressing sign language translation and suggesting promising avenues for future research and development.

## introduction sign language translation from video to spoken text often involves two phases: sign2gloss and gloss2text. in sign2gloss phase, the gloss annotations, are predicted from input videos as shown in the top part of figure 1 , establishing a link between visual expressions and corresponding meanings. the subsequent gloss2text phase, translates these gloss annotations into spoken language. while gloss annotations have strong limitations as linguistic representations @xcite , the emergence of pre-trained large language models, word embeddings, and advances in machine translation open up new possibilities for improvements in gloss2text translation task. in our work, we propose to leverage large language models (llms) pre-trained on expansive and diverse corpora along with novel sign language specific label smoothing figure 1 : an example of ambiguity in sign language is demonstrated by the gloss "bewoelkt (cloudy)," which is represented in multiple translations within the dataset. as shown, ambiguity may share the same meaning but differ in form, such as "wolken (cloudy)," or where the gloss represents the concept meaning, such as "unbeständig (unstable)." loss and data augmentation techniques to improve gloss2text phase of sign language translation task. our contributions are the following: ## approach the goal of our gloss translation system is to convert a series of gloss annotations @xmath0, g 2 , . . . , g t into a spoken word sequence @xmath1, t 2 , . . . , t l . given n pairs where each pair can have different input and output lengths. our approach involves fine-tuning large language models tailored specifically for our task. to effectively train a typical neural machine translation (nmt) model, a corpus of around 1 million parallel samples is often required @xcite . however, the existing sign language datasets are of several orders of magnitude smaller. for instance, the phoenix-2014t german sign language dataset camgoz et al. ( 2018 2020 ), mbart @xcite can be found in appendix b. these models share multilingual characteristics, enabling them to handle diverse language pairs ef-ficiently. however, they differ in the datasets they are pre-trained on, the specific architectures they employ, and their respective training objectives, which affect their performance in sign language translation tasks. ## conclusion we conducted a comprehensive exploration of gloss2txt translation for sign language using large language models and the phoenix-2014t dataset. we evaluated different model architectures, data augmentations, and loss functions. our experiments showed that our semantically aware label smoothing technique significantly improves translation quality over state-of-the-art models.