Clean Text and Full-Body Transformer: M icrosoft’s Submission to the WMT 22 Shared Task on Sign Language Translation (2022)

Abstract: This paper describes Microsoft’s submission to the first shared task on sign language translation at WMT 2022, a public competition tackling sign language to spoken language translation for Swiss German sign language. The task is very challenging due to data scarcity and an unprecedented vocabulary size of more than 20k words on the target side. Moreover, the data is taken from real broadcast news, includes native signing and covers scenarios of long videos. Motivated by recent advances in action recognition, we incorporate full body information by extracting features from a pre-trained I3D model and applying a standard transformer network. The accuracy of the system is furtherimproved by applying careful data cleaning on the target text. We obtain BLEU scores of 0.6 and 0.78 on the test and dev set respectively, which is the best score among the participants of the shared task. Also in the human evaluation the submission reaches the first place. The BLEU score is further improved to 1.08 on the dev set by applying features extracted from a lip reading model.

## introduction sign languages are natural visual languages that are used by deaf and hard of hearing individuals to communicate in everyday life. sign languages are actively being researched. however, there is a huge imbalance in the field of natural language and speech processing between oral and signed languages. since recently, one observes the emergence of a transition shifting sign language processing to be part of the nlp mainstream @xcite . we embrace this development which manifests (among others) in the creation of the first shared task on sign language translation as part of wmt 2022 @xcite . it is great to have real-world sign language data @xcite as the basis of this * equal contribution shared task, manifested in native signers content and an unprecedentedly large vocabulary. nevertheless, this leads to a very challenging task with low performance numbers. when participating in the advances of sign language technologies it is worth recapping that deaf people have much at stake, both to gain and lose from applications that will be enabled here @xcite . we aim to advance the field and the use-cases in a positive way and present our findings in this system paper. in the remainder of this work we first present a brief view on the relevant literature in section 2, then we present the employed data in section 3. subsequently, we describe our submission in section 4, additional experiments in section 5 and we end with a summary in section 6. ## related work in this section, we present a limited overview of related work in sign language translation. we focus this review on the translation direction from sign language to spoken language and dismiss approaches that target the opposite direction, i.e. sign language production. sign language translation started targeting written sign language gloss to spoken language text translations, hence no videos were involved. related works were mainly based on phrase-based systems employing different sets of features @xcite @xcite . then, neural machine translation revolutionized the field. the first research publications on neural sign language translation were based on lstms either with full image input @xcite or utilized human keypoint estimation @xcite . transformer models then replaced the recurrent architectures @xcite @xcite . these models perform a lot better, but suffer from a basic drawback that the input sequences must be limited to a maximum length. previous work @xcite ) has identified the need for strong tokenizers to produce compact representations of the incoming sign language video footage. hence, a considerable body of publications target creating tokenizer models that are often trained on sign language recognition data sets @xcite @xcite or sign spotting data sets @xcite @xcite . there are several data sets relevant for sign language translation. some of the most frequently encountered are rwth-phoenix-weather 2014t @xcite and the csl @xcite (which could be also considered a recognition data set). however, there are promising new data sets appearing: openasl @xcite , sp-10 dataset @xcite ) (covers mainly isolated translations) and how2sign @xcite . ## data to train our system, we used the training data provided by the shared task organizers. the data can be considered real-life-authentic as it stems from broadcast news using two different sources: fo-cusnews and srf. focusnews, henceforth fn, is an online tv channel covering deaf signers with videos of 5 minutes having variable sampling rates of either 25, 30 or 50 fps. srf represents public swiss tv with contents from daily news and weather forecast which are being interpreted by hearing interpreters. the videos are recorded with a sampling rate of 25 fps. all data, therefore, covers swiss german sign language (dsgs). our feature extractors are pretrained on bsl-1k (albanie et al., 2020) and av-hubert @xcite . additionally, we evaluate the effect of introducing a public sign language lexicon that covers isolated signs 1 , which we refer to as lex. it provides main hand shape annotations, one or multiple (mostly one) examples of the sign and an example of how this sign is used in a continuous sentence. we choose a subset that overlaps in vocabulary with either focusnews or srf. as part of the competition, independent dev and test sets are provided, which consist of 420 and 488 utterances respectively. ## submitted system sign languages convey information through the use of manual parameters (hand shape, orientation, location and movement) and non-manual parameters (lips, eyes, head, upper body). to capture most information from the signs, we opt for an rgb-based approach, neglecting the tracked skeleton features by the shared task organizers. for the submitted system we rely on a pre-trained tokenizer for feature extraction and train a sequence-to-sequence model to produce sequences of whole words (no byte pair encoding). we further pre-process the sentences (ground truths of the videos) to clean it. this step is crucial to push the model to focus more on semantics of the data. finally, in order to adhere to the expected output format for the submission, we convert the text back to display format using microsoft's speech service. this applies inverse text normalization, capitalization and punctuation to the output text to make it more readable. the details of various components of the system are described in the next subsections. ## additional experiments we perform additional experiments to assess the impact of having dedicated mouth features as well as a lexicon data set on the model performance. the mouth carries important semantic information in sign languages. in the literature, exploiting lip information has shown to be fundamental for increased performance in sign language recognition and translation @xcite . ## summary in this paper, full body information is applied successfully for a challenging sign language task as part of the wmt 2022 competition. as such, we employed a pre-trained i3d model to extract an embedding for a sequence of frames of the video. the features are further fed as input to a standard transformer network. we obtain reasonable performance of 0.4 in terms of bleu score on the test set. the model is further enhanced by applying careful cleaning to the text output. we obtain the result of 0.6 bleu score on the official test data. based on the automatic bleu scoring this result was the best among the 7 participants of the shared task, but also in the human evaluation our submission reaches the first place. with additional experiments, we validate the usefulness of a pre-trained lip reading model for this task and the addition of a lexical data set. this improves the results to 1.08 reduced bleu on the dev set.