TTIC ’s Submission to WMT - SLT 23 (2023)

Abstract: In this paper, we describe TTIC’s submission to WMT 2023 Sign Language Translation task on the Swiss-German Sign Language (DSGS) to German track. Our approach explores the advantages of using large-scale self-supervised pre-training in the task of sign language translation, over more traditional approaches that rely heavily on supervision, along with costly labels such as gloss annotations. The proposed model consists of a VideoSwin transformer for image encoding, and a T5 model adapted to receive VideoSwin features as input instead of text. In WMT-SLT 22’s development set, this system achieves 2.03 BLEU score, a 59% increase over the previous best reported performance. In the official test set, our primary submission achieves 1.1 BLEU score and 17.0 chrF score.

## introduction sign language translation (slt) is the task of translating a signed language to a written language, typically the lingua franca of the region the signed language is utilized. in recent years, slt has received increased attention from the natural language processing (nlp) and computer vision (cv) communities. the best-performing slt models primarily rely on glosses @xcite , a combination of morpheme translations into the target language along with differentiating phonological features like handshape and location. however, annotating glosses is expensive @xcite , and recent research has begun to move away from gloss-based translation @xcite @xcite , particularly in regimes where larger datasets are available. in this paper, we study large-scale selfsupervision and noisy supervision for swiss-german sign language (dsgs from the german deutschschweizer gebärdensprache) to ger-man slt, as part of the wmt-slt 23 shared task @xcite . given recent findings on self-supervised transformers' perfomance on isolated sign recognition and feature extraction @xcite , we utilize a videoswin @xcite visual feature extractor with bevt pre-training @xcite . additionally, we use t5 @xcite as a sequence-to-sequence translation model into german because of its state-of-the-art performance on american sign language (asl) to english slt with pose input @xcite . depending on the generation algorithm, our model achieves either the highest bleu score @xcite or the highest chrf @xcite in the task's leaderboard. with top-k beam sampling, it achieves 0.8 bleu and 17.3 chrf, and with diverse beam search @xcite , it achieves 1.1 bleu and 17.0 chrf. ## method our model follows the most common gloss-free translation architecture, composed of a visual encoding backbone and a transformer-based model for sequence modeling. our visual backbone is a video swin transformer (videoswin) and our sequence-to-sequence model is a text-to-text transfer transformer (t5). ## experimental results we additionally perform several ablations to quantify the impact of our model's several moving parts. our ablations are performed using t5's efficient-base configuration with 619 million parameters for time efficiency, unless otherwise specified. ## conclusion our experiments evaluate a hierarchical vision transformer on the task of sign language translation for the first time, and demonstrate superior performance over i3d-based translation models. we also show the benefits of using large datasets and self-supervised models for sign language translation, outperforming all previous fully supervised approaches to this task. our final model achieves highest bleu-4 score, highest chrf score, and highest human evaluation score among all participants of the task. however, translation quality remains extremely low.