Sign Language Translation with Sentence Embedding Supervision (2024)

Abstract: State-of-the-art sign language translation (SLT) systems facilitate the learning process through gloss annotations, either in an end2end manner or by involving an intermediate step. Unfortunately, gloss labelled sign language data is usually not available at scale and, when available, gloss annotations widely differ from dataset to dataset. We present a novel approach using sentence embeddings of the target sentences at training time that take the role of glosses. The new kind of supervision does not need any manual annotation but it is learned on raw textual data. As our approach easily facilitates multilinguality, we evaluate it on datasets covering German (PHOENIX-2014T) and American (How2Sign) sign languages and experiment with mono- and multilingual sentence embeddings and translation systems. Our approach significantly outperforms other gloss-free approaches, setting the new state-of-the-art for data sets where glosses are not available and when no additional SLT datasets are used for pretraining, diminishing the gap between gloss-free and gloss-dependent systems.

## introduction sign language translation (slt) aims at generating text from sign language videos. there are several approaches to slt reported in the literature, with sign2text and sign2gloss2text the most widely used. while sign2text directly translates video into text with or without the help of glosses @xcite , sign2gloss2text passes through an intermediate gloss step before translation into spoken language text @xcite . that is, sign2gloss2text breaks down the problem into two independent sub-problems using glosses as a pivot language. a gloss is a textual label associated with a sign, and, although human signers do not in general use them, performance in automatic slt has long been upper bounded by the gloss supervision and their use as an intermediate representation @xcite . the advantage of translation without glosses is that collecting data is much easier. even though translation results are better for approaches that use glosses as intermediate representation @xcite , this comes at the cost of annotating all the video data with glosses which is a time consuming manual task. for many data sets glosses are simply not available. on the plus side, with gloss supervision-based slt architectures, one can take full advantage of the maturity of text2text machine translation between glosses and spoken language text. in this work, we present a novel approach sign2(sem+text), a model that gets rid of glosses and adds supervision through sentence embeddings, sem, pretrained on raw text and finetuned for sign language. our experimental results demonstrate the strength of the novel approach on both standard small datasets with gloss annotation and larger datasets without. in the latter case, we achieve state-of-the-art results for the american sign language (asl) dataset how2sign when no additional slt datasets are used foot_0 improving over @xcite by 4 bleu points. for german sign language (dgs), our new approach achieves translation quality scores between the previous best glossfree system @xcite and the current state-of-the-art using glosses @xcite on the phoenix-2014t dataset. our code and models are publicly available. foot_1 2 related work @xcite proposed three formalisations considering slt as a seq2seq problem that converts a sequence of signs into a sequence of words: (i) sign2text, a model that encodes video frames using pretrained 2d cnns as spatial features and then uses an rnn to generate the text; (ii) gloss2text, a model learning the translation between a sequence of textual glosses and fluent spoken language text; and (iii) a sign2gloss2text model that adds an extra intermediate gloss layer between the video and output text levels of a sign2text architecture to provide additional gloss supervision using a ctc loss. in follow-up work, camgöz et al. (2020) proposed an architecture for joint learning continuous sign language recognition (cslr) and slt which uses the same input as camgöz et al. (2018); @xcite , that is, pretrained visual features, but a transformer @xcite ) for text generation. camgöz et al. (2020) conjectured that gloss2text results with ground-truth glosses provide an upper bound for slt. supporting this assumption, their translation quality on phoenix-2014t as measured by bleu @xcite achieved 24.5 on gloss2text and 21.8 on sign2text. yin and read (2020) used a different visual representation with a multi-cue network @xcite to encode videos. cues included face, hands and pose besides the full frame. with a bleu score of 24.0 they impoved over sign2gloss2text @xcite and concluded that their visual representation was better than the spatial frame embeddings used by the camgöz et al. (2020). chen et al. (2022a,b) used both pretraining of a network based on s3ds @xcite on action recognition for cslr (sign2gloss) and pretraining of a textual transformer (gloss2text) with mbart-25 @xcite . both types of pretraining are progressively adapted to the domain of the task by adding data closer to the domain. an additional mapping network between the vision and language parts allows @xcite to build an end2end sign2text model relying on inter-nal gloss supervision. to the best of our knowledge, @xcite is the current state of the art for both sign2text (@xmath0.95) and sign2gloss2text (@xmath1.71), all on the phoenix-2014t data set. over the last few years, several gloss-free models have emerged @xcite @xcite . @xcite obtains the current state-of-the-art in this category by utilising visual-language pretraining following clip @xcite . on the datasets @xcite ; @xcite ) where the two approaches can be compared, translation quality diminishes by up to 7 bleu points when the glosses are not used @xcite . ## sem-based architectures in our work we build two systems that revolve around textual sentence embeddings, sem, as depicted in figure 1 . the figure presents two independent modules sign2sem and sem2text (left plots) that we later combine in sign2sem2text and sign2(sem+text) in an end2end setting (right plot). pretraining the visual feature sentence embedding model on text. we follow @xcite and train a siamese network with twin subnetworks 1 and 2. we compute the loss as the minimum squared error (mse): pretraining we train both transformers (sltr from scratch and the already pretrained mbart-25) with wikipedia data and then finetune them on the sl datasets. we compute the translation output loss as the cross-entropy: after pretraining each component (sign2sem and sem2text), we combine them together for end2end training. we explore two approaches: an approach that only uses the output loss l o , sign2sem2text, and an approach that integrates an additional supervision loss l e , sign2(sem+text). for sltr, we take the sem before the tanh and pooling (see figure 1 (left-middle)), and project it into the sltr model dimension. the supervision is applied after the sltr encoder. for mbart, the supervision happens right before the mbart. our architectures can be trained both monolingually and mutilingually simply by using multilingual embeddings and merging multilingual training data. ## experimental settings we use two diverse (language and domain) datasets for our experiments: rwth-phoenix- @xcite 11 hours of weather forecast videos from 9 signers. signers use german sign language and both transcriptions and glosses are available. how2sign @xcite 80 hours of instructional videos with speech and transcriptions and their corresponding american sign language videos (glosses unavailable) from 11 signers. detailed statistics for each dataset are provided in appendix a. we preprocess the textual part of the datasets in a way that allows us to compare to the results obtained by @xcite . we tokenise and lowercase the input for both training and evaluation. we apply bpe @xcite with a vocabulary size of 1500 for phoenix-2014t and 5000 for how2sign. when pretraining sem2text sltr, we use a shared (en-de) vocabulary size of 32000. in cases where we use pretrained models, we keep the tokenisation of the model. for video files, we extract frames using ffmpeg. we normalise the images, and resize them to 224x224. in this step, we initially obtain frame features from a pretrained model @xcite , which does not contain gloss information. we then apply pooling to remove the spatial dimensions, followed by batch normalisation with relu, following the approach outlined by @xcite . this generic approach facilitates the combination of datasets in the multilingual setting. we use two multilingual pretrained models that cover both german and english, sbert (reimers and gurevych, 2019) foot_2 for sentence embeddings and mbart @xcite foot_3 as a language model. for further pretraining we use 26 million sentences per language from the english and german wikipedia dumps extracted with wikitailor @xcite . following @xcite and @xcite , we evaluate the models using three common automatic metrics in machine translation: bleu @xcite , chrf @xcite and bleurt @xcite . specifics can be found in appendix c. in all cases, we estimate 95% confidence intervals (ci) via bootstrap resampling @xcite with 1000 samples. ## results and discussion potentially beneficial effects of multilinguality are less evident. monolingual and multilingual approaches are not distinguishable within the 95% cis, possibly due to large differences in the domain of the datasets preventing effective transfer between languages. our best system, sign2(sem+text) with the pretrained text decoder, achieves state-of-the-art results on how2sign when no additional slt dataset is used for pretraining, improving from 8 to 12 bleu points over @xcite . for phoenix-2014t, we surpass all previous glossfree approaches (24 vs 21 bleu), but we are still below the best approach that uses glosses @xcite (24 vs 29 bleu). reconstruction quality: sem2text. in our approach, sentence embeddings take the role of manually produced glosses in previous work. our sem2text translation module defines the upper-bound results for the full system as gloss2text did in previous work. our best sign2(sem+text) models with mbart produce a reconstruction score of bleu 38.0±2.4/23.3±1.0, chrf 57.5±1.9/43.9±0.9 and bleurt 0.588±0.019/0.571±0.008 for phoenix-2014t/how2sign (see ## conclusions we present a new approach to sign language translation using automatically computed sentence embeddings instead of manual gloss labels as intermediate representation with (sign2(sem+text)) and without (sign2sem2text) sem supervision. we outperform the state-of-the-art of gloss-free slt when no additional slt datasets are used for pretraining, closing the gap to gloss-based slt. according to the upper-bound set by sem2text translation quality, there is still room for improvement for the end2end sem-based slt models. in this work, we limited ourselves to existing visual feature extractors, in the future we plan to train a sem-based visual feature extractor on sl datasets in order to get closer to our sem2text upper-bound and match gloss-based performance.