Spatio-temporal Sign Language Representation and Translation (2022)

Abstract: This paper describes the DFKI-MLT submission to the WMT-SLT 2022 sign language translation (SLT) task from Swiss German Sign Language (video) into German (text). State-of-the-art techniques for SLT use a generic seq2seq architecture with customized input embeddings. Instead of word embeddings as used in textual machine translation, SLT systems use features extracted from video frames. Standard approaches often do not benefit from temporal features. In our participation, we present a system that learns spatio-temporal feature representations and translation in a single model, resulting in a real end-to-end architecture expected to better generalize to new data sets. Our best system achieved 5 \pm 1 BLEU points on the development set, but the performance on the test dropped to 0.11 \pm 0.06 BLEU points.

## introduction text-to-text machine translation (mt) is achieving a great success with even (close to) human performance for some language pairs and domains @xcite . however, the situation in sign language translation (slt) is much different. one important reason is that the slt is a low-resource scenario where one does not have the same amount of data as in high-resourced textto-text to achieve a similar level of performance. a more specific reason is that slt involves two modalities, text and video. various problems arise when dealing with these modalities. besides data scarcity, the lack of temporal boundaries in the input videos is a challenge. to overcome the lack of temporal boundaries, the most common solution tends to ignore or not benefit from temporal features. this approach relies on the transformer @xcite capabilities to learn sequence-to-sequence tasks. the state-of-the-art slt technique @xcite is practically a normal transformer but uses a custom embedding layer for 2d features extracted from video frames. in this approach, training a slt system requires a pre-extraction step to convert frame features to vectors and train a transformer separately to translate the vectors into spoken language. this type of approach has been widely used on a very specific dataset, the weather forecast corpus phoenix14t @xcite , where researchers reported a relatively good performance in terms of bleu (∼20) @xcite . despite its good performance on a specific dataset, there is the doubt whether such type of architecture generalizes to new data sets. in order to build a more general technique, we focus on fundamental slt problems such as the design, implementation and evaluation of a fully end-toend model and representation learning for sign language videos. having a fully end-to-end model facilitates the task of data collection and diminishes the need for annotation (e.g. in terms of sign language glosses), which is necessary to build larger and richer datasets. it also allows training video embeddings fully optimized for the translation task. text translation is one of the most mature areas in natural language processing, and therefore we focus here on the sign language representation part of the architecture and use an in-house state-of-the-art transformer for text generation. this paper reports our approach for end-to-end slt used for the wmt-slt translation shared task from swiss german sign language into german. in the next sections we introduce our approach (section 2), experimental setup (section 3), results (section 4) and conclusions & perspectives (section 5). ## our approach the main idea of our approach is to learn feature representation and translation in a single model, and be able to train them together. figure 1(a) sketches our general pre-processing pipeline and both the cnns used for the video representations and the transformers used for the text representations come with large numbers of parameters. as we are operating in a low resource scenario, besides the combination of the two networks, we experiment to find the best trade-off between data size and number of parameters. ## results and analysis our best model according to bleu is obtained with the largest 3d resnet model and reaches 4.8 points on the development set, much higher than the performance of any system on the official test set. however, different metrics do not correlate, and chrf2++ and bleurt -which correlate better with human judgments than bleu-point towards a different model. hypothesis reference die -. die diamantenschleiferei beschäftigt 63 angestellte , davon 17 behinderte , sowohl rollstuhlfahrer als auch gehörlose . der -. man arbeitet von 2004 bis 2009 ausbildungstechnisch mit dem plussport behindertensport schweiz zusammen . und . 3 . für die sommer deaf olympics 2017 standen mehrere städte zur auswahl , nämlich barcelona , buenos aires und ankara . we submitted 5 runs to the shared task, three of them using resnet10_3d and the parameters are provided in a preliminary automatic evaluation has been made available by the organizers and it is shown in in general, translation quality is always very bad, but results are slightly better for the focusnew subset. focusnews' input videos are shorter and this might imply a better alignment between videos and subtitles, improving the training. some of our test outputs contain repetitions of (parts of) sentences from focusnew dataset. since this subcorpus is dominant in the final training ( ## conclusion this paper presented an overview and some insights on spatio-temporal sign language representation which were used in the dfki-mlt submission for the wmt-slt 2022 shared task. to achieve our goal of building a fully end-to-end sign language model, we worked closely on the representation learning of visual features. most of previous techniques for slt simplify the feature representation by extracting spatial features and not benefiting from temporal features. this choice is motivated by the lack of temporal boundaries in sign language videos. to extract spatio-temporal features one can use 2d + 1d cnn approaches but this does not allow a fully end-to-end training as it still requires pretraining in another well-resourced task like object classification. in order to construct a specific representation model for sl and learn temporal modeling in a single model, we choose 3d cnns and trained them from scratch simultaneously with the textual counterparts. the translations produced by this architecture are very short and output only high frequency tokens; in few cases, full fluent and grammatical sentences are constructed but their meaning unrelated to the source. the generation of short sentences might be a limitation of our approach that builds a sentence representation with an output conversion method that does not split a sentence in subunits that can be weighted by the transformer's attention mechanism to generate the output. however, all the systems in this shared task's leaderaboard have translation scores close to zero. this shows the extreme difficulty of slt and how bad current systems generalize to new data sets. we believe that system comparisons with such a bad translation quality do not allow to extract meaningful conclusions. in our future work, we investigate on different temporal modeling coupled with the 3d cnns approach to further pursue the goal of developing a high-quality end-to-end system.