Neural Machine Translation Methods for Translating Text to Sign Language Glosses (2023)

Abstract: State-of-the-art techniques common to low resource Machine Translation (MT) are applied to improve MT of spoken language text to Sign Language (SL) glosses. In our experiments, we improve the performance of the transformer-based models via (1) data augmentation, (2) semi-supervised Neural Machine Translation (NMT), (3) transfer learning and (4) multilingual NMT. The proposed methods are implemented progressively on two German SL corpora containing gloss annotations. Multilingual NMT combined with data augmentation appear to be the most successful setting, yielding statistically significant improvements as measured by three automatic metrics (up to over 6 points BLEU), and confirmed via human evaluation. Our best setting outperforms all previous work that report on the same test-set and is also confirmed on a corpus of the American Sign Language (ASL).

## introduction sign language translation (slt) aims to break the language barrier between the deaf or hard-ofhearing communities and the hearing communities. one challenging aspect of slt is the fact that sign languages (sls) are multi-channeled and non-written languages @xcite . therefore, machine translation (mt) for sls cannot directly take advantage of the recent developments in text-based mt. for this purpose, previous work has used written representations of the sls. one of these representations are glosses, where signs are labeled by words of the corresponding spoken language, often including affixes and markers. it is known that glosses have strong limitations as a linguistic representation @xcite . however, given the current status of slt, we have indications that research on sl gloss translation can still be useful. for instance, translation from spoken language text to sl glosses can be useful for interpreters and educational uses (collins et al., 2012). secondly, sl glosses are the only sl representation having several parallel corpora big enough to train mt, and the results may provide indications for the future treatment of other more appropriate representations. previous research on slt has used glosses as an intermediate step to build mt systems for translating from sls to spoken language text @xcite @xcite or from spoken language text to sls @xcite @xcite . in the latter case, glosses allow building the system in two steps, i.e., text-to-gloss translation and glossto-video production (figure 1 ). the glosses can be given to a system for the generation of sl (avatar animations, autoencoders, gans). our work focuses on the first part of this pipeline, text-to-gloss translation, whose results are responsible for the generated sign animations. we find that prior research, despite its improvements, has still not made a big breakthrough in this direction @xcite . sls are low-resource languages (lrls) with regards to mt, since there is little parallel data @xcite . despite the recent progress of mt for lrls @xcite @xcite , few of these methods have been used for mt of sls, such as data augmentation @xcite @xcite and transfer learning @xcite . other efficient techniques, e.g. semi-supervised nmt @xcite and multilingual nmt @xcite have not been explored. we are therefore inspired to extensively explore the effects of the relevant methods on text-to-gloss translation. to the best of our knowledge, this paper is the first work on text-to-gloss: all code of this work has been open sourced. 1 2 related work the early-stage of text-to-gloss translation systems were built using statistical machine translation (smt; @xcite , in an attempt to translate spoken language into a signing 3d avatar using sl glosses as intermediate. although the system evaluations reported good results based on limited data and automatic metrics, deaf users assessed the system conversely. recently, with the advance of nmt, more promising systems have emerged, based on rnns @xcite or as parts of end-to-end transformer systems @xcite , which contrary to our work do not try particular lrl-related methods. more related to our work, in terms of text-togloss translation using lrl-related techniques, @xcite implement a transformer architecture equipped with an editing agent that learns to synthesize and execute editing actions on the source 1 https://github.com/dfki-signlanguage/ text-to-gloss-sign-language-translation sentence. walsh et al. (2022) examine the effect of different tokenization techniques and embedding approaches such as bert and word2vec on the translation performance. egea gómez et al. (2021) propose a syntax-aware transformer injecting syntactic information into the word embeddings. in their follow-up work, egea gómez et al. (2022) achieve remarkable results with a transfer learning strategy that uses various ways of aggregating linguistic features and takes advantage of a pre-trained mbart model by filtering the original embedding and slicing model weights. in our work, we improve over these transfer learning methods using the warm-start strategy. data augmentation has been seen in gloss-totext translation @xcite @xcite . empirical comparison of our efforts with all state-of-the-art systems is presented in section 5 ( ## methods our experiments (figure 2 ) start from data preprocessing and setting the baseline. we then explore data augmentation, semi-supervised nmt, transfer learning and multilingual nmt as measured by automatic metrics. to confirm the consistency of system improvements between the best performing model and baseline, we conduct human evaluation. ## results in this part, we will present the performance of the various methods on both sl datasets and offer some further analysis. ## conclusion in this paper, we applied several techniques, commonly used in low resource mt scenarios, for mt from spoken language text to sign language glosses. we presented an extensive experimentation including data augmentation (combination of different pre-processing methods, back-and forward-translation), semi-supervised nmt, transfer learning with two different methods and multilingual nmt with different data sizes. the experiments were based on the two known natural datasets including gloss annotation, the rwth-phoenix-weather 2014t dataset and the public dgs corpus. automatic metrics indicate significant improvement on the evaluation scores for both datasets when using most of the above methods, whereas the best results are achieved via a multilingual nmt model (6.18 and 2.65 bleu against the baseline respectively). our best system outperforms all other state-of-the-art systems from previous work that report on the same test-set. additionally, the best setting is confirmed with an experiment run on a corpus of the asl. the conclusions are supported by human evaluation.