Using Neural Machine Translation Methods for Sign Language Translation (2022)

Abstract: We examine methods and techniques, proven to be helpful for the text-to-text translation of spoken languages in the context of gloss-to-text translation systems, where the glosses are the written representation of the signs. We present one of the first works that include experiments on both parallel corpora of the German Sign Language (PHOENIX14T and the Public DGS Corpus). We experiment with two NMT architectures with optimization of their hyperparameters, several tokenization methods and two data augmentation techniques (back-translation and paraphrasing). Through our investigation we achieve a substantial improvement of 5.0 and 2.2 BLEU scores for the models trained on the two corpora respectively. Our RNN models outperform our Transformer models, and the segmentation method we achieve best results with is BPE, whereas back-translation and paraphrasing lead to minor but not significant improvements.

## introduction sign languages (sl), the main medium of exchanging information for the deaf and the hard of hearing, are visual-spatial natural languages with their own linguistic rules. in contrast to the spoken ones, they lack a written form, on one hand, and use face, hands and body to convey meaning, on the other. however, in our society, spoken languages are used by and large, leading to social exclusion in the everyday life of the deaf and hard of hearing. therefore, recent research is making the most out of the technical advances in the fields of natural language processing (nlp), deep neural networks (dnn), and machine translation (mt), with the aim to develop systems that are able to translate between signed and spoken languages in order to fill the gap of communication between the sl speaking communities and the people using vocal language. most latest approaches tackle the problem, dividing it into two sub-tasks: sign language recognition (slr), also called video-to-gloss, and sign language translation (slt), also known as gloss-to-text translation. the latter uses as an intermediate representation the glosses, described in section 3.1 and section 4.2.1. isolating glossto-text translation serves as a building block of a bigger project, which considers sl as a whole and is done in direct co-operation with members of the sl community. for the rest of this work, we focus on the glossto-text sub-task and treat it as a low-resource textto-text machine translation problem. we explore different known techniques for mt of written languages on the glosses, and report our findings during our experiments with: ## related work sign language translation is a relatively new research field with recent findings made possible thanks to the continuous advances in neural machine translation (nmt). several experiments with sl gloss-to-text translation have taken place in the previous decade using statistical phrase-based machine translation @xcite . @xcite and camgoz et al. (2020) use the transformer architecture for sl translation, and are the first to realize an end-to-end system, combining slr and slt, jointly training based on both video embeddings, glosses and text, being currently the state-of-the-art work in the field. @xcite employ the spatial-temporal multi-cue (stmc) network @xcite ) for the task. there have also been several experiments on the opposite direction: text-to-gloss @xcite . to the best of our knowledge, currently @xcite is the only published work experimenting with back-translation in the context of gloss-to-text translation. their research has been conducted parallel and independent from our studies, and has concluded similar results concerning the use of back-translation in a low-resource sl setting. the main difference is that we further focus on other machine translation techniques, e.g. different models and tokenization schemes, whereas they explore in more detail the gloss-text pairs and their linguistic properties, proposing their own rulebased heuristics with the purpose to generate sl glosses, bearing in mind the specifics of the signed languages. the recent work of @xcite , focusing on the problems related to the machine translation between signed and spoken language pairs, reports the first bleu score on the public dgs corpus, but contrary to our work, no details are given on how the models were trained and evaluated and therefore there can be no direct comparison of the results. ## datasets for our experiments we utilize the following corpora of the german sl, which due to the different gloss annotations are used only separately for our experiments. statistics of the two corpora can be seen in ## experiments we separate our experiments in three main groups. in the first one, described in section 5.1 we initially train two baseline models for both corpora and consecutively make changes to them with the goal to investigate how different model architectures and known configurations of neural mt systems influence them. therefore, we use the best performing models from the first group to further continue our experiments in the second one, described in section 5.2, where we apply three different tokenization schemes -bpe, unigram and custom tokenization, on the gloss and on the german sides of the corpora. ultimately, we utilize the models, which produce the best translations up to this point, in the third group of experiments in section 5.3, where we separately look into two data augmentation techniques -back-translation and paraphrasing. all models are trained using mariannmt @xcite and all configuration parameters are detailed in our repository. ## results in this section we report the results from the three groups of experiments we have conducted. ## conclusion and future work in this work we investigated the effect of several methods used in nmt on the gloss-to-text translation task for a sign language. we present one of the first works that does extensive experiments on both backtranslation and paraphrasing via bilingual pivoting, with the intention to create variance in the data. back-translation gave small improvements: +0.2 on the phoenix14t corpus and +0.1 on the dgs corpus. further investigation on the reasons for the limited contribution of the above augmentation techniques may be directed to the extremely lowresource scenario, the amount and domain of the data, or the particular nature of the sign language glosses. all above methods allowed an improvement of 5 bleu points on the test set (22.7 bleu) for the phoenix14t model, and 2.2 bleu points on the test set (3.8 bleu) for the dgs one. in conclusion, in line with previous research @xcite , we believe that in order to achieve better translation performance, research and experiments should concentrate on two major problems -collecting and annotating more resources, and better understanding the nature of the sign languages with the intention to develop new sl-specific tools.