Experimental Machine Translation of the S wiss G erman Sign Language via 3 D Augmentation of Body Keypoints (2022)

Abstract: This paper describes the participation of DFKI-SLT at the Sign Language Translation Task of the Seventh Conference of Machine Translation (WMT22). The system focuses on the translation direction from the Swiss German Sign Language (DSGS) to written German. The original videos of the sign language were analyzed with computer vision models to provide 3D body keypoints. A deep-learning sequence-to-sequence model is trained on a parallel corpus of these body keypoints aligned to written German sentences. Geometric data augmentation occurs during the training process. The body keypoints are augmented by artificial rotation in the three dimensional space. The 3D-transformation is calculated with different angles on every batch of the training process.

## introduction despite the enormous progress of the machine translation (mt) of spoken (and written) languages, the mt of sign languages is in a very early stage @xcite . two major challenges are (a) the multimodal and multilateral nature of the sign languages and (b) the lack of data. on the one side, the multilateral and multimodal nature of the sign languages requires deep-learning topologies that differ substantially from the ones used in text-based mt. on the other side, the lack of data makes difficult the utilization of end-to-end deep learning algorithms, which usually require vast amounts of data. as a result, deep-learning experiments have been executed for very few sign languages (e.g. german sign language, dgs; american sign language, asl) and narrow domains (e.g. weather forecasts), leaving open questions on the generalization of the methods to other sign languages and broader domains. this year's sign language translation (slt) task of the seventh conference of machine translation (wmt22) is contributing significant to this direction, by adding a new language pair (swiss german sign language -dgsg -to german) and allowing extensive experimentation from several participants on the same dataset. our system uses computer vision models to analyze the sign language videos into body keypoints and uses these keypoints as the source-side input of the neural mt transformer, allowing to perform data augmentation via geometrical augmentations. despite the difficulty of this shared task and the low results obtained, we publish this paper as a technical report, with the hope that it can contribute to the further research of this direction. the rest of the paper is organized as following. section 2 positions our contribution amidst related work. section 3 describes the methods for training the system and section 4 the technical set-up of the experiment. section 5 provides and discusses some results, while section 6 gives some conclusion and ideas for further work. ## related work latest work on mt of sign languages has shown significant improvements using deep learning methods from the fields of computer vision and mt. state of the art work @xcite @xcite employees transformers, which are given frame embeddings extracted from the videos of the signers. contrary to the use of pixel-based frame embeddings, @xcite suggests to use body keypoints from the hands, the skeleton and the face as input to the transformers. this requires to split the translation pipeline into a first phase, recognizing 3d keypoints from videos, and has the advantage that they can be augmented by applying transformation techniques. our paper presents an implementation of that idea, applied to the case of dsgs. the use of body keypoints has been considered by @xcite , where skeleton pose information is processed together with the video frame input. @xcite use 2d coordinates of body keypoints to train the neural mt systems, but contrary to our work, they do not perform any geometrical transformations to the keypoints. @xcite analyze the applicability of the pose estimation systems to sign language recognition by evaluating the failure cases of the recognition models. ## method our system consists of three modules. the first module converts images of the signer into intermediate keypoint representations. the second module employs data augmentation to increase sample efficiency and decrease the effect of spurious feature correlations. spurious data correlation in high dimensional spaces can lead to clever hans effects @xcite . the last module is the trainable transformer that translates from keypoint representation to german text, while interacting with the augmentation module. ## experiment setup the experiment took place using only the corpora focusnews, as provided by the shared task organizers, including keypoints precomputed with me-diapipe. due to time restrictions, the srf corpus was not used, since it did not provide any keypoints. the training set had 10,136 sentences, the validation set 420 sentences and the test set 488 sentences. due to problems with the keypoint-subtitle alignment only 393 of the 420 sentences of the validation set were used. for training the model we modified the nmt toolkit joeynmt 1 @xcite , extending the slt branch created by camgöz et al. (2020). we followed the text pre-processing of the previous implementation, which included text lowercasing. the geometrical transformations were done with array computations using numpy @xcite . the automatic evaluation metrics were computed using sacrebleu @xcite . in order to optimize the system we ran several experimental rounds. the training parameters for all rounds can be seen in ## results from the first experiments it was obvious that the use of bleu-4 as a validation metric could not contribute to the optimization, because its values are always zero and also the training time was very short. for this reason we chose chrf as validation metric for our last two experiments. increasing the patience deemed necessary, so that the training mechanism can get enough random samples from the augmentation process. for our best iteration we experimented with both 3 and 4 layers, resulting into slightly better performance with the 4 layer setting. in overall, the results of our experiments, as measured by automatic metrics, showed very low performance. no version of our pipeline could achieve non-zero bleu-4 score on the provided development set, meaning that no n-gram of order 4 was correctly matched between the hypothesis and the reference. the experiments measured with bleu-3 and chrf indicate as better run the configuration with 60 degrees rotation range at the x axis, 10 degrees on the other axes, and a patience of 500. when analyzing the output on the validation set we found that for the 393 different sentences of the validation set, only 15 different translations were repeatedly produced as highlighted in figure 2 and listed in appendix a. the two most common translations make up for 92% of the cases. this behaviour suggests that the model learned two main ## conclusion and further work due to the poor results, very little can be concluded about the effect of the proposed geometric augmentation strategy. as suggested by the preliminary results of the shared task @xcite no group was able to achieve good results on the task. unfortunately, due to the strict workshop timeline we could not perform further experiments to empirically prove the causes of this low performance. we are planning to do this in future work, including an ablation study of the different modules and a comparison with the state-of-the-art on other datasets. further research should be invested in exploring the possible use cases for geometric data augmentation in mt of sl.