Tackling Low-Resourced Sign Language Translation: UPC at WMT - SLT 22 (2022)

Abstract: This paper describes the system developed at the Universitat Politècnica de Catalunya for the Workshop on Machine Translation 2022 Sign Language Translation Task, in particular, for the sign-to-text direction. We use a Transformer model implemented with the Fairseq modeling toolkit. We have experimented with the vocabulary size, data augmentation techniques and pretraining the model with the PHOENIX-14T dataset. Our system obtains 0.50 BLEU score for the test set, improving the organizers’ baseline by 0.38 BLEU. We remark the poor results for both the baseline and our system, and thus, the unreliability of our findings.

## introduction the submission of the universitat politècnica de catalunya (upc) to the wmt22 sign language translation (slt) task experimented with vocabulary size, data augmentation techniques and a pretrained system with the phoenix-14t dataset @xcite . up to the author's knowledge, our implementation is the first to build on fairseq, a popular modeling toolkit by meta ai @xcite . slt is a highly complex task because sign language understanding requires a very precise estimation of the signer pose, especially, of its hands. in addition, sign languages have grammatical structures different from spoken languages, which prevents an easy knowledge transfer. sign languages are represented in continuous and high-dimensional spaces, while the transcribed version of spoken languages are represented by discrete tokens of welldefined vocabularies. moreover, the few available sign language datasets can be considered very lowresourced, as they typically contain less than one hundred thousand sentences @xcite . in particular, the total number of sentences from the two datasets provided in wmt-slt22 is 17k. we focus on the sign language translation task of wmt-slt22 which requires participants to predict the translation in spoken language (written) from a sign language video. specifically, it consists of translation from swiss german sign language (dsgs) videos to german (de) text. the organizers of wmt 2022 slt track propose a transformer baseline @xcite that achieves a very low bleu @xcite score, foot_0 which indicates a very poor translation quality. the training data provided consists of two datasets: focusnews @xcite and srf @xcite . the first one contains 197 episodes in dsgs that have an average length of 5 minutes, which amount for a total duration of 19 hours. the second dataset contains 29 videos from live sign language interpretation, that have an average of 30 minutes length, and totals a duration of 16 hours. the paired subtitles are given in standard german from switzerland, which is a dialect of german. the organizers provide keypoints extracted with two off-the-shelf human pose estimators: open-pose @xcite and mediapipe @xcite . ## baseline system for the sake of self-containment, we briefly define and discuss the baseline model proposed by the organizers for this task @xcite . this solution is built on sockeye @xcite , with results for the official development (dev) partition. regarding the model implementation, they use a transformer encoder-decoder, with a symmetrical number of layers for the encoder and decoder. the architecture has 6 layers, 8 heads, 2048 neurons in the feed forward network layers, and an embedding dimension of 512. the input data are 2d openpose keypoints, concatenating the hands and body landmarks. this results in an array @xmath0, . . . , x t where x @xmath1, k is the number of keypoints selected, and t the number of video frames. three baseline results were provided by the wmt-slt22 organizers, shown in train dataset bleu focusnews 0.216 srf 0.589 focusnews + srf 0.157 ## method our submission also adopts a transformer architecture, which we implement with the fairseq sequence modeling toolkit @xcite . up to the author's knowledge, this is the first time that fairseq is used for sign language video understanding. we publish our source code 2 , offering the slt community a novel tool widely used in machine translation for spoken languages. on top of this, we experiment with the vocabulary size, data augmentation techniques and pretraining the system with phoenix-14t dataset. details are described in this section. ## results we train our systems with focusnews and srf. for both datasets, we provide results for the dev set, which contains recordings from focusnews and srf. we choose the 6 best-performing models for dev to submit to the official challenge submission. we then analyze the effect of the vocabulary size based on the bleu obtained for the dev set. we notice that using 4000 subwords provides the best results in terms of vocabulary size (2-4). therefore, we choose this configuration to experiment with pretraining and data augmentation. for this set of experiments, we see a slight improvement when fine-tuning a network that has been pretrained with the phoenix14-t dataset (6). however, adding data augmentation (5) or a combination of both results of models trained with srf are presented in we optimized our systems to obtain the best bleu metric, without taking other metrics into consideration. however, organizers also compute chrf++ @xcite and bleurt @xcite metrics @xcite . we find that the bleurt score shows a similar performance than bleu. however, for the chrf++ metric, which correlates better with respect to human relative rankings, our models score lower compared to other submissions. we provide some examples of the sentences generated by our best-performing model in ## discussion and conclusion we proposed a pipeline to tackle the sign language translation task for the newly released datasets: ref.: letztes jahr haben viele gehörlosen-medien über die erste gehörlosen universität in europa in bad kreuznach in deutschland berichtet. pred.: am letzten samstag, 22. mai, in der schweiz, in der schweiz, in der schweiz, gsc aarau, gsc aarau. ref.: dazu sind 4 politiker eingeladen, die über für behinderte wichtige themen diskutieren werden, wie zum beispiel tv-untertitel, uno konvention für behinderte usw. pred.: das ziel ist es, dass es, dass die nempflichkeit für die nempflichkeitssetzen kann. ref.: der deutsche fernsehsender zdf bietet filme im internet mit untertitel an, sofern der film vorher im fernseher mit untertitel ausgestrahlt wurde. pred.: der schweizerische gehörlosenbund sgb-fss organisiert mit dem schweizerischen gehörlosenbund sgb-fss, der gehörlosen sportverband der gehörlosen sportverband der gehörl osen sportverband der gehörlosen sportverband der gehörlosen sportverband der gehörlosen sportverband. focusnews and srf. our fresh implementation with fairseq slightly improved the baseline provided by the organizers. our findings showed that when training with focusnews, our baseline system has the best performance for test. the changes in vocabulary size did not affect the test performance. furthermore, we showed that using checkpoint averaging does not help for this task. in all cases, we still think that the results are extremely low, which indicate a really poor translation, and there is potential unreliability of the findings due to the close to 0 bleu score. we consider the results we obtained can be further improved, so we leave some experiments for future work. firstly, we believe that a joint training from the two provided datasets could boost the per-formance of the models by bridging the domain gap between these datasets. secondly, we did not see any improvement by pretraining the models with phoenix14-t dataset. however, we think that solving the wmt-slt22 task must require some sort of transfer learning from a pretrained model.