Frozen Pretrained Transformers for Neural Sign Language Translation (2021)

Abstract: One of the major challenges in sign language translation from a sign language to a spoken language is the lack of parallel corpora. Recent works have achieved promising results on the RWTH-PHOENIX-Weather 2014T dataset, which consists of over eight thousand parallel sentences between German sign language and German. However, from the perspective of neural machine translation, this is still a tiny dataset. To improve the performance of models trained on small datasets, transfer learning can be used. While this has been previously applied in sign language translation for feature extraction, to the best of our knowledge, pretrained language models have not yet been investigated. We use pretrained BERT-base and mBART-50 models to initialize our sign language video to spoken language text translation model. To mitigate overfitting, we apply the frozen pretrained transformer technique: we freeze the majority of parameters during training. Using a pretrained BERT model, we outperform a baseline trained from scratch by 1 to 2 BLEU-4. Our results show that pretrained language models can be used to improve sign language translation performance and that the self-attention patterns in BERT transfer in zero-shot to the encoder and decoder of sign language translation models.

## introduction despite recent advancements in the domain of automated sign language translation (slt), substantial challenges remain. one considerable issue is the lack of labeled data. deep neural networks are data-hungry and neural machine translation models are no exception. the widely used slt dataset rwth-phoenix-weather 2014t @xcite ) contains only 8,257 parallel sentences. as a comparison: for translation between spoken languages, 6,000 parallel sentences is considered a "tiny" amount @xcite . in this work, we investigate how transfer learning can be used to improve the generalization of video-to-text slt models in the absence of large datasets. we integrate pretrained language models such as bert @xcite into our translation models, evaluating translation performance with bert2rnd and bert2bert models @xcite . we also use mbart-50 @xcite , which has been trained on corpora of 50 languages, including german, as our goal is to translate from german sign language (dgs) to german. these pretrained models are prohibitively large for currently available slt datasets. the base version of bert, for example, consists of 12 layers compared to 3 layers in the encoder of the translation model of @xcite . therefore, we perform aggressive layer pruning and parameter freezing in the pretrained models. we use the rwth-phoenix-weather 2014t dataset for which both gloss and text annotations are available. we consider joint continuous sign language recognition (cslr) and slt, or sign2(gloss+text), as it is called by @xcite . we compare several combinations of pretrained transformers and transformers trained from scratch in terms of scores (wer for cslr and bleu-4 for slt) and number of trainable parameters. our results show that the bert based models (bert2rnd, bert2bert) perform best. these models allow us to outperform the baseline, i.e., transformers trained from scratch. due to overfitting, the large mbart-50 model results in significantly worse performance than the baseline. this warrants further investigation in the incorporation of existing language models. we discuss possible options for future work in section 6. the source code of this research project is available at https://github.com/ m-decoster/fpt4slt. ## neural sign language translation several slt systems have been proposed in the past, including rule-based systems @xcite and statistical methods @xcite . we focus specifically on translation from a sign language to a spoken language. to perform this translation, the sign language first needs to be converted into a written or computational form. various notation systems exist, such as glosses and hamnosys @xcite . @xcite focus on text2gloss and gloss2text translation. recent advancements in deep learning and computer vision now allow for sign2text translation, but this requires sizable corpora. several large datasets exist for sign language recognition tasks, in which the goal is to classify glosses from sign language video, e.g., ms-asl (vaezi joze and koller, 2019), wlasl @xcite and autsl @xcite . rwth-phoenix-weather 2014t is a large public dataset for sign language translation @xcite . along with it, a neural slt model is formalized and introduced: a recurrent encoderdecoder with luong attention @xcite . this sign2gloss2text model achieves a bleu-4 score of 18.13 on the test set. in a follow-up work, the recurrent architecture is replaced by transformers @xcite . the authors present a new study showing improvements in bleu-4 scores for gloss2text and sign2text translation. by jointly performing cslr and slt, they increase the bleu-4 test score to 21.32. yin and read (2020) further improve the performance of sign language transformers by using multiple cues (face, hand, full frame and pose information, rather than only full frame information) for cslr and performing sign2gloss2text translation. they achieve a bleu-4 test score of 24 (25.40 using an ensemble of 5 models). these improvements are related to feature extraction rather than network architecture, whereas we aim to improve the translation model by creating a more powerful encoder-decoder model (an orthogonal approach). ## transfer learning and frozen pretrained transformers transfer learning from high-resource to low-resource language pairs can result in better translation performance for low-resource language pairs @xcite . pretraining with huge monolingual corpora can also improve performance of downstream tasks such as sentiment classification and question answering @xcite . pretrained models such as bert can also be adapted as encoders or decoders in a machine translation model to improve translation performance @xcite . transfer learning between different sign languages has previously been used to improve model performance in sign language recognition @xcite . the use of a continuous sign language recognition model as feature extractor for slt, as performed by @xcite , is also a form of transfer learning. to the best of our knowledge, so far no one has leveraged transfer learning of encoders and decoders (rather than feature extractors) for slt from signed to spoken languages. @xcite show that pretrained language models can replace transformers trained from scratch for downstream tasks, even if those downstream tasks are not related to natural language processing (nlp). this is contrary to earlier research, where language models were used in a parameter-efficient transfer learning set-up from one nlp task to another @xcite . self-attention and feedforward layers are frozen and only the layer normalization parameters are fine-tuned. these are, for large language models, only a tiny fraction of the parameters (often less than 1%). as a result, these models, called frozen pretrained transformers (fpts), are more robust against overfitting. we combine the approaches of @xcite and @xcite for slt: we aim to leverage pretrained language models, while freezing the majority of the parameters to avoid overfitting on our extremely small dataset. ## methodology we use the following experimental set-up. first, we reproduce the baseline of @xcite using the code base provided by the authors foot_1 . then, we lay out several experiments with different models with either an fpt encoder or fpt encoder and decoder. we compare several degrees of fine-tuning, ranging from only layer normalization to everything except the self-attention layers. due to computational constraints, we use a sequential approach for hyperparameter tuning. however, we tune two correlated hyperparameters together: the number of layers and degree of fine-tuning. ## results we find an optimal number of layers and loss weights for the different models based on the development set bleu-4 score. we discuss the amount of trainable parameters for the optimal models found for each variant, while at the same time listing the development set score for the optimal models. we finally compare the development set and test set results with the baseline. ## discussion in this work, we have proposed using fpts for neural slt. we have compared several encoderdecoder architectures: ## conclusion we have presented and compared three different approaches to using pretrained language models for a sign2(gloss+text) slt task: bert2rnd, bert2bert and mbart-50. we outperform the baseline, which is a transformer trained from scratch, by replacing the encoder with the first 2 layers of bert-base. we freeze the attention patterns and show that the patterns learned during the training of the bert model transfer in zero-shot to slt. the decoder can also be initialized using a pretrained language model, but we obtain better results if the decoder is trained from scratch. we attempt to integrate mbart-50, a multi-lingual translation model, hoping to obtain translations of a higher quality due to this model having been pretrained on german texts. however, we are unable to reach baseline performance with mbart-50. further research should investigate whether smaller translation models obtain better performance, or whether language models can be used as priors to regularize the slt model. our best result, a bleu-4 score of 22.25 on the test set of rwth-phoenix-weather 2014t, is obtained using a bert2rnd model. the bert2rnd methodology is easy to implement and can be combined with other advances such as improvements in feature extraction.