Approaching Sign Language Gloss Translation as a Low-Resource Machine Translation Task (2021)

Abstract: A cascaded Sign Language Translation system first maps sign videos to gloss annotations and then translates glosses into a spoken languages. This work focuses on the second-stage gloss translation component, which is challenging due to the scarcity of publicly available parallel data. We approach gloss translation as a low-resource machine translation task and investigate two popular methods for improving translation quality: hyperparameter search and backtranslation. We discuss the potentials and pitfalls of these methods based on experiments on the RWTH-PHOENIX-Weather 2014T dataset.

## introduction more than 70 million deaf people around the world use sign language as their primary language to communicate. in a society dominated by hearing people and spoken languages, there is a risk that deaf people may experience inconvenience and isolation. in countries like india, iran, and russia, lack of sign language interpreters hampers access to public services and courts @xcite . automatic sign language translation (slt) has recently gained increasing attention from researchers and would help remove the communication barriers. sign language is not simply a visual form of spoken languages. it has its own linguistic rules, including phonology, morphology, syntax and semantics that are different from other languages @xcite . for example, in american sign language, the subject or object might be omitted in certain situations. there is also a process called topicalization, where prominent information is signed first, resulting in an adjustment to the basic svo word order. linguists use glossing to annotate signs, which can be viewed as a written form of sign language. glosses can be taken as intermediate representations when translating continuous sign utterances to spoken language sentences. previous work on slt adopts either an end-to-end system that maps sign language videos directly to spoken languages, or a cascaded system, as shown in figure 1 , that first relies on continuous sign language recognition (cslr) to produce sign glosses and then passes the glosses into a neural machine translation (nmt) system @xcite @xcite . importantly, @xcite reports that the cascaded system outperforms the end-to-end system by a large margin (18.13 vs. 9.58 bleu). in this work, we focus on improving the nmt component of cascaded systems, which attracts much less attentions compared to the cslr component of cascaded systems @xcite @xcite . sign language gloss translation is a challenging problem due to the scarcity of annotated parallel data. the popular continuous slt dataset, "rwth-phoenix-weather 2014t" @xcite contains 7,096 gloss-text examples in training set. however, the state-of-the-art past-[long-ago] yourself little girl pro-2 want grow-up future do-what sign video sign glosses english translation when you were a little girl, what did you want to do when you grow up? ## related work most of the sign language processing research has focused on sign language recognition @xcite @xcite . recent work started to show an interest in clsr @xcite @xcite @xcite . however, only a few works move forward to tackle this problem as a slt task. @xcite formalized slt in the framework of nmt and released the first publicly available slt dataset, phoenix14t. based on this dataset, @xcite and @xcite explored slt with transformers and developed both end-to-end and cascaded systems where gloss annotations are used as intermediate representations. @xcite proposed a sign language translation system based on human keypoint estimation and also introduced the keti dataset, which consists of korean sign videos and annotations. keti has only 105 gloss annotations. also, the sentences in keti are relatively short as they are related to emergency situations. @xcite introduced the aslg-pc12 dataset, which consists of millions of english sentences and corresponding american sign language glosses. however, the glosses are generated by applying transformation rules on english sentences and are not reliable for the study of slt. ## data and setup we evaluate the effectiveness of hyperparameter search on a parallel gloss-text dataset. for back-translation, we experiment with monolingual data from the same domain as the parallel data and data from a different domain respectively. in this section, we will describe the datasets and nmt models in detail. we will also introduce our experimental setup. ## hyperparameter search hyperparameter selection is crucial to build a good nmt system. it is especially the case for low-resource scenarios when the default hyperparameter settings are very likely to be ineffective. as reported in @xcite and @xcite , the nmt systems developed for low-resource translation tasks disagree a lot with those trained on high-resource corpora on the optimal hyperparameter choices. furthermore, datasets in different domains and language pairs all differ in their hyperparameter preference. they also show that adjusting hyperparameters can cause bleu differences of more than 20 in some datasets. ## back-translation back-translation proposed in @xcite has shown its effectiveness in utilizing monolingual data to improve the translation performance. it is particularly used in low-resource scenarios. when it comes to sign language translation, the written text is always abundant, whereas the glosses and parallel examples are expensive to get and are not sufficient to train a robust nmt model. this sets a good stage for back-translation. the workflow of back-translation is illustrated in figure 3 . in order to train a more robust gloss-text translation model, one first trains a text-gloss model using the phoenix14t parallel data (figure 3 , step 1). this model is then employed to translate monolingual german text in the domain of ted talk to glosses (figure 3 , step 2). this synthetic parallel corpus is then concatenated with the phoenix14t data to train the final gloss-text system (figure 3 , step 3 ). one problem with our implementation of back-translation is that ted talk subtitles have different styles compared to phoenix14t, which is composed of weather reports, in other words, they are in different domains. domain mismatch makes the translation task even more challenging, as the synthetic parallel data might introduce noises to hurt the performance. in order to alleviate this issue, we adopt two domain adaptation methods to aid back-translation. ## conclusions in this paper, we identify one challenging task in sign language translation, that is the translation between sign language glosses and written languages. we argue that the obstacle lies in the sparsity of parallel data. to conquer this problem, we propose to approach sign language gloss translation as a low-resource machine translation task. we investigate the effectiveness of hyperparameter search and back-translation, which are both widely used by machine translation community for low-resource translations. we conclude that hyperparameter search is necessary, whereas back-translation is susceptible to the quality of additional monolingual data. if there is abundant in-domain monolingual data, back-translation is very promising. otherwise, it should be used with domain adaptation techniques, like data selection and fine-tuning to achieve a reasonable performance. given limited parallel data, the upper bound of these low-resource methods is constrained. we thus urge the sign language processing community to put in extra efforts in creating more annotated parallel data.