Data Augmentation for Sign Language Gloss Translation (2021)

Abstract: Sign language translation (SLT) is often decomposed into video-to-gloss recognition and gloss to-text translation, where a gloss is a sequence of transcribed spoken-language words in the order in which they are signed. We focus here on gloss-to-text translation, which we treat as a low-resource neural machine translation (NMT) problem. However, unlike traditional low resource NMT, gloss-to-text translation differs because gloss-text pairs often have a higher lexical overlap and lower syntactic overlap than pairs of spoken languages. We exploit this lexical overlap and handle syntactic divergence by proposing two rule-based heuristics that generate pseudo-parallel gloss-text pairs from monolingual spoken language text. By pre-training on this synthetic data, we improve translation from American Sign Language (ASL) to English and German Sign Language (DGS) to German by up to 3.14 and 2.20 BLEU, respectively.

## introduction sign language is the most natural mode of communication for the deaf. however, in a predominantly hearing society, they often resort to lip-reading, text-based communication, or closedcaptioning to interact with others. sign language translation (slt) is an important research area that aims to improve communication between signers and non-signers while allowing each party to use their preferred language. slt consists of translating a sign language (sl) video into a spoken language (spl) text, and current approaches often decompose this task into two steps: (1) video-to-gloss, or continuous sign language recognition (cslr) @xcite ; (2) gloss-to-text, which is a text-to-text machine translation (mt) task @xcite . in this paper, we focus on gloss-to-text translation. sl data and resources are often scarce, or nonexistent ( §2; @xcite ). gloss-to-text translation is, therefore, an example of an extremely low-resource mt task. however, while there is extensive literature on low-resource mt between spoken languages @xcite @xcite , the dissimilarity between sign and spoken languages calls for novel methods. specifically, as sl glosses borrow the lexical elements from their ambient spoken language, handling syntax and morphology poses greater challenges than lexeme translation ( §3). in this work, we (1) discuss the scarcity of sl data and quantify how the relationship between a sign and spoken language pair is different from a pair of two spoken languages; (2) show that the de facto method for data augmentation using back-translation is not viable in extremely low-resource slt; ## sign vs. spoken language due to the paucity of parallel data for gloss-to-text translation, we can treat it as a low-resource translation problem and apply existing techniques for improving accuracy in such settings. however, we argue that the relationship between glossed sls and their spoken counterparts is different from the usual relationship between two spoken languages. specifically, glossed sls are lexically similar but syntactically different from their spoken counterparts. this contrasts heavily with the relationship among spoken language pairs where lexically similar languages tend also to be syntactically similar the great majority of the time. to demonstrate this empirically, we adopt measures from @xcite to measure the lexical and syntactic similarity between languages, two features also shown to be positively correlated with the effectiveness of performing cross-lingual transfer in mt. lexical similarity between two languages is measured using word overlap: syntactic similarity between two languages is measured by 1-d syn where d syn is the syntactic distance from @xcite calculated by taking the cosine distance between syntactic features adapted from the world atlas of language structures @xcite . figure 2 shows that sign-spoken language pairs are indeed outliers with lower syntactic similarity and higher lexical similarity. we seek to leverage this fact and the high availability of monolingual spoken language data to compensate for the scarcity of sl resources. in the following section, we propose data augmentation techniques using word order modifications to create synthetic sign gloss data from spoken language corpora. ## data augmentation this section discusses methods to improve gloss-to-text translation through data augmentation, specifically those that take monolingual corpora of standard spoken languages and generate pseudo-parallel "gloss" text. we first discuss a standard way of doing so, back-translation, point out its potential failings in the sl setting, then propose a novel rule-based data augmentation algorithm. ## results we evaluate our models across all datasets and sizes using sacrebleu (v1.4.14) @xcite and comet (wmt-large-da-estimator-1719) @xcite . we also compare our results to previous work on phoenix in next, turning to the tuned models, we see that specific and general outperform both the baseline and bt by large margins, demonstrating the effectiveness of our proposed methods. interestingly, general-tuned performs slightly better, in contrast to the previous result. we posit that, similarly to previously reported results on sampling-based back translation @xcite , general is benefiting from the diversity provided by sampling multiple reordering candidates, even if each candidate is of lower quality. looking at figure 3 , we see that the superior performance of our methods holds for all data sizes, but it is particularly pronounced when the parallel-data-only baseline achieves moderate bleu scores in the range of 5-20. this confirms that bt is not a viable data augmentation method when parallel data is not plentiful enough to train a robust back-translation system. ## implications and future work consistent improvements over the baseline across two language pairs by our proposed rulebased augmentation strategies demonstrate that data augmentation using monolingual spoken language data is a promising approach for sign language translation. given the efficiency of our general rules compared to language-specific rules, future work may also include a more focused approach on specifically pre-training the target-side decoder with spoken language sentences so that by learning the syntax of the target spoken language, it can generate fluent sentences from sign language glosses having little to no parallel examples during training.