Scaling Back-Translation with Domain Text Generation for Sign Language Gloss Translation (2023)

Abstract: Sign language gloss translation aims to translate the sign glosses into spoken language texts, which is challenging due to the scarcity of labeled gloss-text parallel data. Back translation (BT), which generates pseudo-parallel data by translating in-domain spoken language texts into sign glosses, has been applied to alleviate the data scarcity problem. However, the lack of large-scale high-quality in-domain spoken language text data limits the effect of BT. In this paper, to overcome the limitation, we propose a Prompt based domain text Generation (PGen) approach to produce the large-scale in-domain spoken language text data. Specifically, PGen randomly concatenates sentences from the original in-domain spoken language text data as prompts to induce a pre-trained language model (i.e., GPT-2) to generate spoken language texts in a similar style. Experimental results on three benchmarks of sign language gloss translation in varied languages demonstrate that BT with spoken language texts generated by PGen significantly outperforms the compared methods. In addition, as the scale of spoken language texts generated by PGen increases, the BT technique can achieve further improvements, demonstrating the effectiveness of our approach. We release the code and data for facilitating future research in this field.

## introduction sign language is the dominant form of communication for the deaf and hearing impaired community. sign language processing has received substantial attention in the last few years and achieved significant progress @xcite @xcite . among them, sign language translation (slt) aims to transform continuous sign language videos into natural spoken language texts @xcite . slt consists of two subtasks: (1) sign language understanding task that recognizes the continued videos to the sign glosses; and (2) spoken language gloss translation (slgt) task that generates the spoken language text of the given sign glosses. in this work we focus on the second sub-task, i.e., slgt. data scarcity has been considered the major limitation of sign language gloss translation @xcite . to alleviate the data scarcity problem, backtranslation @xcite , which translates in-domain spoken language texts into sign glosses to construct synthetic parallel data, has been adopted and achieved certain success in slgt. however, the lack of large-scale high-quality indomain spoken language text data limits the capability of back-translation for the slgt task @xcite . the common practice is adopt-ing data selection @xcite , or data mining @xcite approaches to obtain in-domain data. but theses approaches assume the availability of enough in-domain data or expert knowledge (e.g., language background to mine the in-domain data in a specific language), which prevents them from applying to gloss translation task that has broader scenarios (e.g., more languages and domains). in this work, we propose a prompt-based domain text generation (pgen) to generate largescale high-quality in-domain spoken language text data, motivated by the advances in data augmentation with pretrained language models (plms). the main idea is to induce the large plms to mimic the style of original spoken language texts with prompt-based learning techniques @xcite . our pgen approach is able to generate large-scale in-domain spoken language texts based on the small-scale original monolingual texts and maintains diversity ( §2.1). besides, our approach can be performed without requiring largescale in-domain data or expert knowledge of the sign language domain and maintain the high quality of generated in-domain spoken language texts. finally, we employ a sequence-sequence pretrained model (e.g, mt5) to translate spoken language texts generated by pgen into sign glosses and synthesize gloss-text pseudo-parallel data ( §2.2). we conduct extensive analyses of the spoken language texts generated by pgen. we find that the generated and original spoken language texts share a similar word distribution ( §3.2). to further verify the effectiveness of pgen, we also conduct back-translation experiments for the slgt task with large-scale in-domain spoken language texts generated by pgen ( §3.3). experimental results on three widely used benchmark datasets across languages and domains show that back-translation with spoken language texts generated by pgen significantly outperforms the compared methods. most importantly, as shown in figure 1 , when scaling the spoken language texts generated by pgen in bt approach, the performance of the gloss-totext translation task can achieve constant improvement while conventional data selection approach failed. the contributions of our work are summarized as follows: ## methodology the whole framework of this work includes two components, i.e., the pgen method for in-domain text generation and the bt model (i.e., text-to-gloss translation) for constructing synthetic data. we will introduce more details for these two components in this section. for clarity, we provide the definition of slgt (i.e., gloss-to-text translation task) and notations used throughout the paper as below. task definition. let x and y denote the gloss annotations and spoken language texts, and x and y represent the sentence sets of corresponding languages. the dataset of gloss-text pairs can be expressed as d @xmath0, y i )} n @xmath1, where x @xmath2, y @xmath3. given a sequence of gloss annotations, the task is to output the corresponding fluent and semantically equivalent sentence. ## experiments in this section, we conduct both intrinsic and extrinsic evaluations @xcite for the proposed pgen approach. for intrinsic evaluation ( §3.2), we perform analyses on the domain properties of spoken language texts generated by pgen. as for extrinsic evaluation ( §3.3), we conduct sign gloss translation experiments using backtranslation approach with the generated texts. the performance of the downstream task can indirectly reflect the effectiveness of pgen. ## analysis to gain a deeper understanding on our pgen approach, we conduct extensive analyses in terms of ablation study and translation outputs. ## conclusion in this paper, we propose the pgen approach to produce large-scale in-domain monolingual texts based on the original small-scale texts of the glosstext parallel data. with pgen, we scale backtranslation and achieve significant and consistent improvements on three benchmark datasets for slgt task. extensive analyses suggest that our approach generates monolingual texts with similar linguistic properties as the original monolingual texts, thus outperforms the compared methods in terms of both low-frequency word prediction and long sentence translation. future work includes exploring chatgpt for sign language translation task by using proper prompts @xcite .