Enhancing Translation of M yanmar Sign Language by Transfer Learning and Self-Training (2023)

Abstract: This paper proposes a method to develop a machine translation (MT) system from Myanmar Sign Language (MSL) to Myanmar Written Language (MWL) and vice versa for the deaf community. Translation of MSL is a difficult task since only a small amount of a parallel corpus between MSL and MWL is available. To address the challenge for MT of the low-resource language, transfer learning is applied. An MT model is trained first for a high-resource language pair, American Sign Language (ASL) and English, then it is used as an initial model to train an MT model between MSL and MWL. The mT5 model is used as a base MT model in this transfer learning. Additionally, a self-training technique is applied to generate synthetic translation pairs of MSL and MWL from a large monolingual MWL corpus. Furthermore, since the segmentation of a sentence is required as preprocessing of MT for the Myanmar language, several segmentation schemes are empirically compared. Results of experiments show that both transfer learning and self-training can enhance the performance of the translation between MSL and MWL compared with a baseline model fine-tuned from a small MSL-MWL parallel corpus only.

## introduction in myanmar, approximately 1.1 m of the population is deaf or has a hearing impairment. 1 hard-of-hearing people have difficulty comprehending spoken languages because they cannot distinguish sounds. they mostly rely on myanmar sign language (msl) for communication instead of voice. since the structure of the grammar, syntax, and lexicon of msl are different from myanmar written language (mwl), both deaf and hearing people find it rather difficult to learn. in 2010, the myanmar government launched a project to establish a standard sign language with the aid of the japanese federation of the deaf @xcite . this highlights the importance of supporting and promoting msl to ensure the deaf community has equal access to education and opportunities in myanmar. currently, a relatively small number, 0.006% of deaf people have a university education. this percentage is significantly smaller than that for the general population of myanmar. however, there are few (and limited) assistive technologies available for them. therefore, deaf people require appropriate ways or tools to enhance communication with hearing people as well as to support their education. nowadays, machine translation (mt) plays a role in breaking down language barriers and improving communication between people from various cultures and backgrounds. however, translating low-resource languages is still challenging. one of the solutions to this problem is transfer learning. transfer learning in mt allows models to use knowledge acquired from other languages to enhance their performance in the target language. it can reduce the costs of the construction of large parallel corpora, enabling the development of high-quality mt systems for low-resource languages. another technique to tackle the sparseness of the data is semisupervised learning with self-training, which can construct a parallel corpus automatically. the goal of this paper is to develop a system to translate msl to mwl and vice versa. this is a difficult task since the available parallel corpora are very limited. to address the challenge of translating this low-resource language, we propose an approach that combines transfer learning and self-training. although a few studies have so far been made of the translation of msl as will be reported in subsection 2.2, there has not been any previous attempt to apply those two techniques to the translation of msl. we also carry out several experiments to empirically investigate how effective the transfer learning and self-training are. ## myanmar sign language this section briefly introduces the characteristics of msl, which is the primary communication language for deaf people in myanmar. to convey meaning, there are two types of features: manual features and non-manual features. the manual features can be categorized into three types: hand shape, hand location, and orientation, which represent words and concepts. to convey additional meanings, msl also incorporates non-manual features such as movements of the head, eyes, eyebrows, mouth, shoulders, and facial expressions. the facial expressions represent questions, negation, relative clauses, boundaries between sentences, and the argument structure of some verbs. for example, msl uses non-manual marking, similar to american sign language (asl), to convey yes-or-no questions. that is done by raising the eyebrows and moving the head forward @xcite . msl is a natural language with a diverse variety of linguistic features such as grammar, vocabulary, word order, and so on. such linguistic features are distinct from those of the written language of myanmar. the myanmar language is tonal and syllable-based, whereas msl relies on visual-spatial elements to convey meaning. additionally, the grammar of msl and mwl is different. for instance, the grammatical structures of the sentence "do you travel often?" of msl and mwl are shown in figure 1 . the word order of mwl is "you," "travel," "often," and "go" followed by the question mark. in contrast, in msl, the words "often" and "go" are switched reflecting the visual-spatial nature of the sign language. in addition, the question mark is omitted from the word sequence and indicated by a non-manual gesture. that is, they raise their eyebrows to indicate that the sentence is a question. this study focuses on translating word sequences from msl to mwl and vice versa. sentences in msl are conveyed using glosses, which serve as textual representations of signs. as such, this endeavor can be categorized as a text-to-text translation task, akin to conventional machine translation tasks. our research may pave the way for a comprehensive system that facilitates seamless conversion between msl and mwl. however, it's worth noting that while our current approach can convert an msl gloss into a sign or gesture, it doesn't account for nonmanual features -only the manual features represented by words are considered. expanding the translation process to encompass both manual and non-manual features of msl remains a challenge for future endeavors. ## proposed method this section describes our proposed method for translation between myanmar sign and written languages. we use multilingual pre-trained text-to-text transfer transformer (mt5) @xcite as a base translation system, which is multilingual extension of the text-to-text transfer transformer (t5) @xcite . figure 2 shows a flowchart to train our mt model. our method consists of two basic methodologies. the first is transfer learning. firstly, the parent mt model is obtained by fine-tuning the mt5 model using a parallel corpus of highresource languages, asl and english. then the child mt model is trained by fine-tuning the parent model using a relatively small amount of a parallel corpus of the source and target languages. the other basic method is self-training. a new parallel corpus of msl and mwl is obtained by translating sentences in a monolingual corpus using the initial child mt model. the final mt model is obtained by fine-tuning the parent mt model using the enlarged parallel corpus. ## conclusion this paper proposed a novel method to train a machine translation (mt) model for translating between myanmar sign language (msl) and myanmar written language (mwl). to tackle the problem posed by the fact that msl is an extremely low-resource language, an mt5 pre-trained model was used as the backbone, and then transfer learning and self-training were applied to improve the quality of the mt system. the contribution of this paper is summarized as follows. in the near future, we will extend our method for the translation of msl to include both manual and non-manual features. we will also evaluate our mt model from the practical point of view when it is applied for downstream tasks such as cross-lingual information extraction.