Machine Translation between Spoken Languages and Signed Languages Represented in S ign W riting (2023)

Abstract: This paper presents work on novel machine translation (MT) systems between spoken and signed languages, where signed languages are represented in SignWriting, a sign language writing system. Our work seeks to address the lack of out-of-the-box support for signed languages in current MT systems and is based on the SignBank dataset, which contains pairs of spoken language text and SignWriting content. We introduce novel methods to parse, factorize, decode, and evaluate SignWriting, leveraging ideas from neural factored MT. In a bilingual setup—translating from American Sign Language to (American) English—our method achieves over 30 BLEU, while in two multilingual setups—translating in both directions between spoken languages and signed languages—we achieve over 20 BLEU. We find that common MT techniques used to improve spoken language translation similarly affect the performance of sign language translation. These findings validate our use of an intermediate text representation for signed languages to include them in natural language processing research.

## introduction most current machine translation (mt) systems only support spoken language input and output (text or speech), which excludes around 200 different signed languages used by up to 70 million deaf people foot_1 worldwide from modern language technology. since signed languages are also natural languages, @xcite calls for including sign language processing (slp) in natural language processing (nlp) research. from a technical point of view, slp brings novel challenges to nlp due to the visual-gestural modality of sign language and special linguistic features signwriting has many advantages, like being universal (multilingual), comparatively easy to understand, extensively documented, and computersupported. in addition, despite looking pictographic, it is a well-defined writing system. every sign can be written as a sequence of symbols (box markers, graphemes, and punctuation marks) and their location on a 2-dimensional plane. to our knowledge, this work is the first to create automatic slt systems that use signwriting. our main contributions are as follows: (a) we propose methods to parse ( §3.3), factorize ( §3.4), decode ( §4.3), and evaluate ( §4.3) signwriting sequences; (b) we report experiments on multilingual machine translation systems between signwriting and spoken language text ( §4); (c) we demonstrate that common techniques for low-resource mt are beneficial for signwriting translation systems ( §5). besides, tasks including sign language recognition @xcite , translation, and production involve transforming one sign language representation to another or from/to spoken language text, as shown in figure 2 4 . we find that existing works cover gloss-to-text @xcite (where "text" denotes spoken language text), text-to-gloss @xcite , video-to-text @xcite , pose-to-text @xcite , and text-to-pose @xcite @xcite . ## data and method the data source we use for this research is sign-bank, the largest repository of signpuddles 7 . a signpuddle is a community-driven dictionary where users add parallel examples of signwriting and spoken language text (not necessarily with corresponding videos and glosses). the puddles contain material from various signed languages and linguistic domains (e.g., general literature or bible) without a strict writing standard. we use the sign language datasets @xcite library to load signbank as a tensorflow dataset. ## experiments and results this section introduces three lines of experiments on both bilingual and multilingual signwriting translation. we use transformer models @xcite that support source and target factors. see appendix b for more details on our training configuration. ## conclusion this work explores building bilingual and multilingual translation systems between spoken and signed languages. instead of representing sign language as videos (or as continuous features derived from videos) common in previous research, we propose to represent sign language in signwriting, a sign language writing system. we argue that using a written form is more amenable to well-established nlp techniques. however, encoding or decoding signwriting in an mt system requires specialized tools. therefore, we propose novel methods to parse, factorize, decode, and evaluate signwriting sequences. our factorization technique divides signwriting sequences into meaningful units such as sign symbols and positional numbers. the factors are then encoded or decoded by a factored transformer model. as a result, we achieve over 30 bleu in the bilingual setting and over 20 bleu for some highresource language pairs in both directions in the multilingual setting. using signwriting as an intermediate representation enables us to reuse tools (e.g., evaluation met-rics) from spoken language translation. we also observe striking similarities to spoken language mt in the experiments themselves. for example, low-resource optimizations have a similar impact, and multilingual models exhibit similar transfer effects. these findings validate our use of an intermediate text representation for signed languages to include them in nlp research.