Example-Based Machine Translation from Textto a Hierarchical Representation of Sign Language (2023)

Abstract: This article presents an original method for Text-to-Sign Translation. It compensates data scarcity using a domain-specific parallel corpus of alignments between text and hierarchical formal descriptions of Sign Language videos. Based on the detection of similarities present in the source text, the proposed algorithm recursively exploits matches and substitutions of aligned segments to build multiple candidate translations for a novel statement. This helps preserving Sign Language structures as much as possible before falling back on literal translations too quickly, in a generative way. The resulting translations are in the form of AZee expressions, designed to be used as input to avatar synthesis systems. We present a test set tailored to showcase its potential for expressiveness and generation of idiomatic target language, and observed limitations. This work finally opens prospects on how to evaluate this kind of translation.

## introduction rosetta 1 is a french project that aimed to study accessibility solutions for audiovisual content. one of the experiments consisted in designing an automatic translation system from text to sign language (sl) displayed through animation of a virtual signer. the three main contributions concerning sl in this project were 1) the constitution of rosetta-lsf @xcite , an aligned corpus of text and sl captured using a mocap system, 2) a translation system from text to azee (a representation of sl content), and 3) a system allowing to generate virtual signer animations from azee input @xcite . this article describes the second contribution : the translation system from text to azee. after an overview of the issues and recent works in the field, we explain our method and design choices, and describe the implementation of the translation system. finally, we give preliminary results and discuss the questions raised for evaluation. ## text-to-sign translation the automatic translation of content from a spoken language into a sl is a fairly recent and still largely unexplored research topic. here we are interested in the translation of text as the source language, in our case in french, and video or 3d animation as the target language, in our case french sign language (lsf). in this section, we look at the main challenges encountered with text-to-sign translation. ## discussion first, our anti-matches approach has some advantages, compared to a sequence-based one. indeed, structures that are specific to lsf can be found in the final translations, which is not the case when the language is reduced to a sequence of glosses or another linear representation. in addition, the approach produces results with a certain form of creativity. in lsf, paraphrases or additions are commonly used, and indeed part of the corpus as initially delivered by the translator at the time of video corpus creation. these elements were later aligned as examples, thus frequently appear in the generated translations, although not always strictly necessary. see the example for "al- the work presented also has limitations. we can observe that some anti-matches are incorrect, for instance : "des personnes pro-brexit" (pro-brexit people) vs "des personnes manifestent" (people demonstrate). the syntactic categories of the antimatch and its correction are not the same (adjective vs. verb), which creates problems during the translation process. if we want to translate "des personnes pro-brexit sont dans la rue" (pro-brexit people are in the street), the algorithm suggests "pro-brexit" as an anti-match for "manifestent", but the result is syntactically unacceptable : "*des personnes manifestent sont dans la rue". the syntactic category of each phrase should be taken into account to prevent such errors and to improve the anti-matching results. finally, a considerable number of fallbacks are present in the output of the algorithm : 3 per result on average. as explained in section 4.1, this is not ideal, and the size of the corpus is undoubtedly a contributing factor : if we increase the number of examples and alignments, the number of fallbacks will decrease and the quality of the translations should hopefully improve. ## conclusion and prospects we have presented a new system of automatic translation from text to azee, based on an example-based machine translation approach, the hierarchical representation of sl azee and an aligned corpus of french text and azee descriptions extracted from the rosetta-lsf corpus. a prototyping implementation of the system has been made and tested on some examples, thus providing a proof of concept. the capacities of this system and the size of the corpus still need to be extended before real evaluations can be carried out. but we can already stress that the evaluation of such a system will not be easy, since it proposes a translation from one language to a representation of another language, not directly readable. automatic evaluation metrics could be considered using target translations references, which are hierarchical, and tree edit distances instead of the levenshtein-type ones used for sequences, e.g. bleu scores. metrics for the evaluation by human of the quality of translations, such as the one proposed in the european qt21 project 5 , provide a scoring sheet with types of errors produced by the translation system, which allows to highlight the shortcomings of the systems and the aspects to improve. this project has proposed a framework for describing and defining custom translation quality metrics. some of the error categories are defined assuming text as a target, which does not apply in our case. a category called "fluency" allows us to evaluate the quality of an utterance, regardless of whether it is the result of a translation. in our case, the target is not even a language utterance, thus this category will need some adjustments. what remains is the category of errors linked to the translation process itself, categorised as "accuracy". it would be interesting to study if this kind of evaluation could be adapted to our system. the issue is to define these categories in the case of sl. it is common or indeed often preferred in sl to introduce contextual information, for example expression (d) figure 1 for "alsace", which should not be 5. https://www.qt21.eu judged as an unwanted addition. furthermore, as azee can be used to generate virtual signer animations which are directly "readable" by language users, fluency error categories could be taken into consideration after this step to complete the evaluation. the establishment of a robust and comprehensive evaluation protocol is clearly a subject of study in its own right, which needs to be pursued in the near future.