Enhancing Accessible Communication: from E uropean P ortuguese to P ortuguese S ign L anguage (2023)

Abstract: Portuguese Sign Language (LGP) is the official language in deaf education in Portugal. Current approaches in developing a translation system between European Portuguese and LGP rely on hand-crafted rules. In this paper, we present a fully automatic corpora-driven rule-based machine translation system between European Portuguese and LGP glosses, and also two neural machine translation models. We also contribute with the LGP-5-Domain corpus, composed of five different text domains, built with the help of our rule-based system, and used to train the neural models. In addition, we provide a gold collection, annotated by LGP experts, that can be used for future evaluations. Compared with the only similar available translation system, PE2LGP, results are always improved with the new rule-based model, which competes for the highest scores with one of the neural models.

## introduction according to the portuguese association of deaf people foot_0 , there are around thirty thousand deaf people that use portuguese sign language (lgp) in their daily lives. however, european portuguese (ep) and lgp are two different linguistic systems, and communication between hearing and hearingimpaired people is difficult, leading to a communication gap between both groups. there have been some attempts at developing a system that translates ep into lgp glosses @xcite @xcite , but the majority only present toy examples, relying on a small set of hand-crafted rules and disregarding non-manual movements. pe2lgp @xcite is a rule-based translation system from ep to lgp. some of its rules are hand-crafted and some are automatically extracted from a linguistic corpus with annotations of lgp videos foot_1 , from now on colin. colin is the only existing lgp annotated corpus (to the best of our knowledge). it currently consists of 113 hours of video recordings, with 20 of them being annotated at various linguistic levels (elan foot_2 @xcite was used). the videos in the corpus were recorded between 1992 and 2019 and feature hearing-impaired signers ranging from 4 to 89 years old. pe2lgp still has some limitations, as the subset of colin used to develop its rules is very small (three minutes) and it is necessary to perform some manual tasks to create pe2lgp translator's grammar. in this paper, we take advantage of an extended version of the corpus used in pe2lgp and present a rule-based system that is now fully automatic. we use this rule-based system to create a parallel corpus between ep and lgp, from now on lgp-5-domain (lgp5), with text from different domains -simple sentences, social media, poetry, dialogue, and news. lgp5 is then used to fine-tune two large multilingual neural machine translation models, and evaluated on a gold collection, built by lgp experts foot_3 . with this work, we hope to contribute to benchmarking this task. to illustrate the task at hand, since "ana" is a proper name, it should be fingerspelled -represented by dt(a-n-a). in the declarative sentence of the second row "rainha" (queen) is translated to "mulher rei" (woman king). ## related work we will only consider the translation to a sign language and not from a sign language, which is out of the scope of this paper. early automatic translation systems, such as the ones discussed in @xcite @xcite , adopted a rule-based approach for performing the translation between the source and the target languages. an example of an (expertdefined) rule employed in the atlaslang mts system @xcite is shown in equation 1 . equation 1 checks if a word has a feminine gender and adds the term female after it. notice that the involved types of grammar could vary. for instance, the work described in @xcite uses a lexicalized tree adjoining grammar to translate english words into american sign language glosses. regarding lgp, there are a few attempts at creating a translation system such as the ones described in @xcite @xcite . virtualsign @xcite is a system that performs bidirectional translation between portuguese and lgp. regarding the text-to-sign translation, the input sentence is passed through a set of grammar rules and, afterward, the system directly associates each word in the sentence with a corresponding sign stored in a database. this database contains the needed information to animate a 3d avatar. unfortunately, no information is available regarding the creation of the translation rules and the evaluation of this system. pe2lgp has a rule-based translation system @xcite , an avatar to perform lgp, and a database with signs @xcite . its rules were semi-automatically extracted from colin. in this work, we take advantage of the current corpus and we also automate the whole process. another work that should be mentioned is if2lgp @xcite which consists of two modules. the first is responsible for conducting syntactic and morphological analysis, while the second contains translation rules to convert portuguese words into lgp glosses. however, the creation of these translation rules was based on a small dataset of ten sentences. over the past years, significant progress has been made in the field of sign language translation, thanks to the advancements in statistical and neural machine translation. for instance, the approach proposed by @xcite uses a phrase-based method, trained using parallel corpora; also, the atlaslang nmt system @xcite employs a neural machine translation approach. regarding lgp, a neural approach is also explored by @xcite . it adopts a hybrid structure combining rule-based and neural machine translation approaches. according to the authors, the dataset consists of 150,000 sentences. however, the grammar and the data are not available. ## the lgp5 dataset in the following, we describe the lgp5 dataset that comprises 37,500 automatically annotated sentences from 5 different domains (7,500 from each domain) that were used to train the neural models. additionally, 200 sentences were manually annotated (gold collection), 40 from each domain. ## experiments for the evaluation experiments, we evaluated our systems with three metrics: bleu-4 @xcite , rougel @xcite , and ter @xcite . in a preliminary experiment, we evaluated our models in a test set described in @xcite . this test set has 58 sentences ep/lgp. we also evaluated our models against pe2lgp. next, we used our gold collection to evaluate the same models ( m2m is not always the one that obtains the highest scores, as our rule-based system is the best system in the social media dataset, for all the measures, and in some of the other domains, for some of the metrics. pe2lgp initially seemed to have the best ter score in the poetry domain. however, a closer look revealed that it failed to provide a translation for four out of the forty sentences. this affected the ter calculation, making the final scores appear higher than they actually were. considering this, pe2lgp's scores were consistently worse than what was initially thought, and in reality, our rule-based model had the best ter score in the poetry domain. it is not possible to fully determine the bestperforming model. however, our rule-based model and the fine-tuned m2m stand out. within these systems, it is clear that the performance depends on the type of sentences to be translated. specifically, m2m achieved higher results in simpler sentences and in the domain of dialogues, whereas rb excels in the domains of social media and poetry. from this, we can infer that the m2m was able to better generalize for unseen data for simpler sentence structures. on the other hand, the rules extracted from the corpus proved to be more effective for contemporary and poetry sentence styles, aligning with the informal and formal discourse present in the videos within the corpus. analyzing the translations from both the rulebased and the m2m models and comparing them with the reference translation, we perceived that the majority of the errors are due to: in appendix a, ## conclusion and future work we contribute with a fully automatic rule-based approach to translate ep to lgp and also with two neural models. we also contribute with an automatically labeled dataset (37,500 pairs ep/lgp) with texts from 5 different domains and a gold collection of 200 sentences (40 from each style). our rulebased system and the neural m2m model share the podium in all scenarios. we benchmark, in this way, the task of ep to lgp translation. for future work, different methods of evaluation should be used to measure fluency and adequacy. to better validate the results, the gold collection should be extended. ethics statement: in order to uphold the quality of our work for the end-users, that is deaf people speaking lgp, the gold set was created by lgp experts, who received certification for their work. furthermore, both the data and code utilized in our research are accessible, in order to promote transparency and reproducibility.