Translating S panish into S panish S ign L anguage: Combining Rules and Data-driven Approaches (2022)

Abstract: This paper presents a series of experiments on translating between spoken Spanish and Spanish Sign Language glosses (LSE), including enriching Neural Machine Translation (NMT) systems with linguistic features, and creating synthetic data to pretrain and later on finetune a neural translation model. We found evidence that pretraining over a large corpus of LSE synthetic data aligned to Spanish sentences could markedly improve the performance of the translation models.

## introduction the widening of access to technology is crucial in today's highly interconnected online world, and it is important that technologies are made available across languages and for people with different needs. the world federation of the deaf foot_0 states that 70 million people communicate in one of the 400 sign languages (sls) around the world. jointly with the united nations, they supported a resolution foot_1 in order to include the deaf and hard-of-hearing (dhh) community in all matters concerning the provision of technology for them foot_2 , respect the linguistic and cultural identity of signers, and improve access to education and services. according to the spanish national confederation of deaf people (cnse) foot_3 , approximately 2.3% of spain's population experience hearing loss to some degree and a large number of them use lengua de signos española (lse) as their primary means of communication. also, ethnologue estimates that there are between 45 to 75 thousand lse signers. lse was first described in the late 18th century, while only recently a grammar @xcite has been written to capture the features of the language. in 2011, lse was recognized as an official language, and there has been a greater focus on providing resources for signers and learners. as with other sls, lse is produced in the visualspatial modality @xcite rather than the oralauditory modality of spoken languages. manual and non-manual (facial expression, body position) features including the space around the signer can be articulated simultaneously to produce meaning. whereas textual forms are well-established in spoken languages, those capturing the spatio-temporal nature of sls including hamnosys @xcite are long extant but not widely known or used by signers @xcite . the most frequently encountered representation of sls are glosses -a lexeme-based representation using the ambient spoken language of the region where the sl is native. for example, glosses for lse are written in spanish. one criticism of glosses is that a great deal of semantic information is lost @xcite . however, their linearity as text is a beneficial input format for machine learning (ml) models. machine translation (mt) has advanced significantly in recent years, specially thanks to the development of methods based on deep neural networks, reaching quality levels comparable to humans @xcite for spoken languages. despite these advances, mt is in its infancy when it comes to translation between spoken and sign languages or between different sign languages. in this paper we address a little researched topic in mt, that of translating between spanish and lse using a combination of rule-based and neural approaches. we present experiments on building mt systems between spoken spanish and lse using a small parallel corpus of sentences and gloss sequences. we first show a baseline system using only the parallel data, and then present two techniques for improving this baseline: enriching the representation of words and glosses with linguistic information; and using a large corpus of synthetic data for creating a pretrained model, and then finetuning using the original training data. as we will see, this last approach is the one with the most promising results. the rest of this paper is structured as follows: section 2 presents related work on lse and sls in general; section 3 introduces the dataset we base our research on, the id/dl corpus; section 4 describes the different experiments we carried out with this dataset; section 5 shows the evaluation of the experiments over the test partition; and finally section 6 presents some conclusions and future work. ## related work the scarcity of linguistic resources constitutes a major barrier in the adaptation of latest technology to sls @xcite . in fact, sls are considered extremely low resource languages @xcite for mt models. this section explores computational resources and systems existing for lse, sl translation (slt) and processing. ## corpus for our experiments, we use the id/dl corpus @xcite foot_6 , made up of 416 parallel spanish-lse utterances. below, we show one example of the parallel text samples composing this corpus: spanish: deberá tener preparadas las fotografías y documentos necesarios lse: futuro tú obligatorio preparar plural fotografía y plural documento plural necesario we randomly split the dataset into 266 training utterances, 75 dev utterances and 75 test utterances. as can be observed, the train partition contains 290 unique glosses, which cover 89.9% of dev glosses and 85.6% of test glosses. consequently, there are a lot of glosses both in the dev and test sets that the mt models will never see at training time (out-of-vocabulary glosses). the models might overfit train patterns while some of the input sequences may not be properly learnt leading to inaccurate predictions. also, we notice that the glosses in the lse lexicon seem not to be representative enough of the glosses found in the id/dl corpus, as less than 50% of the glosses found on the train, dev and test sets are in the lexicon, which is in line with the problems mentioned in section 2.1. ## experiments we have carried out a series of experiments on building mt models between (spoken) spanish and lse. although id/dl corpus is not a fully comprehensive representation of lse, it is one of the few available lse resources with a suitable format to experiment with ml algorithms. in the present work, we first create a baseline for both translation directions lse↔spanish (section 4.1); then we incorporate linguistic features to boost our mt model (section 4.2); and finally we pretrain a transformer model on synthetic data generated using data augmentation rules, and finetune it with the id/dl training data (section 4.3). all the results in this section are evaluated against the id/dl development set, while section 5 shows results over the test set. ## results and discussion we chose the model that yielded the best results according to bleu-char for each of the described experiments, and we evaluated them over the test set. the models that incorporated linguistic features performed similarly for the dev split. however, this performance is not reflected on the test split, where most models achieve a few points less than the baseline. one of the models, though, seems to have some improvement over the baseline: the word+pos model in the @xmath0. but the word+dep model, which was the most promising on dev, did not bring any improvement over test. in order to understand the big difference in performance achieved by the finetuned models, we measured the vocabulary coverage obtained by the synthetic data corpus created from ancora. the dev and test sets coverage obtained using the ancora and the training split are much higher than using the training split alone. this is because rule 10 in on the other hand, there is no guarantee that the glosses created by rule 10 are indeed valid signs, so this rule is probably fabricating glosses that have no counterpart in lse. it would be possible to alleviate this problem using some other heuristics. one way of doing this could be obtaining the closest gloss in the embeddings space that is an actual lse sign, but since the lse lexicon coverage is so low, further research is needed to get a larger set of valid glosses and signs that could lead better insights on this process. ## conclusions and future work we presented experiments to build machine translation models between spanish and lse glosses. our experiments are based on the id/dl corpus, a small parallel set of spanish sentences aligned with their corresponding lse glosses, about the restricted domain of identity card and driving license renovations. although glosses are not a full representation of all the complexities of a sign language, they are comprehensive enough and suitable for ml purposes. first we carried out experiments on infusing linguistic features on a neural model for trying to improve its performance. the results of these experiments were mixed: on dev, the use of words combined with dependency labels seemed to improve performance, but on test the best improvements were achieved using a combination of words and pos labels. then we took the spanish ancora corpus and transformed it using a rule-based system inspired by the lse grammar to create a synthetic parallel corpus of spanish aligned with lse sequences that is considerably larger than the id/dl corpus. we found that pretraining on this synthetic corpus, and then finetuning with the original id/dl training corpus achieves a marked performance improvement (around 20 points on bleu-char and 15 points bleu-word) over training using only the id/dl training corpus. this improvement could be explained in part due to the high coverage of glosses achieved by using the synthetic data, but we have to take in consideration that the process could have also created some glosses that may have no real-world counterpart in lse. we propose some possible improvements on the process, such as using a heuristic to find appropriate sign glosses when a nonexistent gloss is used. furthermore, given that the use of linguistic information showed some potential improvements in some scenarios, we would like to try combining both methods by getting linguistic information for the synthetic data as well for pretraining. also, as the id/dl corpus we used is rather small, we would like to see to what extent our approach generalises for other lse corpora that belong to other domains. we also want to try our approaches on other pairs of spoken and sign languages. finally, as the dataset is rather small, we could try to use simpler a statistical method, such as phrased-based mt, and combine it with the our rule approach to see if there are also improvements in that scenario.