Bootstrapping Pre-trained Word Embedding Models for Sign Language Gloss Translation (2024)

Abstract: This paper explores a novel method to modify existing pre-trained word embedding models of spoken languages for Sign Language glosses. These newly-generated embeddings are described, visualised, and then used in the encoder and/or decoder of models for the Text2Gloss and Gloss2Text task of machine translation. In two translation settings (one including data augmentation-based pre-training and a baseline), we find that bootstrapped word embeddings for glosses improve translation across four Signed/spoken language pairs. Many improvements are statistically significant, including those where the bootstrapped gloss embedding models are used.Languages included: American Sign Language, Finnish Sign Language, Spanish Sign Language, Sign Language of The Netherlands.

## introduction there has been a surge in research interest on sign language machine translation (slmt) in recent years, but the data scarcity problem @xcite and lack of standardised annotated data @xcite remain substantial obstacles to overcome. at the heart of the labelling problem is the fact that although writing and transcription systems exist for sls @xcite , none are used dayto-day by signers. glosses are a semantic labelling @xcite . this work focuses on translation between text and glosses. tool for signs. they typically use lexemes from the ambient spoken language of the hearing community where the sl is used in order to convey the semantic sense of a given sign. however, glosses cannot be considered an orthographic system for sls as they often differ between datasets, are not used by signers to write their languages @xcite , and may not include linguistic phenomena which are crucial to understand an utterance @xcite . slmt is inherently multimodal @xcite , and it is helpful to conceptualise it as a constellation of sub-tasks at the interface of nlp and computer vision. end-to-end slmt between sl video and text in a spoken language exists, but performs poorly compared to translation broken down into intermediate steps where signs are represented by some orthographic form (e.g. in glosses @xcite , or sl notation system @xcite ) -except for restricted domains and datasets (e.g. @xcite @xcite ). these subtasks are neatly shown in a diagram from @xcite overview of the field in figure 1 . even though both text-to-sl gloss (text2gloss) and sl gloss-to-text (gloss2text) are sequenceto-sequence tasks using machine-readable text, the amount of parallel data available for any sl is orders of magnitude smaller than equivalent pairs of spoken languages. according to @xcite , the largest parallel corpus between sl glosses and text available to researchers foot_0 contains 21,000 parallel utterances @xcite . it is reasonable to refer to all sls as extremely low resource languages @xcite , and therefore data augmentation approaches must be adopted in order to improve the performance of translation models which include them. in this paper, a novel method to generate semantic representations for sign language (sl) glosses is described. they are created by bootstrapping pre-trained word embedding models from spoken languages which already exist and their use is demonstrated in multilingual text2gloss and gloss2text machine translation experimental settings. this paper is structured as follows: in section 2, previous work where linguistic information is used to supplement gloss representations and its impact on slmt is described, as well as work to create computational semantic resources for sls in general. section 3 sets out the process to generate sl gloss embeddings from pre-trained word embeddings, before section 4 demonstrates their use in translation experiments. findings from these experiments are described in section 5 and discussed in sections 6 and 7, along with potential future research directions using these embedding representations. ## background one way of mitigating the semantic bottleneck created by gloss representation of signs is to explore techniques for low-resource neural machine translation @xcite . these include data augmentation methods involving linguistic features (armengol estapé and ruiz costa-jussà, 2021) as well as techniques specifically designed for text2gloss translation @xcite . zhu and colleagues' (2023) comprehensive study of these methods found, for dgs foot_1 corpora, that: (1) a combination of data augmentation strategies, and (2) transfer learning foot_2 are viable methods to improve translation performance for text2gloss. they also highlight that it is important to ensure that these findings are generalisable to other sls so further investigation such as the present study is required. other studies focused on injecting linguistic features into the embedding table for text2gloss and gloss2text @xcite , and found that transfer learning was again beneficial, as well as using linguistic features such as part-of-speech (pos) and syntactic dependency tags -including pos tags for sl glosses @xcite . it may also be beneficial to use semantic information about signs into translation models, instead of (or as well as) using syntactic or grammatical information. no previous study with a parallel methodology was found, but other studies do use embeddings as part of slmt models. walsh and colleagues (2022) use sentence-level word2vec @xcite or bert @xcite embeddings to support text2notation (in hamnosys @xcite ) translation. other studies use visual embeddings to support joint sign2gloss2text @xcite , sl recognition @xcite , or to encode phonological information for isolated sl recognition @xcite . this paper investigates using semantic information about words and glosses as a transfer learning strategy, and also its performance in combination with the syntax-based data augmentation methods seen in previous works. ## sign language gloss embeddings the motivation behind the present methodology is the proposition that, by mapping id-glosses and their equivalent senses from a signbank, it is possible to alter the weights of a pre-trained word embedding model from a spoken language in order to simulate the semantic interactions between signs in a given sl. as such, each gloss in a given sl is mapped to pre-trained embedding weights in one of three ways, along with some examples from the sls in this study: the rest of the weights in the original pretrained word embedding models remain the same if there is no gloss with the same label and are retained in the model to allow for the mapping of out-of-vocabulary token mapping. ## translation experiments for each language pair, we perform both text2gloss and gloss2text experiments in two settings. firstly, a baseline (section 4.1) with each parallel corpus for each language. then, following previous similar experimental setups @xcite @xcite , a warm start transfer learning approach (section 4.2) is executed. in other words, first a translation model is pretrained with a larger silver corpus and shared silver and gold vocabulary, and then finetuned on the same parallel data as the baseline. translation experiments are performed using opennmt-py 3.4.2 @xcite . open-nmt is an open source translation toolkit which is based on lstm encoder-decoder model with attention. all other running parameters are set to default, unless stated in appendix a. ## results and analysis for @xmath0, the baseline models with any kind of embeddings improved over the baseline in chrf and meteor, but only pt+ftboth performed better on the bleu metric and this difference was not significant (@xmath1.25, @xmath2. all pt+ft conditions had significantly higher bleu scores than the baseline. within the pt+ft experimental setups, all metrics were markedly higher in the embedding setups, and pt+ft-enc (@xmath3.03, @xmath4.03, @xmath5. for @xmath6, pt+ft tends to be a better strategy with pt+ft-enc being the only setup which performs significantly better than the baseline (@xmath7.03, @xmath8, and higher scores in both metrics. the fact that pt+ft-both performs significantly better than pt+ft in text2gloss, and pt+ftenc than baseline in gloss2text, is particularly promising as these conditions include the bootstrapped word embedding models for lse. for @xmath9, using embeddings improves bleu scores in all setups, but only pt+ft-dec (with ngt bootstrapped glosses) in meteor as well as being the only significant improvement on bleu (@xmath10.01, @xmath11. the results are the mirror image in pt+ft: all setups except pt+ft-dec significantly improve over the baseline, and pt+ft-enc with only dutch word2vec embeddings improves over pt+ft (@xmath12.05, @xmath13. however, in @xmath14, as well as pretraining and finetuning, seems to damage the performance of this translation direction. across both of these language pairs, compared to the bleu scores the meteor scores are also quite weak (compare lse and finsl results). in this language pair in particular, for @xmath15, the use of word2vec embeddings improves performance on most settings on both metrics. in the pt+ft setting, encoder english embeddings and both english and asl embeddings improve significantly over the baseline (pt+ft-both: @xmath16.00, @xmath17. for @xmath18, over the baseline, significant improvements are seen when asl embeddings are used in the encoder to support glosses: pt+ft-enc (@xmath19.02, @xmath20, and pt+ft-both: (@xmath21.01, @xmath22. it is therefore reasonable to infer that: (a) richer semantic representations for asl, and (b) warm-start transfer learning on a larger silver dataset and joint vocabulary provides a real boost to translation performance and generalisibilty. it may be the case that there are no marked improvements within pt+ft as the en/asl test partition is the smallest between all four language pairs (see in the asl/en language pair in general, the me-teor scores are much stronger compared to the others in this study. perhaps a more even total of unique tokens in both asl and english lexica contributes to this. as for @xmath23, pt+ft with silver data provides an improvement in metrics across the board. significant improvements are only seen in gloss2text: finsl embeddings significantly improve in baseline-enc over the baseline (@xmath24.02, @xmath25, and in pt+ft-enc over pt+ft (@xmath26.04, @xmath27, but curiously not over the baseline which has a very low bleu @xmath28. the results for finsl/fi may be considered rather unusual on the whole. it is possible that the very low vocabulary size of the finsl gold utterances (@xmath29, the disparity foot_15 between this and the finnish vocabulary size (@xmath30, the replacement of all pronouns with just two lexemes (see section 4.3), and the low token coverage of the finnish word2vec model of finnish tokens (24% in both the gold and silver+gold datasets) contribute to these results. besides this, the finsl dataset contains a few signs corresponding to descriptive markers @xcite (e.g. " kvkk" for 'whole object' and " kvmk" for 'shape and size') that are frequent (around 19% of the total signs in the training set). these signs are not lexical and have no corresponding ambient language lexemes, so an "unknown" random embedding was assigned to them. these tokens' high frequency may also explain the low performance of the finsl↔fi experiments. in the future, we want to explore the possibility of creating embeddings for these markers using the average embedding of their corresponding finnish descriptions. ## discussion and limitations so far, this exploratory work shows that using semantic representations tailored to sls (in this case word2vec embeddings adapted to particular sl settings) is a promising avenue of research. overall, the results present a positive outlook concerning the effectiveness of including bootstrapped word embedding models in the encoder and/or decoder of opennmt for text2gloss and gloss2text translation. in all pt+ft-* settings, the use of embeddings improved translation performance in at least one setting. this is also true in the baseline setting, apart from with @xmath31. many of these improvements were significant, and those where pt+ft-embedding significantly improved against pt+ft are particularly notable. however, it is necessary to examine more data augmentation methods, types/sizes of word embedding models, sub-word tokenisation, and techniques to adapt semantic representations for the extremely low-resource setting of sl processing. it may also be worthwhile to attempt this approach on low-resource pairs of spoken languages, especially those with little or no written data @xcite as anchor word embeddings already exist for spoken languages @xcite . other practical tasks involving word embedding model support may include the tagging and parsing of sl gloss data @xcite @xcite . besides the use of opennmt for experiments, trying alternative open source translation toolkits such as marianmt @xcite for lse). pretrained models like mbart @xcite could also be a fruitful direction of research. some preliminary experiments following egea @xcite were also attempted, using a mbart translation approach for lse↔spanish. however, some issues were found when applying the present method to mbart: firstly, the model uses sentencepiece tokenisation, while this study's embeddings are created with simple whitespace tokenisation. furthermore, the mbart model expects a unified embedding space between source and target languages, which could skew the results for glosses that have the same surface form as ambient language words. it is possible to overcome these limitations, but given time and resource constraints the mbart experiments remain out of the scope of this work, and it is planned to explore them further in the future. it would also be rewarding to explore other lexical sl resources such as signpuddle 21 which has been used in work on text2notation @xcite translation work. in addition, when sign-nets @xcite are further developed and contain rich metalinguistic information for many sls, these will be a crucial resource for further studies in this area. some researchers may disagree with the use of glosses as a representation in sl processing 21 https://www.signbank.org/signpuddle/ altogether, and disprefer splitting slmt into a pipeline of intermediate tasks instead of treating it as an end-to-end task @xcite . this is a valid position, and other work involving semantic representations in, for example, video2text could be complementary to studies like the present one. recent innovations into data-intensive methods such as 0-shot mt and nlp tasks often exclude sls, because even though messy, unorganised, and seemingly irrelevant text data can be used for tasks in many spoken languages, this is not necessarily the case for the multimodal nature of sls @xcite . however, recent research into true 0-shot translation; using llms to read and interpret reference material about the grammar of a language @xcite -may aid slmt and sl processing beyond that. the large amount of experimental settings and limited computing resources available also meant that it was not possible to complete all of the evaluation that was initially planned. for example, from the insights gleaned from @xmath32, it would be interesting to quantitatively investigate the connection between word embedding model's vocab coverage and model performance. qualitative analysis, though present, was unfortunately minimal and not formal and it would be greatly beneficial to expand it. ## concluding remarks this study cast a wide net in order to devise novel methods to create semantic representations for sl glosses, and test their effectiveness when being used in slmt. these experiments showed mixed but overall positive results, whereby bootstrapped pre-trained word embeddings from a spoken language can be modified with the present methodology in order to represent the semantic relations between sl glosses. it also provides further evidence that pretraining on silver data is effective across language pairs. future work will benefit from further experimentation with the methods undertaken to generate vector representation for signs whether represented by gloss, sl notation system, pose, or video frame. these embedding representations sit at the interface of nlp and computer vision-based approaches to slmt, and characterise the need to follow both avenues of this field of research in a complimentary manner.