Part-of-Speech tagging S panish S ign L anguage data and its applications in Sign Language machine translation (2023)

Abstract: This paper examines the use of manually part-of-speech tagged sign language gloss data in the Text2Gloss and Gloss2Text translation tasks, as well as running an LSTM-based sequence labelling model on the same glosses for automatic part-of-speech tagging. We find that a combination of tag-enhanced glosses and pretraining the neural model positively impacts performance in the translation tasks. The results of the tagging task are limited, but provide a methodological framework for further research into tagging sign language gloss data.

## introduction lengua de signos española (lse) 1 is a sign language (sl) of spain with an estimated 45-75, 000 signers @xcite in that country. creating sl technology is an inherently multimodal task @xcite as sls are produced in the visual-spatial modality @xcite . it is also a challenging as sls lack a commonly-used written representation @xcite . instead, researchers rely on nonstandardised glosses which may be considered a suboptimal representation of the rich semantics of signs @xcite . however, while there is not enough data available to build large machine learning (ml)-based end-toend (e2e) models, they are a valuable tool and can be enriched with linguistic information posthoc by researchers @xcite . compared to spoken languages, tools and techniques for natural language processing (nlp) in sign language glosses are less established. this is particularly true with regards to linguistic processing techniques @xcite such as tag-1 spanish sign language ging and parsing. this paper focuses on the partof-speech (pos) tagging of lse, its application in linguistically-informed sl machine translation (slt), and the next steps in linguistic processing for sls. it also uses data from the isignos @xcite corpus, and introduces a method to linearise and lexicalise the glosses found there. in section 2 we describe previous relevant work on lse, and information on linguistic tagging for sls. next, section 3 describes the pre-processing of gloss data, before section 4 proposes a novel approach to incorporate manual pos tags into neural machine translation models -as well as another experiment training a naïve pos-tagger on these glosses. this section, and section 5 provide insight into experimental results, and their implication in the wider field. ## previous work on lse there exists only a small body of suitable lse corpora and parallel resources for use in nlp tasks, which means this sl could be considered an extremely low resource language @xcite . despite this scarcity, researchers have adopted strategies to mitigate this limitation when building processing and translation tools for lse. for example, @xcite explored two approaches for @xmath0. researchers and experts crafted 154 rules to transform spanish text into lse glosses, and also built a completely statistical translation model. the linguistically-informed rule-based model outperformed the statistical model on a small, domain-restricted dataset 2 . this corpus is labelled with lexeme-based glosses, which resemble words in the ambient spoken language (spanish) and are similar to representations found in sl dictionaries and signbanks -for example dilse @xcite . in contrast, work by @xcite uses glosses from a storytelling corpus which contain grammatical and phonological information such as handshape and mouthing. later work @xcite also took a hybrid approach for text2gloss (t2g) and gloss2text (g2t) transformation between spanish and lse, using a rule-based synthetic data to pretrain a neural translation model. ## future work this study uncovers some interesting findings, and provides the means of generating a sizeable parallel resource for spanish-lse, as well as english for most of the isignos videos. by way of future research directions, it would be interesting to test this method on the id/dl corpus -or non-lse sl datasets -in order to make these results comparable with @xcite and @xcite . in addition, it would be desirable to run other pos taggers on this dataset for comparability such as the more recent roberta 0-shot tagger @xcite or the hmm-based apertium pos tagger @xcite which is tailored towards low-resource languages by means of the baum-welch algorithm. in order to robustly confirm differences between experimental setups, future studies would benefit from statistical significance testing as advocated in @xcite . more generally, further work on grammatical parsing in sls would be beneficial. there is debate as to whether lse has pos in the traditional sense @xcite , and that these categories are more flexible across sls. regardless, further studies are vital to increasing the volume of resources available, and to answer the call for more computational linguistic based resources for these vibrant languages.