Syntax-aware Transformers for Neural Machine Translation: The Case of Text to Sign Gloss Translation (2021)

Abstract: It is well-established that the preferred mode of communication of the deaf and hard of hearing (DHH) community are Sign Languages (SLs), but they are considered low resource languages where natural language processing technologies are of concern. In this paper we study the problem of text to SL gloss Machine Translation (MT) using Transformer-based architectures. Despite the significant advances of MT for spoken languages in the recent couple of decades, MT is in its infancy when it comes to SLs. We enrich a Transformer-based architecture aggregating syntactic information extracted from a dependency parser to word-embeddings. We test our model on a well-known dataset showing that the syntax-aware model obtains performance gains in terms of MT evaluation metrics.

## introduction access to information is a human right and crossing language barriers is essential for global information exchange and unobstructed, fair communication. however, we are still far from the goal of making information accessible to all a reality. the world health organisation (who) reports that there are some 466 million people in the world today with disabling hearing loss foot_0 ; moreover, it is estimated that this number will double by 2050. according to the world federation of the deaf (wfd), over 70 million people are deaf and communicate primarily via a sign language (sl). it is well-established that the preferred mode of communication of the deaf and hard of hearing (dhh) community are sls @xcite ), but they are considered extremely low resource languages @xcite , and lag further behind in terms of the provision of language technologies available to dhh people. 150 sls have been classified around the world @xcite while there may be upwards of 400 according to sil international foot_1 . creating accessible-to-all technological solutions may also mitigate the effect of more variable reading literacy rate in the dhh community @xcite . the written language is usually the ambient spoken language in the geographical area signers are found (e.g. english in the british sign language area), and providing resources in native sl could benefit the provision and uptake of sign language technology. machine translation (mt) @xcite is a core technique for reducing language barriers that has advanced, and seen many breakthroughs since it began in the 1950s @xcite , to reach quality levels comparable to humans @xcite . despite the significant advances of mt for spoken languages in the recent couple of decades, mt is in its infancy when it comes to sls. the output of mt between spoken languages tends to be text, but there are further considerations for researchers doing sign language translation (slt). full writing systems exist for sl (e.g. ham-nosys @xcite , sigml @xcite ), but are not always the output or used at all in slt. sl glosses are a lexeme-based representation of signs where classifier predicates, manual and non-manual cues @xcite are distilled into a lexical representation, usually in the ambient spoken language. the articulators in sls include hand configuration and trajectory, facial articulators including lip position and eyebrow configuration, and spatial articulation including eye gaze and body position @xcite ) -all used to convey meaning. glosses, and the text2gloss process, are an essential step in the mt pipeline between spoken and sign languages -even though they are considered a flawed representation which hinder the extraction of meaning by some researchers @xcite . although some current approaches to sl translation follow an endto-end paradigm, translating into glosses offers an intermediate representation which could drive the generation of the actual virtual signs (e.g. by an avatar) @xcite . a growing number of researchers @xcite have been using innovative methods to leverage the limited supply of sl gloss corpora and resources for sl technology. in spite of the impressive results achieved by neural machine translation (nmt) when massive parallel data-sets are available for training using just token level information, recent research (armengol estapé and ruiz costa-jussà, 2021) shows that morphological and syntactic information extracted from linguistic processors can be of help for out-of-domain machine translation or rich morphology languages. in this work, we make transformer models for nmt 'syntax-aware' -where syntactic information embeddings are included as well as word embeddings in the encoder part of the model. the rationale behind including syntactic embeddings draws from the success of word embeddings improving natural language processing tasks including syntactic parsing itself @xcite , and from context-sensitive embeddings pioneered in transformer models @xcite @xcite . we posit that encoding syntactic information will in turn boost the performance of text2gloss as we show with our experimental results. the rest of the paper is organised in the following way: in the next section we briefly introduce the project in the context of which this work is being carried out. then, in section 3, we present related work on sl translation and background on nmt and in section 4 we describe the nmt architecture we use in our experiments. in section 5 we describe the experimental methodology including data and evaluation metrics while in section 6 we present quantitative results. section 7 analyses the results while section 8 closes the paper and discusses further work which could expand this avenue of research. ## the signon project signon foot_2 is a horizon 2020 project which aims to develop a communication service that translates between sign and spoken (in both text and audio modalities) languages and caters for the communication needs between dhh and hearing individuals @xcite . currently, human interpreters are the main medium for sign-to-spoken, spoken-to-sign and sign-to-sign language translation. the availability and cost of these professionals is often a limiting factor in communication between signers and non-signers. the signon communication service will translate between sign and spoken languages, bridging language gaps when professional interpretation is unavailable. a key piece of this project is the server which will host the translation engine, which imposes demanding requirements in terms of latency and efficiency. ## related work the bottleneck to creating sl technology primarily lies in the training data available, such as from existing corpora and lexica. certain corpora may be overly domain-specific @xcite , containing only sentence fragments or example signs as part of a lexicon @xcite , have little variation in individual signers or the framing of the signer in 3d space @xcite , or simply too small in size to be applied to large neural models alone @xcite . the next section describes current methods to mitigate the data-scarcity problem, and state-ofthe-art models and studies with sign language gloss data -including text2gloss, gloss2text, and efforts towards end-to-end (e2e) slt. ## system overview: a syntax-aware transformer for text2gloss our model is an encoder-decoder architecture which consists of augmenting the input embeddings to the encoder via including lexical dependency information. as can be noted from importantly, our text2gloss model has been developed considering the efficiency requirements demanded for the signon project. therefore, the size of the architecture has been selected to produce accurate but also lightweight translations. figure 1 shows the different modules composing our system. the neural architecture employed is based on multi-attention layers @xcite , which has produced excellent results when modelling long input sequences. more specifically, the encoder and decoder are composed by three multiattention layers with four attention heads. the internal dimensions for the fully connected network are set to 1024 and the output units to 512. the encoder transforms inputs to latent vectors, whilst the decoder produces word probabilities from the encoded latent representations. our system augments the discriminative power of the embeddings inputted to the encoder by aggregating syntactic information to word embeddings. unlike (armengol estapé and ruiz costa-jussà, 2021) (which added encoders to manage injected features), we integrate an additional table that contains the vector embeddings for the syntactic tags. the word and syntax embeddings are sum up producing an aggregated embedding that is input to the encoder. both tables were set to have a vector length of 512. to accommodate input text to the neural model, we process it employing subword tokenisation and dependency tags are produced using the model de core news sm available in the spacy foot_4 library. the dependency tags we incorporate are from the tiger dependency bank @xcite , included in the german model, and designed specifically to categorise words in german @xcite . an example of these tags with a german sentence is shown in figure 2 . then, word and syntax tokens were aligned with the corresponding words as shown in figure 1 . for the tokeniser, a sentence piece model @xcite was trained using only the training corpus with a vocabulary size of 3000, keeping some tokens for control. regarding the training, adam optimiser with a learning rate of 10 -5 and a batch size of 64 was applied to optimise cross categorical entropy for 500 epochs. text generation was carried out using figure 2: lexical dependency tree diagram of the sentence "on the weekend it gets a little warmer". key to tags: @xmath0, @xmath1, @xmath2, @xmath3. ## methods & materials in this section, we present the methods and materials used in this research. firstly, we introduce the dataset used and performance metrics and other implementation details are described. ## results here, we present the results from our experiment. as the objective of this research is evaluating the benefits of injecting syntactic information for text2gloss translation, we compare two models with the same architecture: one including, and one not including lexical dependency information. those models are denoted as syntax and no-syntax respectively in this and subsequent sections. ## discussion in the previous section, we have described quantitatively the results produced from our selected metrics. additionally, this section presents a qualitative analysis of the benefits produced for text2gloss translation including lexical information in the transformer model. as can be noted in both examples, the no-syntax model needs more epochs to produce coherent translations and tends to repeat some patterns leading to corrupted outputs in some cases. this effect is quite remarkable in the second example, for which no-syntax retains repeating patterns after 100 epochs while syntax produces more coherent translations. this fact might lead to the no-syntax model obtaining a slightly higher meteor than syntax (see 6.1), while syntax substantially outperformed its competitor in terms of sacrebleu. the fast-learning capacity exhibited by the syntax model could be advantageous for our project, since domain-adaptation is an expected feature for the system under development. also, we have shown that injecting syntactic information to the encoder enables more accurate models without wholesale architecture modifications. the feature injection could be extended to other lexical features, such as part-of-speech tags, via integrating a new embedding table. ## conclusion in this paper we present a syntax-aware transformer for text2gloss. to make the model syntax-aware we inject word dependency tags to augment the discriminative power of embeddings inputted to encoder. the fashion in which we expand transformers to include lexical dependency features involves minor modifications in the neural architecture leading to negligible impact on computational complexity of the model. as the results of this research show, injecting syntax dependencies can boost text2gloss model performances. namely, our syntax-aware model overcame traditional transformers in terms of bleu, ter and rouge-l f1. meanwhile, the meteor metric was slightly worse for our model. furthermore, we have shown that syntax information can also assist in model learning leading to a faster modelling of complex patterns. this preliminary research constitutes a promising starting point to reach the objectives expected for the signon project, in which it is planned to deployed resource-hungry translation models in cloud-based computing servers. further work could compare the impact of other individual, or combinations of, other linguistic features such as part of speech tags which are used in other studies using syntactic tagging for nmt @xcite . it may also use more widely-used lexical dependency tags such as the universal dependencies treebank @xcite . moreover, we are currently exploring data augmentation techniques to expand the scarce availability of sl data. example 1 source und nun die wettervorhersage für morgen samstag den zwölften september (en) and now the weather forecast for tomorrow saturday the twelfth of september target jetzt wetter morgen samstag zwoelf september (en) now weather tomorrow saturday twelve september syntax 5 jetzt wetter wetter (en) now weather weather 50 jetzt wetter wie-aussehen morgen samstag fuenfte mai (en) now weather look tomorrow saturday fifth may 100 jetzt wetter wie-aussehen morgen samstag zwoelf september (en) now weather look tomorrow saturday twelve september 150 jetzt wetter wie-aussehen morgen samstag zwoelf september (en) now weather look tomorrow saturday twelve september no-syntax 5 jetzt wetter wie wie wie-ausse...ausseauss (en) now weather how how ausse...ausseauss 50 jetzt wetter wie-aussehen morgen samstag fuenfzehn september (en) now weather look tomorrow saturday fifteen september 100 jetzt morgen wetter wie-aussehen samstag zwoelf september (en) now tomorrow weather look saturday twelve september 150 jetzt morgen wetter wie-aussehen samstag zwoelf september (en) now tomorrow weather look saturday twelve september example 2 source vom nordmeer zieht ein kräftiges tief heran und bringt uns ab den morgenstunden heftige schneefälle zum teil auch gefrierenden regen (en) from the north sea, a strong deep pulls up and brings us violent snowfalls from the morning hours, sometimes freezing rain target kraeftig ab morgen frueh meistens schnee schneien kalt regen (en) skimpy from tomorrow early mostly snow snow cold rain syntax 5 kommen region kommen (en) come region come 50 tief kommen morgen kommen regen kommen regen kommen (en) deep come tomorrow come rain come rain come 100 tief kommen regen kommen mitte berg kommen (en) now weather look tomorrow saturday twelve september 150 jetzt in-kommend tief kommen regen kommen mild (en) now in-coming deep come rain come mild no-syntax 5 region kommen region kommen regen (en) region come region come rain 50 morgen kommen tief kommen regen kommen regen kommen regen kommen regen kommen (en) tomorrow come deep come rain come rain come rain come rain come 100 tmorgen regen tief kommen region kommen regen koennen schnee regen gefrieren glatt gefahr glatt gefahr (en) tomorrow rain deep come region come rain can snow rain freeze smooth danger smooth danger 150 morgen meistens schnee regen glatt region kommen regen gefahr glatt regen gefahr glatt regen gefahr (en) tomorrow mostly snow rain smooth region come rain danger smooth rain danger smooth rain danger