Linguistically Informed Transformers for Text to A merican S ign L anguage Translation (2024)

Abstract: In this paper we propose a framework for automatic translation of English text to American Sign Language (ASL) which leverages a linguistically informed transformer model to translate English sentences into ASL gloss sequences. These glosses are then associated with respective ASL videos, effectively representing English text in ASL. To facilitate experimentation, we create an English-ASL parallel dataset on banking domain.Our preliminary results demonstrated that the linguistically informed transformer model achieves a 97.83% ROUGE-L score for text-to-gloss translation on the ASLG-PC12 dataset. Furthermore, fine-tuning the transformer model on the banking domain dataset yields an 89.47% ROUGE-L score when fine-tuned on ASLG-PC12 + banking domain dataset. These results demonstrate the effectiveness of the linguistically informed model for both general and domain-specific translations. To facilitate parallel dataset generation in banking-domain, we choose ASL despite having limited benchmarks and data corpus compared to some of the other sign languages.

## introduction sign languages (sl) are the primary means of communications for the deaf community. it is a non-verbal form of communication where deaf individuals use their hands, arms, and facial expressions to share thoughts and ideas. unlike spoken languages that rely on sound, asl employs gestures. recent linguistic studies have confirmed that sls, like other spoken languages, is a complete natural language with its own syntactical structures and intricate morphological and phonological properties. this complexity includes both sequential and simultaneous affixation of manual and nonmanual elements in its structure. a challenging aspect of sign language translation (slt) is that sign languages (sls) are multichanneled and do not have a written form, as noted by @xcite . consequently, advancements in text-based machine translation (mt) cannot be directly applied to sls. historically, researchers have used written representations of sls to facilitate translation. one common method involves using glosses, which are labels in the spoken language that correspond to sign language components, often including affixes and markers. these glosses act as an intermediary step in developing mt systems that translate between sls and spoken languages (noted by @xcite ; @xcite ), and vice versa ( @xcite ; @xcite ). other notable works include @xcite that proposed an approach using neural machine translation (nmt) and motion graphs to generate sign language videos for a given text. @xcite have proposed a method of converting the text to sign language glosses, extracting the poses for each gloss and translating the poses to a video. in an earlier attempt, @xcite have proposed a method translating text to indian sign language (isl) using lexical functional grammar while @xcite talks about generating animated avatar using ham-nosys for text to sl translation. for translations from spoken languages to sls, glosses are used to build the system in two phases: translating text to glosses, and then converting these glosses into video (see figure 1 ). these glosses are then input into systems that generate sl content, such as avatar animations or auto-encoder based video generators. our present research specifically targets the text-to-gloss translation phase, which is crucial for producing accurate sign language animations. however, despite improvements in this area, significant breakthroughs remain elusive, as indicated by @xcite . in this paper we propose a framework for automatic translation of english text to asl. the key contributions and results of this work are as follows: the above results show a significant improvement from the baseline gru-b model. aslg-pc12 @xcite , the largest text-to-asl gloss dataset, offers 87,710 samples but pales in comparison to mainstream language pairs like english to french. its focus on news and politics limits its applicability across domains, necessitating domain-specific models, increasing data scarcity challenges. ## sign language (sl) linguistic issues sign languages (sl) are visual-spatial natural languages, utilizing manual and non-manual components for linguistic communication @xcite . manual components include hand shape, orientation, position, and movement, while nonmanual components consist of facial expressions, eye gaze, and body posture. signers utilize a three-dimensional signing space segmented into 27 cubical regions @xcite . each sign formation adheres to complex constraints akin to spoken languages, with sl morphology primarily derivational. the closed lexical class in sl encompasses classifier hand shapes, discourse markers, and non-manual signs @xcite . classifier hand shapes offer specific hand configurations representing referent characteristics, as shown in figure 2 . american sign language (asl) is known to follow a topic-comment structure. this structure positions the main subject or theme (the topic) at the sentence's outset, followed by more specific information (the comment) @xcite . by establishing context early in the sentence, asl users efficiently convey complex ideas (figure 3 ). one important aspect of asl's topic-comment structure is the flexibility in word ordering. while the default order is topic-comment, asl allows for variations based on emphasis and context. for instance, a speaker might choose to emphasize a particular aspect of the comment by placing it before the topic. this flexibility adds nuance and richness to asl communication, enabling speakers to convey subtle meanings and emotions effectively @xcite . the flexibility in the structure is the reason why a simple rule-based approach is not possible for text to asl gloss translation. ## the linguistically informed the existing nmt models excel at capturing intricate data patterns without requiring manual feature engineering, offering end-to-end solutions. however, they often overlook latent linguistic traits crucial for extracting pertinent information. to address this, we propose a transformer-based architecture that integrates word embeddings from the encoder part with diverse linguistic features inherent in text, enhancing automatic text-to-asl gloss translation. ## experiments dataset: the aslg-pc12 corpus @xcite : consists of 87,710 bilingual sentences. it contains 1,027,100 english words and 906,477 gloss words, along with 4,662 english word singletons and 6,561 gloss word singletons. the vocabulary for both sign gloss annotation and spoken language comprises 16,788 and 12,344 terms, respectively. asl-bank dataset: considering the specificity of the terminology used in banking contexts, we have also built a set of 3597 text-asl gloss pairs through domain experts. the collected phrases are sourced from banking-related texts and provided to american sign language (asl) experts for manual translation into asl gloss. refer appendix d for data statistics and appendix e for sample data. fine-tuning: we divided the aslg-pc12 corpus into 52,626 sentences for training, 17,542 sentences for validation, and 17,542 sentences for testing @xcite and use it to fine-tune a t5-small, t5-base and bart-base model on a100 gpu for 50 epochs (experiment a). for the asl-bank dataset, we used 3,166 sentences for training, 395 sentences for validation and 396 sentences for testing. we fine-tuned the three transformer models from experiment a on a100 gpu for 50 epochs. refer appendix a and b for more details. evaluation: apart from using the standard mt evaluation parameters like, rouge-l @xcite and blue @xcite scores we also advocate using a modified bertscore @xcite as performance metrics. as the bert models are trained on natural english text, we cannot rely on the sentence embeddings it gives for the asl gloss sequences for the reasons explained in introduction. hence, we proposed to get the word embeddings of each gloss present in the asl gloss sequence and aggregate them to get the sentence embedding of the asl gloss sequence which can be further used to calculate the cosine similarity score. ## results the results are reported by first comparing the model performance upon fine-tuning on aslg-pc12 @xcite dataset between our models of choice t5-small, t5-base @xcite and bart-base @xcite and a gru based model from @xcite ( ## conclusion in this paper, we present a linguistically informed transformer architecture towards automatic translation of english text to american sign language. the proposed model not only aims at addressing the poor generalization capability of traditional structured prediction models but also exploit the linguistic characteristics present within a text to improve the performance of the translation. we evaluate the performance of the proposed model with respect to a popular baseline model from @xcite . we observed that the proposed transformer based model along with an additional linguistic information performs much better than existing baseline system. limitations 1. this work specifically focuses on text to asl gloss translation. hence, the fine-tuned models cannot be used for generating glosses in other sign languages like indian sign language or british sign language due to differences in structure. the linguistic embeddings which are gcn's output hidden states are combined with the last hidden state of the encoder part as described in section 3.3 during both the fine-tuning processes. in that way, the gcn is trained along with the transformer model.