Including Facial Expressions in Contextual Embeddings for Sign Language Generation (2023)

Abstract: State-of-the-art sign language generation frameworks lack expressivity and naturalness which is the result of only focusing manual signs, neglecting the affective, grammatical and semantic functions of facial expressions. The purpose of this work is to augment semantic representation of sign language through grounding facial expressions. We study the effect of modeling the relationship between text, gloss, and facial expressions on the performance of the sign generation systems. In particular, we propose a Dual Encoder Transformer able to generate manual signs as well as facial expressions by capturing the similarities and differences found in text and sign gloss annotation. We take into consideration the role of facial muscle activity to express intensities of manual signs by being the first to employ facial action units in sign language generation. We perform a series of experiments showing that our proposed model improves the quality of automatically generated sign language.

## introduction communication between the deaf and hard of hearing (dhh) people and hearing non-signing people may be facilitated by emerging language technologies. dhh individuals are medically underserved worldwide @xcite due to the lack of doctors who can understand and use sign language. also, educational resources that are available in sign language are limited especially in stem fields @xcite . although the americans with disabilities act (united states department of justice, 2010) requires government services, public accommodations, and commercial facilities to communicate effectively with dhh individuals, the reality is far from ideal. sign language interpreters are not always available, and communicating through text is not always feasible as written languages are completely different from signed languages. in contrast to sign language recognition (slr) which has been studied for several decades @xcite in the computer vision community @xcite , sign language generation (slg) is a more recent and less explored research topic @xcite @xcite . missing a rich, grounded semantic representation, the existing slg frameworks are far from generating understandable and natural sign language. sign languages use spatiotemporal modalities and encode semantic information in manual signs and facial expressions. a major focus in slg has been put on manual signs, neglecting the affective, grammatical, and semantic roles of facial expressions. in this work, we bring insights from computational linguistics to study the role of and include facial expressions in automated slg. apart from using facial landmarks encoding the contours of the face, eyes, nose, and mouth, we are the first to explore using facial action units (aus) to learn semantic spaces or representations for sign language generation. in addition, with insights from multimodal transformer architecture design, we present a novel application of the dual encoder transformer model for slg, which takes as input spoken text and glosses, computes the correlation between both inputs and generates skeleton poses with facial landmarks and facial aus. previous work used either gloss or text to generate sign language or used text-to-gloss (t2g) prediction as an intermediary step @xcite . our model architecture, on the other hand, allows us to capture information otherwise lost when using gloss only and captures differences between text and gloss, which is especially useful for highlighting adjectives otherwise lost in gloss annotation. we perform several experiments using the phoenix14-t weather forecast dataset and show that our model performs better than baseline models using only gloss or text. in summary, our main contributions are the following: more than 70 million deaf and hard of hearing worldwide use one of 300 existing sign languages as their primary language @xcite . in this section, we explain the linguistic characteristics of sign languages, the importance of facial expressions to convey meaning, and elaborate on prior work in slg. ## sign language dataset in this work, we use the publicly available phoenix14t dataset @xcite , fre-quently used as a benchmark dataset for slr and slg tasks. the dataset comprises a collection of weather forecast videos in german sign language (dgs), segmented into sentences and accompanied by german transcripts from the news anchor and sign-gloss annotations. phoenix14t contains videos of 9 different signers with 1066 different sign glosses and 2887 different german words. the video resolution is 210 by 260 pixels per frame and 30 frames per second. the dataset is partitioned into training, validation, and test sets with respectively 7,096, 519, and 642 sentences. ## methods: dual encoder transformer for sign language generation in this section, we present our proposed model, the dual encoder transformer for sign language generation. given the loss of information that occurs when translating from text-to-gloss, our novel architecture takes into account the information from text and gloss as well as their similarities and differences to generate sign language in the form of skeleton poses and facial landmarks shown in figure 3 . for that purpose, we learn the conditional probability @xmath0, z) of producing a sequence of signs @xmath1, . . . , y t ) with t frames, given the text of a spoken language sentence x @xmath2, . . . , x n ) with n words and the corresponding glosses @xmath3, . . . , z u ) with u glosses. our work is inspired by the progressive transformer @xcite , which allows translation from a symbolic representation (words or glosses) to a continuous domain (joint and face landmark coordinates) by employing positional encoding to permit the processing of inputs with varied lengths. in contrast to the progressive transformer, which uses one encoder to use either text or glosses to generate skeleton poses, we employ two encoders, one for text and one for glosses, to capture information from both sources and create a combined representation from the encoder outputs to represent correlations between text and glosses. in the following, we will describe the different components of the dual-encoder transformer. ## discussion and conclusion in this work, for the first time, we attempt to augment contextual embeddings for sign language by learning a joint meaning representation that includes fine-grained facial expressions. our results show that the proposed semantic representation is richer and linguistically grounded. although our proposed model helped bridge the loss of information by taking into account text, gloss, and their similarities and differences, there are still several challenges to be tackled by a multidisciplinary scientific community. complex hand shapes with pointing fingers are very challenging to generate. the first step to improving the generation of the fingers is in improving methods to recognize finger movements more accurately. similarly, we need tools that are more robust in detecting facial expressions even in situations of occlusion. we also realize that slg models are overfitting specific sign languages instead of learning generalized representations of signs. we chose to work with a german sign language since that is the only dataset with gloss annotation that could help us study our hypotheses. the how2sign dataset @xcite is a feasible dataset for asl, but it does not allow any model example 1 example 2 tg2s (ours) tg2s (ours) ground truth ground truth figure 5: examples in which our model failed to generate the correct phonology of signs. example 1 depicts inaccuracies in hand shape, orientation, and movement. example 2 shows the difficulty of the model to capture pointing hand shapes. to extract facial landmarks, facial action units, or facial expressions from the original video frames since the faces are blurred. in the future, we hope to see new datasets with better and more diverse annotations for different sign languages that would allow the design of a natural and usable sign language generation system.