Modeling Intensification for Sign Language Generation: A Computational Approach (2022)

Abstract: End-to-end sign language generation models do not accurately represent the prosody in sign language. A lack of temporal and spatial variations leads to poor-quality generated presentations that confuse human interpreters. In this paper, we aim to improve the prosody in generated sign languages by modeling intensification in a data-driven manner. We present different strategies grounded in linguistics of sign language that inform how intensity modifiers can be represented in gloss annotations. To employ our strategies, we first annotate a subset of the benchmark PHOENIX-14T, a German Sign Language dataset, with different levels of intensification. We then use a supervised intensity tagger to extend the annotated dataset and obtain labels for the remaining portion of it. This enhanced dataset is then used to train state-of-the-art transformer models for sign language generation. We find that our efforts in intensification modeling yield better results when evaluated with automatic metrics. Human evaluation also indicates a higher preference of the videos generated using our model.

## introduction similar to spoken languages, sign language has rich grammar rules and unique linguistic structures @xcite . elements of prosody such as rhythm, stress, or lengthening play important roles in distinguishing meaning and signaling intensification in sign language (figure 1 ), similar to spoken languages @xcite . thus, it is important for sign language generation (slg) systems to be able to learn accurately from the data and generate presentations that respect prosody. much of the current study on prosodic markers such as intensifiers @xcite 2008; * the first three authors have equal contribution. less clouds very cloudy wolke wolke 10 video frames 17 video frames sign not repeated sign repeated no delay delayed beginning smaller space use larger space use figure 1 : in sign language, modifiers are represented spatially and temporally. here, two signers from phoenix-14t manually sign german "less clouds", and "very cloudy". both of these signs have the same gloss representation: wolke (cloud in german). they are figuratively the same sign, but the duration, repetition, temporal pauses, and continuations determine the exact meaning. this information is lost during sign language translation and evaluation. @xcite are based on linguistic theories of spoken languages and need to be adapted to signed languages, as prosody is represented in the visual modality @xcite . semantic differences are signaled in the visual modality using spatial and temporal presentations such as iconicity, gesture duration, as well as temporal pauses @xcite . such distinctive properties present challenges in slg systems to generate presentations with better prosody. several slg systems have been proposed in recent years motivated by their importance to the deaf and hard of hearing (dhh) communities @xcite @xcite . transformerbased models @xcite have been shown to outperform other neural models @xcite in generating sign language from gloss annotations -a shortened approximation of spoken language that has mappings to signs. one of the key limitations of the state-of-the-art models is that the prosody of the sign videos generated by these models does not change with the semantics of the signs @xcite . in this paper, we take a step toward the goal of modeling prosody in sign language generation by modeling intensification. we refer to intensification as the presence of intensity modifiers that quantify nouns, adjectives or adverbs in a sentence. the intensity modifiers can either be an amplifier (e.g., lot of rain) or a diminisher (e.g., little rain). studies in the linguistics of signed languages show that intensity modifiers change the duration and tactile emphasis in the produced sign @xcite . thus, intensification modeling can impact prosody of generated signs. however, this potential of intensification is not realized within current models because they depend on gloss representation. intensity modifiers are often excluded in gloss representation because they are a sparse approximation of spoken language. as shown in figure 1 , the spatial and temporal properties of signs differ dramatically even when they map to the same gloss. state-ofthe-art models cannot be aware of this temporal and spatial manipulation by modifiers if they are not represented in the gloss training data. our initial analysis of the phoenix-14t @xcite , a german sign language dataset, reveals that 23% of the data has at least one adjective or adverb in the text transcript, but none in the gloss representation. since adjectives and adverbs (e.g., little) often act as intensity modifiers, they are likely to be under-represented in the gloss as well. this observation motivates the need for explicit modeling of intensification in the gloss representation and modifying state-of-the-art models to incorporate this additional information. we hypothesize this to have an overall improvement in the models' performance both quantitatively in terms of automated metrics and qualitatively in terms of human evaluation. to this end, drawing on linguistics and cognitive science studies of sign language, we 1. introduce gloss enhancement strategies grounded in linguistics that respect the role of modifiers with various levels of intensity. ## intensification in sign language gloss annotations in the german sign language weather forecast corpus, phoenix-14t, are simple german words that often do not capture the subtleties of sign language. for example, "very cloudy" and "slightly cloudy" are both represented by a single gloss "wolke" (cloud). our analysis shows that in 23 percent of the data, the gloss representation does not contain any adjectives or adverbs present in the text transcript. since intensity modifiers are usually adjectives/adverbs that quantify intensity of other words, we expect them to be missing from the gloss representation as well. hence, in order for the model to represent intensity modifiers in its latent space, it is necessary to include them in the training data. ## model in this section, we first introduce a baseline model that has been widely adopted for the sign language generation task (section §4.1). to better model the signer's dynamic intensification choices during sign production, we further propose a dynamic selection model (figure 3 ) that makes use of inputs with different intensity modification strategies. ## evaluations and results evaluation of sign language generation is challenging due to the lack of an automatic metric to assess the quality of generated signs. the standard practice @xcite is to translate the poses back to the text domain and compare with ground truth text. this is called back-translation. such automatic evaluation however, cannot accurately capture the quality of the generated signs @xcite . thus, to complement our automatic evaluation, we ask sign language experts to evaluate the generated signs. lastly, we perform a qualitative analysis of the back translated text to i) confirm increased presence of intensity modifiers, ii) identify limitations of our models, and iii) pitfalls of existing metrics. ## discussion and conclusion one limitation of our study is the lack of spatial and temporal context in the automatic back-translation evaluation. the lack of a proper evaluation metric is a problem that needs to be addressed by an orchestrated effort from different fields surrounding the sign language research community. the necessity of more research in related fields is further highlighted by the fact that there are very few publicly available resources for sign language with glosses, limiting our choice and scope of datasets to the phoenix-14t dataset. some corpora exists for american sign language such as how2sign @xcite , but without glosses, it renders certain sign language processing infeasible. another limitation is the cumulative error propagation that dissipates through the intensity classifier, progressive transformer and back-translation, amplifying the total error. there is no dataset or method to do individual error analyses for each part of this pipeline. thus, our error analyses were conducted in an incremental fashion as the errors in later stages of the pipeline depend on earlier errors. despite these limitations, we show that the strategies of intensification, grounded in the linguistics of signed languages, contribute to the improvement of end-to-end sign language generation systems. this modeling effort is supported by our metricbased and human evaluation results. for future work, we plan to further analyze the effects of these strategies on the perception of sign language understanding. we also plan to expand on the intensity modifier paradigm to further research in modeling prosody in sign language.