Reconsidering Sentence-Level Sign Language Translation (2024)

Abstract: Historically, sign language machine translation has been posed as a sentence-level task: datasets consisting of continuous narratives are chopped up and presented to the model as isolated clips. In this work, we explore the limitations of this task framing. First, we survey a number of linguistic phenomena in sign languages that depend on discourse-level context. Then as a case study, we perform the first human baseline for sign language translation that actually substitutes a human into the machine learning task framing, rather than provide the human with the entire document as context. This human baseline—for ASL to English translation on the How2Sign dataset—shows that for 33% of sentences in our sample, our fluent Deaf signer annotators were only able to understand key parts of the clip in light of additional discourse-level context. These results underscore the importance of understanding and sanity checking examples when adapting machine learning to new domains.

## introduction one of the key challenges in sign language processing is that methods from mainstream natural language processing (nlp) are tailored primarily to text and secondarily to speech. much of the work in this space therefore focuses on generalizing these methods to video, in order to capture this oft-neglected dimension of linguistic diversity @xcite . one such carryover is that sign language machine translation (mt) is framed as a sentence-level task. although continuous sign language datasets are usually derived from long-form signed content (e.g., interpreted news broadcasts), they are preprocessed into short clips associated with each sentence in the spoken language transcript (which may not themselves correspond to discrete sentences in the continuously translated sign language version), and models are trained and evaluated on these clips in isolation. in this work, we examine the limitations of this task framing, which-like many other sign language modeling decisions @xcite -was adopted somewhat uncritically, and ask: what is the right unit of translation for sign language? machine translation between spoken languages is typically posed as a sentence-level task, and although it largely works, there are known intersentential dependencies like anaphora that are impossible to resolve in isolation @xcite . these dependencies are especially troublesome for language pairs that have mismatches in grammatical features like pronoun dropping, tense marking, or gradations of register. the situation is perhaps even more pronounced for translation between spoken languages and sign languages. sign languages are not just spoken languages produced with the hands: the grammar of sign languages is shaped by the nature of the visualspatial modality @xcite . while utterances produced by non-native signers tend to resemble the syntax of the region's spoken language, native signing often expresses concepts in a fundamentally different way that is richly grounded in spatial world understanding and, more importantly here, the discourse context. when deprived of that context, the viewer may catastrophically fail to understand the meaning of an utterance and therefore be unable to translate it. we describe some linguistic phenomena relevant to cross-modal translation in section 3. to the best of our knowledge, no sign language mt benchmarks provide baselines for human performance that actually ask humans to perform the same task that they expect of the model. reference translations are given in the dataset by construction, either as the source text or by discourse-level trans-lation. human judgments are used at the discourse level to quality-check preprocessing or to evaluate model-generated outputs, but not to sanity check the task framing itself. we therefore provide in section 4 the first such human baseline, for american sign language (asl) to english translation on the how2sign dataset @xcite , as a case study. how2sign consists of informal instructional ("how to") narratives, which is a particularly illustrative domain. before even scoring results against ground truth references, we find that for 33.3% of instances in our sample, our fluent deaf signer annotators felt that they could not fully perform the translation given only the sentence-level clip-but could, given additional discourse-level context. most of these errors were due to features of sign languages that lack direct analogues in spoken languages. when we do compute metrics, we get a surprisingly low score of 19.8 bleu (56.6 bleurt) for the sentence-level task, which increases with additional context but only to 21.5 (59.5). we disaggregate these results for each of five distinct interpreters in the how2sign test set, and find that sentence-level results vary from 5.2 bleu (45.7 bleurt) to 39.5 (70.0) across individuals. scores are higher for interpreters who stay closer to english; context is more important for those who don't. we hope that these results and analysis will encourage the sign language mt field to reconsider whether computational benefits of the sentencelevel task framing outweigh its quality and alignment limitations, and to continue to pare back unfounded modeling assumptions by understanding datasets more deeply and crafting benchmarks more deliberately. ## long-range linguistic dependencies in this section, we outline a number of long-range dependencies in the grammar of sign languages, primarily asl, which may be truncated with sentencelevel clipping. these features are not necessarily universal to all sign languages, but they are relatively common insofar as they are motivated by the visual-spatial modality @xcite . we create example figures using clips from the how2sign dataset @xcite ; we omit the signers' faces in the figures for privacy but note that facial expressions and mouthing are important in sign language. ## case study in order to explore how these phenomena surface in real sign language translation datasets, we perform a human baseline for asl to english translation on how2sign @xcite across different amounts of provided context. to the best of our knowledge, this is the first time human performance has been measured for the sentence-level sign language machine translation task. how2sign was constructed by having 11 signers-5 hearing, 4 deaf, and 2 hard of hearing-watch english-captioned instructional "how to" videos from the earlier how2 dataset @xcite a first time to understand the content, then a second time at 0.75x speed while performing a live interpretation. the captions (from the original speech track) were manually realigned to the signing, with an average sentence duration of 8.67 seconds. ## conclusion in this paper, we argued that the costs of the sentence-level sign language mt task framing are higher than many might expect, with many relevant discourse-level phenomena being related to the visual-spatial modality and cross-modal translation. we supported this with a case study: the first human baseline for sentence-level sign language mt, from asl to english on the how2sign dataset. we found that discourse context was necessary to fully understand and translate a large fraction of sentences (33.3%), and this effect is itself attenuated by the prevalence of signing data that does not represent the more challenging aspects of asl due to its use of non-native or live interpreters. we hope that this inspires more in-depth analysis grounded in firsthand experience with sign languages, to avoid perpetuating systemic bias in the way we conceptualize sign language tasks @xcite .