Fingerspelling within Sign Language Translation (2025)

Abstract: Fingerspelling poses challenges for sign language processing due to its high-frequency motion and use for open-vocabulary terms. While prior work has studied fingerspelling recognition, there has been little attention to evaluating how well sign language translation models understand fingerspelling in the context of entire sentences—and improving this capability. We manually annotate instances of fingerspelling within FLEURS-ASL and use them to evaluate the effect of two simple measures to improve fingerspelling recognition within American Sign Language to English translation: 1) use a model family (ByT5) with character- rather than subword-level tokenization, and 2) mix fingerspelling recognition data into the translation training mixture. We find that 1) substantially improves understanding of fingerspelling (and translation quality overall), but the effect of 2) is mixed.

## introduction fingerspelling is a system used within sign languages to borrow words from spoken languages by spelling them out using a manual alphabet @xcite . each sign language has its own manual alphabet, which is either 1-handed (e.g., in american sign language) or 2-handed (e.g., in british sign language) @xcite . while fingerspelling is only one component of sign languages, it is often an important one; for example, in american sign language, fingerspelling constitutes 12-35% of signing @xcite . fingerspelling poses challenges for receptive sign language processing because a) its hand movements are quick, small, & highly coarticulated/blended @xcite ) and b) it is primarily used for out-of-vocabulary terms (such as proper nouns or domain-specific vocabulary) that don't have native signs @xcite . while there is a body of prior work studying handshape classification for fingerspelling @xcite and then fingerspelling recognition for entire phrases @xcite @xcite @xcite , there has been little attention on evaluating how well translation models understand fingerspelling within the context of entire sentences, and improving this specific important capability. in some ways, one expects this task to be harder than fingerspelling recognition (because it requires full sign language understanding), but in other ways it should also be easier (because the pace/neatness of natural fingerspelling is tailored to the available context and easier to understand given lexical context @xcite @xcite . we study this question-fingerspelling within sign language translation-in the context of translation from american sign language to english, trained on a noisy ∼2800-hour superset of youtube-asl @xcite and evaluated on fleurs-asl @xcite . in order to measure the improvement in fingerspelling specifically, we manually annotate all the instances of fingerspelling within fleurs-asl foot_0 and score the character-level accuracy of spans within predicted translations that are identified to correspond to the fingerspelled terms. (we automatically identify the relevant spans with an llm.) we use this setup to evaluate two simple measures: we find that 1) provides substantial improvements in overall translation quality (38.8 → 45.4 bleurt); more of this gain comes from sentences with fingerspelling (37.3 → 44.8 bleurt) than without (42.5 → 46.5 bleurt); and the accuracy of fingerspelled phrases within those translations improves dramatically (76.6% @xmath0.6% cer). on the other hand, adding 2) on top of 1) gives mixed or negative results (45.4 @xmath1.1 bleurt, 41.6% @xmath2.1% cer). we hope that these findings and the fleurs-asl-fs evaluation protocol will be useful for understanding the behavior of fingerspelling within sign language machine translation. in particular, we think that character-level tokenization should become standard practice for sign language translation models, or at least that future work should prioritize achieving the same benefits with fewer compromises. ## related work in this section we survey related work in three areas: spelling in llms, machine learning for fin-gerspelling, and datasets & modeling approaches for learning from sign language data with weakly aligned captions. ## fleurs-asl-fs fleurs-asl (tanzer, 2024) is an extension of the massively multilingual flores (nllb team, 2022) / fleurs @xcite translation benchmarks for text and speech respectively into american sign language video. because flores is drawn from wikipedia articles, it is relatively rich in proper nouns and other specialized vocabulary, making it a good testbed for fingerspelling. we annotate each of the 1749 sentences in fleurs-asl with the sequence of phrases that are fingerspelled within them; we refer to these annotations as "fleurs-asl-fs". 2 the first author (a non-native proficient signer) performed these annotations given access to the entire video context and the reference captions; the intent was to recover the ground truth fingerspelled terms, not measure the ability of a human to recognize them from the video alone. subjectively, this task was easy and not especially time-consuming; it could generally be performed in real time while watching the video @xmath3. in the vast majority of cases, these fingerspelled terms were subspans of the reference text, but sometimes the translation into asl used different or extra terms. the definition of fingerspelling is somewhat fuzzy at the margins, so we made the following decisions on inclusion in annotations: ## experiments in this section, we describe our experiments on two simple measures that could improve the quality of fingerspelling within sign language translation: using character-level tokenization and jointly training on independently collected fingerspelling data. ## conclusion in this paper we examined the role of fingerspelling within sign language machine translation, using fsboard, noisy youtube-asl, and fleurs-asl with new fingerspelling annotations. we found that byt5 (character-level tokenization) substantially improves upon t5 (subword-level tokenization) on continuous asl fingerspelling recognition, asl fingerspelling recognition within translation, and asl to english translation overall. cotraining translation models on fingerspelling recognition data, on the other hand, provides no clear improvement. we hope that these results will encourage researchers to consider and improve upon characterlevel tokenization for sign language translation models, and that the community will continue to reevaluate sign language modeling decisions that were inherited from mainstream machine translation, where the tradeoffs may no longer be relevant or have different balance in this domain @xcite .