FLEURS - ASL : Including A merican S ign L anguage in Massively Multilingual Multitask Evaluation (2025)

Abstract: Sign language translation has historically been peripheral to mainstream machine translation research. In order to help converge the fields, we introduce FLEURS-ASL, an extension of the multiway parallel benchmarks FLORES (for text) and FLEURS (for speech) to support their first sign language (as video), American Sign Language, translated by 5 Certified Deaf Interpreters. FLEURS-ASL can be used to evaluate a variety of tasks—primarily sentence- and discourse-level translation—between ASL and 200 other languages as text, or 102 languages as speech. We provide baselines for tasks from ASL to English text using a unified modeling approach that incorporates timestamp tokens and previous text tokens in a 34-second context window, trained on random video clips from YouTube-ASL. This model meets or exceeds the performance of phrase-level baselines while supporting a multitude of new tasks. We also use FLEURS-ASL to show that multimodal frontier models have virtually no understanding of ASL, underscoring the importance of including sign languages in standard evaluation suites.

## introduction despite calls to include sign languages in mainstream natural language processing @xcite , sign language translation remains peripheral to the machine translation field. the sign language translation field uses individual benchmarks for each language-such as how2sign @xcite for asl-constructed as train/test splits from one underlying dataset, often with overlapping signers. meanwhile, mainstream machine translation has developed independently constructed massively multilingual benchmarks like flores @xcite @xcite (a set of short documents translated into 200 languages), which has been extended to @xcite (speech for 102 languages) and belebele @xcite (reading comprehension questions for 122 languages). multiway benchmarks admit better comparisons of translation quality across languages because they reduce irrelevant variation in topic or style. independently constructed benchmarks give a more representative view of real-world performance because they test out-of-domain generalization rather than held-out generalization; fresh queries from users do not necessarily come from the training distribution. while youtube-asl @xcite takes a first step towards this kind of robust evaluation by reporting zero-shot scores on how2sign (i.e., scores without finetuning on how2sign), this test set (like others in sign language translation) is limited in many ways beyond just lack of standardization. for example, the content is translated live after only one look-over, by a combination of hearing interpreters and deaf signers (not specified to be professional deaf interpreters), with errors and redos included in-stream but skipped with sentence-level clip boundaries (complicating discourse-level evaluation) @xcite . of course, zero-shot evaluation on a held-out test split also forces researchers to discard valuable training data. in this paper, we introduce fleurs-asl, an extension of flores/fleurs to support their first sign language, american sign language. fleurs-asl can be used to evaluate a variety of tasks-primarily sentence-and discourselevel translation, but also in principle caption alignment, retrieval, and multiple-choice receptive comprehension-between asl and all the languages supported by flores/fleurs. fleurs-asl was translated by 5 certified deaf interpreters with ∼5-6 hours of preparation per 1 hour of translated content, ensuring a high quality bar. we provide several sets of baselines. first, we provide human baselines for sentence-level and discourse-level translation, at 13.0 bleu (64.6 bleurt) and 13.5 bleu respectively. second, as a model baseline for the suite of fleurs-asl tasks, we introduce a unified sign language to text modeling approach that builds upon youtube-asl @xcite 's t5 baseline. inspired by whisper's approach for speech to text tasks @xcite , we extend the context window to 34 seconds of signing, include 256 tokens of prior text context, incorporate timestamp tokens as both input and output, and train on random video clips. the use of multi-caption random clips heeds the call of @xcite to explore methods that can incorporate more context and are more tolerant to caption misalignment. this exploratory method meets or exceeds the performance of sentence-level baselines (3.7 vs. 2.9 bleu, 37.2 vs. 33.6 bleurt) while enabling new tasks through chunked autoregression, such as discourse-level translation, timed translation, and (albeit poorly) caption alignment-despite minimal tuning due to the large design space. and third, we use fleurs-asl to show that recent multimodal frontier models such as gpt-4o, claude 3 opus, and gemini 1.5 pro display virtually no understanding of asl, underscoring the importance of including sign language tasks in standard evaluation suites. we publicly release fleurs-asl at this link. we hope that it will encourage development and help to evaluate more generally capable sign language models, perhaps drawing inspiration from our new unified modeling approach. we also hope that the practices and findings from our data collection will aid in the construction of high-quality eval sets for more of the world's sign languages. see section 6 for a discussion of limitations; while there is great value to standardized benchmarks, we expect that more evaluation will be needed specifically for sign languages to capture variety across vision and language aspects and ensure consistent performance. ## the fleurs-asl benchmark the flores benchmark consists of 3001 sentences across 842 articles drawn from english wikipedia, split into a dev set of 997 sentences (281 articles), devtest set of 1012 sentences (281 articles), and test set of 992 sentences (280 articles). however, they do not release the test set so-like fleurs-we build on only the dev and devtest sets. we randomly split these sets into halves (4 chunks in total), each intended to be translated by a different interpreter. this strikes a compromise between signer diversity and the feasibility of recruiting/onboarding qualified interpreters. 3 naturally, works that perform self-supervised pretraining on uncaptioned videos already use random clips. @xcite ## baselines we provide three sets of baselines. first, we give human baselines for sentence-and discourse-level translation. second, we train new versions of the youtube-asl baselines that can support many 10 we ensure that the signer-dependent splits have disjoint train and test sets in light of the content overlap for signers #0 and #1+#2. fleurs-asl @xmath0, integrating prior text history, timestamps, and a longer context window (rather than just caption-level translation). and third, we evaluate current multimodal frontier models and show that they have virtually no understanding of asl. ## conclusion in this paper, we presented fleurs-asl, an extension of the flores/fleurs translation benchmarks to their first sign language, american sign language. fleurs-asl distinguishes itself from prior sign language benchmarks not just with its flexibility (supporting multiple tasks and translation directions by virtue of extending a larger benchmark) but also its domain-specific quality considerations (use of certified deaf interpreters translating english text with substantial preparation time, rather than semi-live interpretation following a video). we provided human baselines, domain-specific model baselines (using a new unified multitask approach that operates on random clips), and multimodal frontier model baselines; these collectively show that there is still a long way to go to reach usable performance on asl to english translation. we hope that fleurs-asl will encourage more focus on this task-framed in a way that deemphasizes strictly aligned captions and therefore admits use of noisier datasets-and serve as scaffolding to extend flores/fleurs to even more of the world's sign languages.