SONAR - SLT : Multilingual Sign Language Translation via Language-Agnostic Sentence Embedding Supervision (2025)

Abstract: Sign language translation (SLT) is typically trained with text in a single spoken language, which limits scalability and cross-language generalization. Earlier approaches have replaced gloss supervision with text-based sentence embeddings, but up to now, these remain tied to a specific language and modality. In contrast, here we employ language-agnostic, multimodal embeddings trained on text and speech from multiple languages to supervise SLT, enabling direct multilingual translation. To address data scarcity, we propose a coupled augmentation method that combines multilingual target augmentations (i.e. translations into many languages) with video-level perturbations, improving model robustness. Experiments show consistent BLEURT gains over text-only sentence embedding supervision, with larger improvements in low-resource settings. Our results demonstrate that language-agnostic embedding supervision, combined with coupled augmentation, provides a scalable and semantically robust alternative to traditional SLT training.

## introduction sign languages (sls) are inherently visual and culturally embedded. each sl has evolved independently and is closely tied to the communities and spoken languages of its region. as a result, most sign language translation (slt) datasets are built around a single sign-spoken language pair (e.g., @xmath0, which makes it difficult to scale models across languages or to combine datasets. training a system for a new target language typically requires a separate model and fresh parallel data collection. historically, slt systems have relied on manually provided gloss supervision (camgoz et al., 2018), discrete word-like labels whose design and availability are language-, culture-, and regionspecific. even gloss-free slt approaches assume that sign inputs should be supervised by text from the co-occurring spoken language, keeping the learning signal tied to a single language @xcite @xcite and limiting cross-dataset reuse and generalization. recent work by @xcite has reduced the reliance on glosses by supervising slt with text-based sentence embeddings. this yields better semantic alignment, but the embeddings remain modality-specific and typically require dataset-specific fine-tuning. furthermore, compared to large pre-trained models that exploit vast text corpora, these text-only embeddings show limited cross-lingual transfer and reduced robustness. this raises the key question: can languageagnostic, multimodal sentence embedding supervision replace text-only alignment in slt? we hypothesize that language-agnostic, multimodal sentence embeddings can reduce the residual dependence on text. concretely, we build on sonar @xcite , a pretrained multilingual and multimodal embedding space that jointly rep-resents text and speech. sonar embeddings are claimed to be language-agnostic. our approach aligns sign representations directly with languageagnostic semantic vectors, thereby decoupling supervision from any specific spoken language and removing the need for glosses. our model integrates multiple modalities and supports direct supervision across all 200 languages covered by sonar (see figure 1 ). in contrast to prior systems that relied on additional stages or separate models for multi-target translation, our method enables direct translation into multiple languages within a single model. a major obstacle for slt is the scarcity of annotated data. recent work on self-supervised pre-training from unannotated or anonymized data @xcite has shown promise in addressing this challenge. this motivates our second question: can target-language augmentation further alleviate data scarcity and enhance robustness, particularly when combined with video augmentation? our coupled multiple target language and video perturbation augmentation strategy addresses these challenges by combining (i) targetlanguage augmentation, which pairs each sign sample with parallel sentences in multiple languages, and (ii) video augmentation, which perturbs the visual stream through spatial, temporal, and photometric transformations. these augmentations are complementary: multiple target-language augmentation strengthens semantic supervision without requiring new sign recordings, while video augmentation improves the invariance of the sign encoder. together, they yield a more robust slt model and provide a scalable, semantically grounded alternative to traditional training, unifying supervision across languages and modalities while reducing dependence on language-, culture-, and regionspecific annotations. in all, our contributions can be summarized as: ## conclusion we presented a scalable slt framework that breaks the traditional close dependency between sign and spoken languages in training data and system development. by aligning sign language videos with multilingual, multimodal sentence embeddings from sonar, our approach yields a languageagnostic semantic representation that generalizes across both sign languages and spoken targets. this reduces reliance on language-model priors and prioritizes visual grounding and slt-specific grammar over surface-level text patterns. experiments show that language-agnostic supervision enables robust translation even under sign-target mismatches. multilingual text augmentations, combined with visual augmentation, improves performance on phoenix-2014t despite limited data. ablations further confirm the advantages of this approach in preserving semantic adequacy. current evaluation practices often emphasize surface overlap rather than meaning. future work should develop metrics aligned with semantic similarity and extend supervision to low-resource sign languages and continuous signing in the wild.