How to Align Multiple Signed Language Corpora for Better Sign-to-Sign Translations? (2025)

Abstract: There are more than 300 documented signed languages worldwide, which are indispensable avenues for computational linguists to study cross-cultural and cross-linguistic factors that affect automatic sign understanding and generation. Yet, these are studied under critically low-resource settings, especially when examining multiple signed languages simultaneously. In this work, we hypothesize that a linguistically informed alignment algorithm can improve the results of sign-to-sign translation models. To this end, we first conduct a qualitative analysis of similarities and differences across three signed languages: American Sign Language (ASL), Chinese Sign Language (CSL), and German Sign Language (DGS). We then introduce a novel generation and alignment algorithm for translating one sign language to another, exploring Large Language Models (LLMs) as intermediary translators and paraphrasers. We also compile a dataset of sign-to-sign translation pairs between these signed languages. Our model trained on this dataset performs well on automatic metrics for sign-to-sign translation and generation. Our code and data will be available for the camera-ready version of the paper.

## introduction despite the growing need for advanced signing technologies, signed language (sl) resources remain scarce, posing significant challenges for computational linguistic research and accessibility within deaf or hard-of-hearing (dhh) communities. a crucial step toward improving these technologies is the development of larger, multilingual signed corpora. while recent efforts have focused on collecting pre-aligned video datasets-such as @xcite curating videos for 10 sls and @xcite compiling bible-based videos in 47 sls-the real challenge lies in the figure 1 : this figure depicts how we translate from one sign language to another. inputs are shown at the top (skeletal body positions for csl), and then the outputs are at the bottom for dgs. glosses, which are intermediary textual representations (such as, '山' for the sign for 'mountain' in csl), are also used with sign sequences. at its core, a paraphrasing alignment algorithm matches instances between different corpora. automatic alignment of signs rather than just data collection. even though these are significant data collection initiatives, the main bottleneck lies in automatically aligning signs instead of collecting pre-aligned videos. so, in this paper, we work towards ways to align sign language representations automatically across three signed languages: american sign language (asl), chinese sign language (csl), and german sign language (dgs). ## related work the study of signed languages has seen considerable advancements across two main fronts: sign language recognition (slr) and sign language generation (slg). slr has progressed from early visual recognition @xcite @xcite @xcite , segmentation @xcite efforts to sophisticated models capable of end-to-end translation @xcite @xcite , heavily relying on deep learning techniques like cnn/rnn @xcite and transformer-based models @xcite @xcite @xcite for state-of-the-art performances. concurrently, slg has greatly benefited from larger datasets, such as phoenix-14t @xcite and csl-daily @xcite , gradually increasing the accuracy of the generations @xcite @xcite @xcite @xcite . there have also been works focusing on more prosodic generations -i.e., the intensity, duration, and repetition of signs -with better use of the signing space and with more awareness of the intensifiers in facial expressions @xcite . @xcite , german sign language (dgs) @xcite , and american sign language (asl) @xcite with their glosses. similar patterns are shared across these sls in terms of the use of signing space (e.g., hands moving from top to bottom, as shown in the orange area), and hand gestures (e.g., bent fingers as an iconic representation of a snowflake, as shown in the blue areas) are shared across three languages. contrarily, the duration, and repetition of hand movements may differ due to other linguistic factors (e.g., the sign is repeated twice in dgs, while only once in csl and asl). amidst these developments, translation and alignment in slg have emerged as critical challenges. notable efforts in this area include applying nmt methods to translate spoken language text into sl glosses @xcite . they demonstrate substantial improvements in both dgs and asl corpora. similarly, earlier studies like @xcite and projects such as deep-asl @xcite have explored statistical and deep learning approaches to address the alignment of english text and asl gloss. bidirectional translation systems, exemplified by @xcite , have introduced generative models to enhance alignment between asl and english, marking significant strides toward more nuanced translation mechanisms. also, established challenges @xcite and review articles @xcite for translations between spoken and signed languages-such as swiss german and swiss german sign language (dsgs)have shown the successes and challenges of current state-of-the-art neural translation systems. in addition to bilingual translation systems, there has been recent interest in collecting and curating multilingual sign translation corpora. notably, works of @xcite curating a subset of videos from the spreadthesign initiative to form a dataset with parallel data for 10 sls across various domains, and @xcite ) presenting a single-domain signed videos in 47 sls. our contribution builds on these established paths by focusing on the in-the-wild alignment between different sls-where already aligned signed videos are not present-aiming to leverage unique linguistic features inherent to sls. this novel approach, which builds upon the foundational work in both slr and slg, as well as the specific translation and alignment challenges addressed by recent research, represents a pioneering effort to enable direct, meaningful communication across diverse sign language communities. ## aligning different signed languages to align different sl corpora, we first analyze them linguistically. in a data-driven manner, we also identify the challenges of aligning these separate datasets. as a result of these analyses, we propose an approach that uses a paraphrase detection module. qualitative linguistic analysis historically, distinct sls have evolved across various regions since as early as the 5 th century bc @xcite , each with its own set of features and rules, from phonology and syntax to semantics and pragmatics @xcite . these visual lan-guages harness gestures, facial expressions, and the signing space, leveraging shared cognitive abilities and linguistic conventions that, to some extent, unify sls globally. however, significant linguistic differences exist among sls and between them and spoken languages, highlighting the importance of technological interventions in bridging these gaps. for instance, despite english being a commonly spoken language in both the u.s. and the u.k., american sign language (asl) and british sign language (bsl) remain mutually incomprehensible @xcite , as the former has been developed from old french sign language, various village sls, and home sign systems; in contrast, the latter is developed from old british sign language. these differences underscore the critical role of technology in exploring the meta-linguistic skills that transcend the spoken-sign language divide. as shown in figure 2 , the signs with the same meaning (snowing) have shared signing spaces and are more interpretable across different sls. building on this observation, we hypothesize that signed data (containing videos, body positions, text, and glosses) that has already been collected for different sls under different corpora can be aligned. quantitative linguistic analysis signed languages can represent information in different aspects, such as gestures, movements, facial expressions, and sign duration. different sls also have different prosodic characteristics. to quantify these differences, we first analyze the average length of signed video frames by dividing the frame counts by the number of signs in the video. this can be regarded as an approximation of the information representation statistic for an individual sign. we use glosses as proxies to detect the number of signs in the video. since the how2sign dataset (asl) does not contain glosses, we use the texts instead. we show these preliminary analyses in it can be observed that asl tends to utilize a longer time to present a single sign, while dgs uses a shorter duration on average. even though this is a loose approximation of the information content per sign, it provides an initial understanding of how the temporality of signed languages may affect the alignment. based on these analyses, we further discuss the challenges of sign-to-sign translation in appendix a. ## our approach in this section, we introduce our approach to address the challenges of sign-to-sign translation. we focus on combining already-present sign video datasets in any language and align with another dataset using paraphrase detection algorithms. with this approach, we create a new corpus by pairing sign videos from csl-daily, phoenix-14t, and how2sign corpora. in this section, we describe the details of our alignment methodology, inspired by our lexical analyses in section §3. this happens in three steps: preprocessing, paraphrase detection, and postprocessing. ## our generation model the sign language translation (slt) task generally involves multiple modular subtasks: signto-gloss, gloss-to-text, text-to-gloss, and gloss-tosign translations. our approach also uses this modular approach with additional constraints of the source and target sls. in this section, we first introduce a transformer-based baseline model commonly used in the literature for sign-to-text translation (section §5.1). then, to better utilize the multi-modality of the sls, we further propose a multitasking model (figure 3 ) that makes use of the corresponding glosses that come with the input sign sequence to introduce more in-domain knowledge into the encoder part. lastly, we introduce an llm-based translation model, which is posited to be pre-trained with extensive in-domain knowledge that can be better utilized for the gloss-to-gloss translation task. ## evaluations and results in this section, we present the automatic metrics we use and the evaluation paradigm for sls. then, we discuss the results of our experiments with these metrics. ## discussions we discuss essential challenges that demand future efforts in sign-to-sign generation. one such challenge is the difficulty of cross-linguistic and crosscultural alignments. similar to spoken languages, sls can be affected by the physical and cultural factors of the user population. thus, sign representations are not unified across sls. there have been initiatives in the sign language processing community to use signwriting as an intermediary unified textual representation @xcite , but this is still in the process of wide adoption. in any case, as a community, we need more data with diverse and sociocultural relevance in addition to limited weather forecast-or bible-based datasets. for further discussion on the sociolinguistic aspects of this problem, please refer to appendix §f. another point of discussion is surrounding automatic evaluation metrics for signing. current evaluations are restricted to the back-translation results of the generated sign videos, which lack source text generated (back-translation) paired target da haben wir am morgen schnee und 明天|| 白天|| 会|| 下雨 。 下雪 || 了||，|| 今天|| 真冷。 schneeregen . here we have snow and sleet in the morning. it will rain during the day tomorrow. it's snowing, it's so cold today. 我|| 想|| 一杯|| 牛奶 ||，|| 你|| 要|| 早饭|| 我|| 吃|| 的|| 是|| 面包 now, a typical day starts with 什么|| 饮料|| ？ || 和|| 牛奶|| 。 breakfast. i want a glass of milk, what do you want? i had bread and milk for breakfast. hi hallo und guten abend hallo und guten abend hello and good night hello and good night ## conclusions and future work in this work, we address the problem of crosslingual slt, introducing a challenge for automatic video translation between sls with a focus on automatic, linguistically informed alignment. our effort facilitates cross-lingual sign language understanding and offers insights into signed languages' social, cognitive, and linguistic nuances, improving our understanding of their use across communities. we release the first automatically aligned corpus with cross-lingual pairs that span three sls, which can serve as a benchmark for future research. we show that llms can also be instrumental as intermediary translators, yet further experimentation and incorporation are necessary to judge their efficacy. we demonstrate that incorporating the gloss information can assist in understanding the video, which highlights the need for using intermediary textual representation to integrate more structure or stronger signals for better translation systems. future work could involve better body pose extraction techniques to better understand cross-linguistic and cross-cultural semantics and prosody of sls. also, text-to-video retrieval approaches @xcite can be used to verify video alignments across different sls.