S ign A lign LM : Integrating Multimodal Sign Language Processing into Large Language Models (2025)

Abstract: Deaf and Hard-of-Hearing (DHH) users increasingly utilize Large Language Models (LLMs), yet face significant challenges due to these models’ limited understanding of sign language grammar, multimodal sign inputs, and Deaf cultural contexts. Further, current approaches that try to address these limitations, frequently reduce sign language processing (SLP) to traditional translation tasks, neglecting the multimodal and linguistic complexity inherent in signed languages. In this paper, we present an empirical investigation informed by learning theory into natively integrating sign language support within LLMs, directly addressing the documented needs of DHH users. We introduce the first text-based and multimodal LLMs capable of sign language processing called SignAlignLM, and propose new prompting and fine-tuning strategies incorporating sign linguistic rules and conventions. We show that LLMs can be generalized interfaces for both spoken and signed languages if trained with a multitasking paradigm. Our code and model checkpoints are open-source.

## introduction communication inherently integrates symbols, gestures, and sensory experiences, particularly evident in the rich, multimodal nature of sign languages. despite technological progress, deaf or hard-of-hearing (dhh) individuals continue to face significant barriers when engaging with computational models (such as large language models (llms)), primarily due to the neglect of sign languages' unique linguistic features. even though most signers find novel ways of interacting with newly emerging technologies @xcite , figure 1 : deaf users have specific requests pertaining to the development of llms, as shown above. we show that text-based and multimodal open llms when prompted or fine-tuned, can learn to perform sign language processing tasks. further, while fine-tuning, multitasking on both spoken (openorca) and signed (phoenix-14t) corpora alleviates forgetting of spoken language capabilities (e.g., qa tasks in english). llms still have significant room for improvement to be more accessible and useful for them. current computational approaches typically simplify sign language processing (slp) into a translation problem, inadequately capturing their multimodal complexity and interactive aspects. insights from cognitive science and linguistics emphasize the importance of aligning computational models with the multimodal and spatial properties of sign languages to facilitate meaningful interactions. a recent study by @xcite highlights the outcomes if this alignment is not properly achieved: 44.1% of dhh llm users report difficulty asking questions, and 22.1% are dissatisfied due to limited sign language support. responding directly to the articulated needs of the dhh community, this paper proposes a new family of text-based and multimodal models capable of sign-language processing, with prompting and fine-tuning strategies explicitly designed to em-bed these linguistic structures. through rigorous evaluations across diverse slp tasks, our results demonstrate substantial improvements in handling sign language inputs without compromising spoken language performance. in this paper, we explore ways in which we can natively integrate sign language support into llms to be as useful to signers as they are for spoken language users. we claim that current llms 1) lack sign language-specific tasks in pre-training, 2) are not prompt-tuned with signing or glossing rules, and 3) are overfit to spoken language tasks foot_1 . to this end, we introduce the first family of llms capable of sign language processing, called sig-nalignlm. we also conduct an in-depth empirical analysis of their performance, backed by a learning theory-based account. we first start with prompttuning as the basic algorithm of incorporating sign language into llms. we include sign language grammar rules with additional socio-linguistic conventions in our prompt analysis. backed by theoretical analysis, we further hypothesize that combining spoken and signed capabilities is achievable through multitasking-interleaving sign language tasks with spoken language tasks during the finetuning stage. in more detail, our contributions are described as follows, 1. we survey literature on deaf user needs from llms and how to integrate sign language into them. 2. we fine-tune text-only and multimodal llms on various sign language processing tasks for the first time, 3. we empirically study the problem of catastrophic forgetting during fine-tuning on sign language data, providing solutions to resolve this issue. 4. we introduce multimodal and text-based llms fine-tuned on slp tasks and analyze in detail whether they satisfy the requirements set by signers. our results show that fine-tuning large, pre-trained models offers new generalization capabilities compared to previous sign recognition training strategies, e.g., via in-context learning. from the personal interviews presented in @xcite , there are three major areas that dhh users would like to see improvements with llms: 1) llms should understand diverse spoken language use by the deaf , 2) llms should have a deep understanding of the dhh community, and 3) llms should accept visual sign language as input. essentially, signers want llms to understand sl grammar order, or at least the gloss notation-an intermediary textual representation for signs. furthermore, signers want sign languagespecific datasets to be used in the training of the llms. also, they request that video-based sign understanding be included in llms. here, it is necessary to distinguish reading and writing in spoken versus sign languages. most bilingual signers default to reading and writing in spoken languages or modified versions of them instead of sls while interacting with llms due to lack of effective interfaces @xcite @xcite @xcite . we are specifically interested in the problem of interfacing with signers using text-based or multimodal llms, which helps signers to read and write in sls while also enhancing their reading and writing capabilities in spoken languages (samuel j. supalla, 2021). as a concrete example, we are aiming to create an llm that bilingual signers use to converse with using text or videos, instead of using german to chat with llms. these concrete requirements by dhh motivate our work and open up the following several important scientific questions and research areas: our results point to a future where language models can also be pre-trained on sls without significant degradation of their spoken language capabilities, marking an essential step for the wider adoption of sls into llm pipelines. this has broader implications for creating llm-based tools that meet the requests of signers. ## experimental setup in this section, we introduce the details of the data, tasks, and the text-based and multimodal llms we use in the experiments (see figure 2 ). ## turning llms into sign interfaces in this section, we empirically and theoretically explore ways of turning llms into sign languagecapable models using three algorithms: in-context learning, supervised fine tuning, and multitask finetuning. many current proprietary or open-source llms do not consider sign language data during their training process (e.g., due to lack of signers or expertise in deaf culture). this is also noticed by deaf users and is requested to be mediated in @xcite . we believe this lack of accessibility can be mitigated in two ways: 1) including sl-specific data in pretraining or 2) using techniques such as prompt-tuning or fine-tuning with various slp tasks. in addition, sign languages do not exist in isolation of spoken languages, so in order to be a faithful interface, an llm should be able to communicate both in spoken and signed forms. hence, we investigate how these modalities can be combined using multitasking. ## findings in this section, we present our results and discuss our findings under five research questions. we outline all of these questions in the following sections and give answers to them with our findings. for further discussion of these findings and their position in the slp research literature, please refer to appendix § e. ## related work besides text-based models like llama @xcite , mixtral @xcite , qwen @xcite , orca @xcite , phi @xcite , multimodal models have been gaining popularity, especially in computer vision communities. large vision-language models such as llava @xcite , video-llama @xcite , video-llava @xcite , languagebind @xcite , multimodal-gpt @xcite , lavis @xcite , lavila @xcite , and univl @xcite propose to align representations of combinations of images, videos, text, and/or speech signals with human judgments. further details of these and similar models have been discussed in a survey paper by @xcite . however, none of these models claim to include slp tasks in their pre-training or fine-tuning data. through our theoretical and empirical studies, this paper aims to address this gap. the absence of literature using large models for slp is mainly due to the low-resource nature of sls @xcite . however, there have been several lines of research applying transformer-based language models to sign language translation @xcite @xcite , sign language understanding @xcite , sign generation @xcite , signwriting translation @xcite , incorporating facial expressions @xcite , modeling prosody @xcite , and sign language segmentation @xcite . @xcite an early work that leverages (smaller, but still large) language models with shared vocabularies for slp. they focus on older models (without rlhf, @xcite . further, @xcite ; @xcite give a more recent application of llms as part of a translation pipeline, and @xcite finetunes diffusion-based llms for sign avatar generation. however, none involves instruct-tuning large language models (text-based or multimodal) with both spoken and signed capabilities, which we introduce in this paper for the first time. in addition to the slp and llm literature, sl education works are important for this work. in the sl pedagogy literature, some works focus on case studies of gloss-based intermediary textual constructs as ways of asl to english literacy @xcite , a formal distinction between sign and spoken language reading (supalla, 2017), and reading assessments for dhh signers @xcite . these works have influenced our choice of glosses as intermediary representations for text-based llms. we believe that text-based and video-based language models can be helpful as reading and writing companions that use glosses or videos to interface with signers. ## conclusion incorporating the rich, multimodal aspects of sign languages into language modeling requires considering the linguistic, cognitive and cultural contexts of them. in this paper, we have introduced the first family of llms capable of sign language processing. we prompted, fine-tuned, and compared these text-only and multimodal language models for various sign language processing tasks. we have provided language theory grounding and analyzed our results with implications on how much llms can meet the needs of signers without losing capabilities in spoken languages. from our findings, it can be claimed that llms can be fine-tuned to sls, and in-context learning can help to create an off-the-shelf llm tailored towards the deaf and hard-of-hearing community, which can be accomplished without forgetting spoken language capabilities. moving forward, training bigger models with larger multilingual corpora is a promising next step for a broader set of novel sign language processing tasks. we hope this initial family of llms, along with our exploration of linguistically and cognitively-informed prompting nuances, marks the first step toward making llms equally accessible and capable in both signed and spoken languages. we make all our code available at https://github.com/merterm/signrep and model checkpoints available at https:// huggingface.co/merterm/signrep . we will update our model suite as newer open-source llms, datasets, and slp tasks become available.