An Efficient Gloss-Free Sign Language Translation Using Spatial Configurations and Motion Dynamics with LLM s (2025)

Abstract: Gloss-free Sign Language Translation (SLT) converts sign videos into spoken language sentences without relying on glosses, which are the written representations of signs. Recently, Large Language Models (LLMs) have shown remarkable translation performance in gloss-free methods by harnessing their powerful natural language generation capabilities. However, these methods often rely on domain-specific fine-tuning of visual encoders to achieve optimal results. By contrast, we emphasize the importance of capturing the spatial configurations and motion dynamics in sign language. With this in mind, we introduce Spatial and Motion-based Sign Language Translation (SpaMo), a novel LLM-based SLT framework. The core idea of SpaMo is simple yet effective: instead of domain-specific tuning, we use off-the-shelf visual encoders to extract spatial and motion features, which are then input into an LLM along with a language prompt. Additionally, we employ a visual-text alignment process as a lightweight warm-up step before applying SLT supervision. Our experiments demonstrate that SpaMo achieves state-of-the-art performance on three popular datasets—PHOENIX14T, CSL-Daily, and How2Sign—without visual fine-tuning.

## introduction sign language is a visual means of communication primarily used by deaf communities, relying on physical movements rather than spoken words. in this paper, we tackle sign language translation (slt), focusing on converting sign videos into spoken language sentences. early slt methods @xcite @xcite have primarily relied on glosses-written representations of signs using corresponding words. glosses provide a structured form of sign language, which helps identify semantic boundaries within continuous sign sequences. this, in turn, allows the models to better comprehend the overall content of the sign videos @xcite ). however, annotating glosses is a labor-intensive and time-consuming process that requires expertise in sign language. this significantly hinders the expansion of sign language datasets and limits the development of slt methods @xcite @xcite . to address these limitations, there has been a shift towards gloss-free methods that rely solely on the sign videos and corresponding translations. while these methods still underperform compared to the gloss-based methods, efforts have been made to reduce the performance gap by focusing on temporal semantic structures @xcite and aligning visual and textual modalities @xcite @xcite . recently, llms have demonstrated remarkable translation performance in a gloss-free setting by harnessing their powerful language generation capabilities. however, the modality gap between the continuous sign videos and discrete text poses a challenge for the llms in effectively understanding the sign videos. to address this, many methods fine-tune their visual encoders to be more domainspecific to sign language @xcite @xcite . that said, fine-tuning visual encoders can be resource-heavy and time-consuming, making it impractical for real-world applications, especially when considering the diversity of sign languages. this leads to an important question: is domainspecific tuning of visual encoders necessary to achieve optimal performance in llm-based slt? we argue that focusing on the inherent characteristics of sign language could reduce the need for such extensive fine-tuning. first, visual encoders trained on general domains @xcite have already proven highly effective in downstream tasks such as action recognition @xcite and video captioning @xcite . moreover, llms are capable of retaining rich visual information from these general encoders in their latent space @xcite . rather than emphasizing fine-tuning, we shift our attention to the crucial roles of spatial configurations and motion dynamics in sign language. spatial configurations include the arranging and positioning of signs within the signing space, including hand shapes, facial expressions, and body postures. these components work together to distinguish different signs and convey their intended meanings @xcite . as shown in figure 1a , the signs for "cold" and "winter" both use the same handshape, with a shivering motion of the fists. the primary difference lies in the facial expressions: "cold" is typically accompanied by a tense or grimaced expression, while "winter" may feature a more neutral expression. motion dynamics, on the other hand, involve the path, speed, and rhythm of hand movements, illustrating how movements alter the meanings of signs over time @xcite . as shown in figure 1b , the signs for "chair" and "sit" both use the same "h" handshape and involve the interaction of both hands. however, the motion differentiates these signs: "chair" involves a repetitive tapping motion, while "sit" involves a single, smooth motion. these examples highlight the importance of the spatial configurations and motion dynamics in conveying accurate messages in sign language. to this end, we introduce a novel gloss-free framework, spatial and motion-based sign language translation (spamo). spamo is designed to fully exploit the spatial configurations and motion dynamics in the sign videos using off-the-shelf visual encoders, all without the need for domainspecific fine-tuning. as shown in figure 2 , the core idea is simple: we extract spatial features (spatial configurations) and motion features (motion dynamics) using two different visual encoders, and feed these into an llm with a language prompt. specifically, we use a pre-trained image encoder (e.g., vit) as spatial encoder (se) to individually encode each frame for its spatial features. to further refine the spatial features, we apply s 2 scaling @xcite , which processes a sign image at multiple scales. additionally, we use a video encoder (e.g., videomae) as motion encoder (me) to encode sign clips (groups of sign frames) into the motion features. to capture finer motion dynamics, we apply a sliding window approach, which results in implicit gloss-level representations @xcite . next, sign adapter (sa), comprising multi-layer perceptron (mlp) layers, transfers these features to the llm. to further bridge the modality gap, we propose visual-text alignment (vt-align), a training strategy that aligns the visual features with the llm's embedding space, promoting more efficient training and improved translation performance. in all, our contributions can be summarized as: • we introduce spamo, a novel gloss-free framework based on llms. our method is simple yet effective, focusing on conveying core elements of sign language to llms without domain-specific tuning of visual encoders. ## method we first give an overview of our framework in section 3.1. we then explain spatial encoder (se) and motion encoder (me) in sections 3.2 and 3.3, respectively. next, we discuss sign adapter (sa) in section 3.4 and vt-align in section 3.5. finally, we explain the training details in section 3.6. ## conclusion in this paper, we introduced spamo, a novel glossfree slt framework based on llms. apart from the previous methods that rely on domain-specific fine-tuning of their visual encoders, spamo focuses on capturing the spatial configurations and motion dynamics, eliminating the need for resourceintensive fine-tuning. we also proposed vt-align, a training strategy that effectively aligns and narrows the modality gap between the sign videos and target texts, enabling the transformation of the sign videos into inputs interpretable by the llm. our approach achieved state-of-the-art results on three popular datasets. furthermore, we provided the first comprehensive analysis of how the llm interprets the sign videos within its embedding space and translates them into corresponding text.