Non-autoregressive Modeling for Sign-gloss to Texts Translation (2025)

Abstract: Automatic sign language translation has seen significant advancements, driven by progress in computer vision and natural language processing. While end to end sign-to-text translation systems are available, many systems still rely on a gloss-based representation–an intermediate symbolic representation that functions as a bridge between sign language and its written counterpart. This paper focuses on the gloss-to-text (gloss2text) task, a key step in the sign-to-text translation pipeline, which has traditionally been addressed using autoregressive (AR) modeling approaches. In this study, we propose the use of non-autoregressive (NAR) modeling techniques, including non-autoregressive Transformer (NAT) and diffusion models, tailored to the unique characteristics of gloss2text. Specifically, we introduce PointerLevT, a novel NAT-based model designed to enhance performance in this task. Our experiments demonstrate that NAR models achieve higher accuracy than pre-trained AR models with less data, while also matching the performance of fine-tuned AR models such as mBART. Furthermore, we evaluate inference speed and find that NAR models benefit from parallel generation, resulting in faster inference. However, they require more time to achieve an optimal balance between accuracy and speed, particularly in the multistep denoising process of diffusion models.

## introduction deafness and hearing loss affect over 1.5 billion people worldwide, with 430 million experiencing disabling hearing loss foot_0 . sign languages serve as an alternative to verbal speech, yet communication barriers between deaf individuals and non-sign language users can lead to social isolation and limited figure 1 : framed sign video (sourced from @xcite ) is converted into glosses, and then to texts. a promising alternative is sign glosses, a written representation of sign language that captures the core meaning of signs 4 . unlike standard written texts, glosses follow distinct linguistic rules in grammar, word selection, and sequential expression. converting sign glosses into natural text-known as the gloss-to-text (gloss2text) problem (see figure 1 )-is typically framed as a lowresource machine translation task. due to the scarcity of parallel gloss-text data, traditional approaches often rely on data augmentation and ar training to adapt neural models for this task @xcite . ar models are widely used in nlp and have demonstrated strong performance in numerous tasks @xcite @xcite . however, they come with inherent limitations, including strong dependence on large datasets for effective learning, error accumulation during sequential generation, and high computational costs due to their step-by-step decoding nature. given these constraints, and considering the unique characteristics of the gloss2text task-namely, its low-resource setting, high lexical overlap between glosses and text, and the challenge of inferring natural language from simplified gloss sequences-we explore nar modeling as a promising alternative. our approach primarily focuses on improving effectiveness while also considering efficiency as a complementary aspect. to achieve this, we investigate two types of nar models. the first approach is based on the edit-based levenshtein transformer (levt) @xcite , which refines predictions iteratively through insertion and deletion operations. to enhance its performance, we introduce pointerlevt, an improved version that integrates a pointer network, allowing for better alignment between glosses and their corresponding textual representations. edit-based nar models are particularly well-suited for tasks requiring minimal corrections or modifications, balancing effectiveness and efficiency. the second approach leverages diffusion-based sequence models, including vanilla diffusion models @xcite and diffuseq @xcite . these models condition their sampling steps on sign glosses, guiding the model through a probabilistic denoising process that refines a noisy sequence into the target text. diffusion models offer robustness against noise, better handle variability in input, and provide stochasticity for possible exploration of text generation, which show potential to enhance gloss2text generation accuracy. our experiments show that these nar models outperform ar counterparts trained from scratch on the same small dataset, demonstrating advantages in either effectiveness, efficiency, or both. this highlights the potential of nar approaches for gloss2text task and similar low-resource and noisy-input nlp problems such as grammatical error correction, post-editing, and text infilling or modification. ## conclusion this paper investigates two non-autoregressive (nar) approaches to address the task of translating sign glosses into standard text (gloss2text). while large pretrained models such as mbart and mt5 typically achieve strong performance, our experiments reveal that these nar models trained with a smaller size of dataset not only match the accuracy of fine-tuned large pre-trained language models but also significantly outperform smaller, distilled models of comparable capacity. in particular, the edit-based nar strategy strikes a notably favorable balance between accuracy and efficiency, positioning it as a viable alternative to resource-intensive pretrained models. meanwhile, the conditional diffusion-based approach attains very high accuracy and could benefit from further optimization to enhance its efficiency, making it well-suited for future research. moreover, the findings suggest that these methods can be applied to broader text-editing tasks resembling gloss2text, such as grammatical error correction and post-editing. this underscores the potential of novel modeling techniques in both sign language translation and general nlp applications. future work may focus on refining both editbased and diffusion-based nar models to bolster effectiveness and efficiency across diverse realworld scenarios.