S ign M usketeers: An Efficient Multi-Stream Approach for Sign Language Translation at Scale (2025)

Abstract: A persistent challenge in sign language video processing, including the task of sign language to written language translation, is how we train efficient model given the nature of videos. Informed by the nature and linguistics of signed languages, our proposed method focuses on just the most relevant parts in a signing video: the face, hands and body posture of the signer. However, instead of using pose estimation coordinates from off-the-shelf pose tracking models, which have inconsistent performance for hands and faces, we propose to learn the complex handshapes and rich facial expressions of sign languages in a self-supervised fashion. Our approach is based on learning from individual frames (rather than video sequences) and is therefore much more efficient than prior work on sign language pre-training. Compared to a recent model trained on publicly avaiable data that established a new state of the art in sign language translation on the How2Sign dataset, our approach yields similar translation performance, using less than 3% of the compute.

## introduction recent work on sign language processing spans human-computer interaction @xcite , computer vision @xcite , and natural language processing (nlp) @xcite research. the nature of signed languages, which involve the use of manual features (handshape, orientation, location, and movement) and non-manual features (facial expressions, head movements, body pose), presents challenges for machine learning models @xcite . in particular, how to effectively and efficiently represent signed languages while preserving their inherent attributes remains a persistent challenge. our focus is on the task of translation from sign language video to a written language (sign language translation, or slt). this is one of the most practically important tasks, necessary to bridge (part of) the communication gap between deaf and hard of hearing (dhh) populations and hearing populations @xcite . recent work @xcite proposed a self-supervised video pretraining approach to handle sign language translation, which achieved state-of-the-art performance on the how2sign dataset of american sign language (asl) to english translation @xcite . the intuition behind this approach is to extend the pre-training of a strong video model (in this case, hiera @xcite ) with a largescale unannotated set of sign language videos (in this case from youtube-asl @xcite ) and use this extended pre-trained model as a feature extractor for the supervised translation task. however, this pre-training is extremely costly: its longest pre-training run uses 64 a100 80gb gpus for 14 days, making the approach infeasible for many researchers and practitioners. the approach in @xcite implicitly treats sign language sequences like any other (long) videos. but signed languages are, first and foremost, languages, and like any other language, they possess linguistic properties that may provide an inductive bias about the more important aspects of the video @xcite . in this work, we ask the question: can we infuse basic linguistic properties of signed languages into self-supervised pre-training to develop a scalable compute-friendly approach? in the context of signed language processing, the use of off-the-shelf human pose estimator models @xcite has been one of the most common ways of incorporating the linguistic constraints of sign language into models. the intuition of using pose estimation is that it removes irrelevant features that do not affect the meaning of signs, such as body shape and visual background, and therefore focuses entirely on the linguistically relevant aspects of hand, face and body pose @xcite . however, this pose-based approach has several limitations that make it sub-optimal for capturing the details of signed languages, particularly in the representation of hands @xcite and faces @xcite . first, the human pose estimator models used in existing methods @xcite @xcite are typically trained on everyday handshapes, which are often less complex than the handshapes found in signed languages. second, human pose estimators are unreliable in capturing crucial non-manual components which are essential for signed languages, such as eye gaze. 1 our approach is inspired by the multistream/multi-channel property of signed languages ( §2.2); that is, the fact that they consist of a combination of actions performed largely independently by multiple body parts (channels). specifically, our proposed method ( §3, see overview in fig. 1 ) focuses on the most relevant parts of a signing video-the face, hands, and body pose of the signer ( §3.1)-to handle sign language translation at scale ( §2.1). instead of fully relying on off-the-shelf human pose estimators, we propose to learn rep-resentations of handshapes and facial expressions directly from signing videos using self-supervised learning ( §3.2, 2.3.) our method captures the intricacies of handshapes and facial expressions for the supervised training stage ( §3.3), without the need for extensive pre-training data or computational resources. this allows us to overcome the limitations of human pose estimators and preserve the crucial linguistic information conveyed through handshapes, eye gaze, and facial expressions in signed languages. we name our approach sign-musketeers: like the heroes of dumas' books (dumas, 1894), three image channels (face and two hand boxes) join forces with a fourth companion (pose features) in the quest for glory (accurate sign language translation). we conduct experiments ( §4) on how2sign and find that our approach achieves competitive performance while using a smaller model (in terms of number of parameters), with 41x less pre-training data and 160x fewer pre-training iterations ( §4.1) , using roughly 3% of the compute resources of the previous state-of-the-art approach @xcite . we also provide ablation experiments showing the value of individual design decisions ( §a). ## method below we first describe the data preprocessing (frame parsing) procedure, then the self-supervised pre-training of feature extractors, and finally the supervised learning of the asl video to english translation system. ## experiments as in other work @xcite , we use the youtube-asl and how2sign @xcite datasets. youtube-asl contains roughly 600,000 clips, or roughly 700 hours, of asl video with weakly aligned english text translations. 4 how2sign consists of 31,128 / 1,741 / 2,322 clips for the training / validation / test sets. for the self-supervised training of dinov2-hand and dinov2-face, due to computational constraints we limit ourselves to 1 million random face crops and 1 million random hand crops from youtube-asl. we note that the state-of-the-art method proposed by @xcite sees about 50 million frames during pre-training. in line with previous studies, we employ the following training schedules for the supervised (translation) stage, using a stride of 2 for every video clip: h2s: supervised training exclusively on the how2sign dataset, without using the youtube-asl dataset. yt: supervised training solely on youtube-asl, and evaluation on how2sign in a zero-shot setting. @xmath0: supervised training on youtube-asl, followed by supervised fine-tuning on how2sign. note that @xcite and @xcite include an additional training schedule, yt + h2s, which involves training on a mixture of yt and h2s. for the supervised training stage, similarly to other work @xcite , we initiliaze our t5 from a t5.1.1-base pre-trained checkpoint. we use a batch size of 128 ( 16 per gpu running on 8 gpus), with other hyperparameters identical to @xcite . additionally, when further fine-tuning on how2sign after training on youtube-asl, we perform an extra 5,000 steps of fine-tuning. ## conclusion signmusketeers is a data-and compute-efficient method for sign language translation at scale. it uses a multi-stream encoding scheme that focuses on important parts of a signing video (facial expressions, hands, and body pose) and requires only individual frames during self-supervised pretraining, in contrast to prior work on pre-training that requires long duration videos. signmusketeers achieves competitive performance with roughly 40× less data and 50× less computation than prior work. limitations this research has several key limitations. the most significant is that despite matching or exceeding previous studies' results, the overall performance remains inadequate. therefore, machine translation cannot yet reliably replace human sign language interpreters across diverse contexts. another limitation stems from the much smaller training datasets compared to those typically used for developing self-supervised models in the speech and text domains. additionally, since this study focuses solely on american sign language, we cannot determine whether our approach would work effectively for other sign languages, even though different sign languages share common features and communication methods. also, in this work we focus only on the sign language translation task from video to text. it will be important to adapt this approach to the opposite direction (text to video) and other sign language processing tasks such as isolated sign language recognition, continuous sign language recognition, and fingerspelling detection. to overcome these challenges, future research should focus on building larger datasets and incorporating multiple sign languages into the training process. is the multi-stream approach beneficial compared to just using the original frames? to evaluate the effectiveness of the multi-stream approach, we compare the performance of using only the original frames (uncropped) with that of the multistream model. as shown in