Continual Learning in Multilingual Sign Language Translation (2025)

Abstract: The field of sign language translation (SLT) is still in its infancy, as evidenced by the low translation quality, even when using deep learn- ing approaches. Probably because of this, many common approaches in other machine learning fields have not been explored in sign language. Here, we focus on continual learning for mul- tilingual SLT. We experiment with three con- tinual learning methods and compare them to four more naive baseline and fine-tuning ap- proaches. We work with four sign languages (ASL, BSL, CSL and DGS) and three spo- ken languages (Chinese, English and German). Our results show that incremental fine-tuning is the best performing approach both in terms of translation quality and transfer capabilities, and that continual learning approaches are not yet fully competitive given the current SOTA in SLT.

## introduction continual learning, incremental learning and life-long learning are equivalent terms for machine learning approaches that learn sequentially or from dynamic data, where a sequence can be either a succession of tasks, datasets, languages, etc @xcite @xcite @xcite . data in real-world applications is rarely static; it typically arrives as a continuous stream, with only a fraction available at the beginning. this dynamic flow necessitates that deployed systems adapt incrementally to new information over time. continual learning is critical for such scenarios, as it allows models to integrate new data-such as additional sign languages-without requiring complete retraining or redeployment. one of the main challenges in continual learning is catastrophic forgetting @xcite @xcite , and tackling this issue is central to continual learning research. researchers work on methods to enable models to adapt to a sequence of tasks while retaining previously acquired knowledge @xcite @xcite . while the phenomenon of catastrophic forgetting and its mitigation has been extensively studied in the field of continual learning, these investigations have primarily focused on traditional natural language processing (nlp) tasks and/or multilingual nlp contexts, and never before on sign language. @xcite define a taxonomy for continual learning approaches based on their location within the machine learning pipeline, e.g., when feeding the data, at the architecture level, during optimization, etc. their five classes correspond to approaches based on replay (saving, recovering or mimicking old data), regularization (either weight, feature or function regularization adding terms that take into account the old model), optimization (gradient modification or projection, meta-learning, etc.), representation (self-supervision, (continual) pre-training, adaptation of a fixed backbone) and architecture (including task-specific or adaptive parameters). in our work and for comparison purposes, we consider an example from each of the replay, regularization, and architecture classes. sign language translation (slt) is a hard problem. it involves translating from an input video to text or speech with traditionally small amounts of parallel data such as the phoenix2014t dataset @xcite with 7096 video-text pairs for german sign language, and the csl-daily dataset @xcite with 18400 for chinese sign language. these numbers are very far from the millions of parallel sentences used to train textto-text transformer models for machine translation. state-of-the-art slt systems @xcite have traditionally relied on intermediate (manual) annotations called glosses and this has been limiting the size of the training data for a long time. recently, the field has been significantly advanced by the emergence of large, non-curated datasets (without glosses), such as bobsl @xcite and youtube-sl-25 @xcite both with more than 1 million videotext pairs, the first for british sign language and the second one for a combination of more than 25 sign languages. this new data, especially that coming from youtube, is motivating sign language translation systems with a strong focus on pre-training. to the best of our knowledge, we are the first to examine the effects of continual learning in multilingual sign language translation. this work is developed on the eve of forthcoming (and hopefully open) large multilingual sign language translation models, which, similar to machine translation, will enable fine-tuning and life-long learning. we present a gloss-free multilingual transformer model for sign language translation, pretrained on chinese sign language (csl) to chinese, german sign language (dgs) to german, and american sign language (asl) to english using the phoenix2014t, csl-daily, and how2sign datasets. to evaluate its performance, we conduct experiments on three languages from the spreadthesign-ten (sp-10) dataset @xcite -chinese sign language, german sign language, and british sign language (bsl)-in a sequential learning setup, comparing continual and non-continual fine-tuning approaches. foot_0 2 related work continual learning approaches. as explained in the introduction, multiple approaches exist in the literature to address the task. relevant to our work, and based on the taxonomy of @xcite , are regularization-based, replay-based, and architecture-based approaches. regularizationbased methods stabilize model parameters by adding regularization terms. these methods are simple to implement but require access to the previous model for reference, and can focus on weight or function regularization @xcite . weight regularization adjusts parameters based on the importance of the old model, often using the fisher information matrix (fim), as in elastic weight consolidation (ewc) @xcite and its variants like memory aware synapses (mas) @xcite . function regularization preserves the outputs of the previous model through knowledge distillation, with the old model as the teacher and the current model as the student. techniques like lwf @xcite , lwm @xcite , icarl @xcite , eeil @xcite , and lucir @xcite use either new or old training samples for distillation loss. replay-based methods have proven to be the most effective among the three main continual learning (cl) approaches-regularizationbased, replay-based, and architecture-based-as demonstrated by van de @xcite . replay-based methods retain a small set of old training samples in a memory buffer, which are later used for training alongside the current data. some methods, like reservoir sampling @xcite , ring buffer @xcite , and mean-of-feature @xcite , employ fixed strategies for sampling from memory. more sophisticated techniques focus on optimization, such as gss @xcite , which maximizes sample diversity, and gem @xcite and a-gem @xcite , which create individual constraints during training. architecture-based methods, which focus on task-specific parameters, have gained renewed attention with the rise of parameter-efficient fine-tuning (peft). early approaches, like progressive networks @xcite , created a new neural network for each task, connected to previous ones via lateral links. recent advancements include color @xcite , which trains task-specific lora modules, moral @xcite , which combines mixture-of-experts with lora for life-long learning, and the approach by @xcite , which extends pre-trained transformers with adapters. continual learning in multilinguality and machine translation. @xcite introduce a method for integrating new languages into multilingual translation models by a simple update of the vocabulary, allowing quick adaptation to languages with different scripts while maintaining minimal performance loss on existing language pairs. coria et al. (2022) analyzes cross-lingual transfer in continual learning for sequence labeling using multilingual bert, finding that despite some forgetting, forward transfer is retained, with most past language knowledge stored in the word representation encoder rather than the task-specific classifier. related to this, @xcite examines catastrophic forgetting in a multilingual setting with up 10924 to 51 languages. they introduce an effective learning rate scheduling method that reduces forgetting and performs well across various continual learning techniques. similarly, @xcite develops a two-stage approach to enhance pre-trained multilingual neural machine translation (mnmt) models, employing contrastive learning for adapting to new data and collaborative distillation for consolidating knowledge. they also introduce a knowledge transfer method that integrates external model insights into existing mnmt models and a dual importance-based model division technique that focuses on parameters crucial for incremental tasks, thus improving performance on new languages while preserving the quality of existing translations @xcite . two studies diverge from traditional continual learning between language pairs by focusing on multi-hop continual learning or evaluating continual learning under language shift, which simulates sequential learning of a single task across a stream of input from different languages. m'hamdi et al. ## continual learning in multilingual sign language translation in this section, we formally define continual learning for multilingual slt and introduce the models that will be evaluated in subsequent sections. ## experimental setting in this section, we describe the data and preprocessing we use for training and testing the above approaches, the evaluation metrics and methodology, and describe the base model on top of which all the experiments are performed. ## summary and conclusion we presented the first study on continual learning for multilingual sign language translation. our multilingual setting involves three language pairs: @xmath0, @xmath1, and @xmath2. we first pre-trained a multilingual slt model using two different video feature extraction methods, i3d and s3d, with s3d achieving better translation quality under the same conditions. we then compare a range of standard fine-tuning and continual learning methods. multilingual and incremental fine-tuning are the methods that offer the best average translation quality and incremental fine-tuning is also able to minimize the effect of catastrophic forgetting. experience replay is the best within the continual learning methods irrespective of the size of the memory buffer. the language pair order plays a role in our experiments. we observe that languages seen later during training are improved compared to an early appearance. we also observe that the adaptation to @xmath3. curiously, translation quality into english is good even if the multilingual base model is trained with asl data and the continual fine-tuning is done with a bsl corpus. this indicates that, given the low overall translation quality, the decoder performance (text generation into english in this case) is the most important factor. stronger base multilingual models are needed for sign language to achieve comparable results to their textual machine translation counterparts.