Towards Online Continuous Sign Language Recognition and Translation (2024)

Abstract: Research on continuous sign language recognition (CSLR) is essential to bridge the communication gap between deaf and hearing individuals. Numerous previous studies have trained their models using the connectionist temporal classification (CTC) loss. During inference, these CTC-based models generally require the entire sign video as input to make predictions, a process known as offline recognition, which suffers from high latency and substantial memory usage. In this work, we take the first step towards online CSLR. Our approach consists of three phases: 1) developing a sign dictionary; 2) training an isolated sign language recognition model on the dictionary; and 3) employing a sliding window approach on the input sign sequence, feeding each sign clip to the optimized model for online recognition. Additionally, our online recognition model can be extended to support online translation by integrating a gloss-to-text network and can enhance the performance of any offline model. With these extensions, our online approach achieves new state-of-the-art performance on three popular benchmarks across various task settings. Code and models are available at https://github.com/FangyunWei/SLRT.

## introduction sign languages are visual languages conveyed through hand shapes, body movements, and facial expressions. the domain of sign language recognition (slr) @xcite has recently attracted considerable attention, particularly for its potential to bridge the communication gap between the hearing and deaf communities. slr can be categorized into isolated sign language recognition (islr) @xcite and continuous sign language recognition (cslr) @xcite . islr, a supervised classification task, aims to accurately predict the gloss foot_0 for each individual sign. in contrast, as no annotations of sign boundaries are provided, cslr is a weakly supervised task. in this context, a well-optimized model is able to predict a sequence of glosses from a continuous sign video containing multiple signs. compared to islr, cslr is more challenging but also more practical. the primary objective of this work is to develop an online cslr system. drawing inspiration from the advancements in speech recognition @xcite , numerous cslr models are trained using the established connectionist temporal classification (ctc) loss @xcite with sentence-level annotations. during inference, these models typically process the entire sign video to make predictions, leading to issues like high latency and significant memory consumption. this method is known as offline recognition, as depicted in fig. 1a . unlike modern speech recognition systems, which can recognize spoken words on the fly, cslr still lags behind due to the lack of practical online recognition solutions, which are essential in real-world scenarios such as live conversations or emergency situations. although ctc-based methods can be adapted for online recognition using a sliding window technique, our empirical findings show that the discrepancy between training (using entire, untrimmed sign videos) and inference (using short, trimmed sign clips) results in suboptimal performance. in this paper, we take the first step towards practical online cslr. instead of directly training with the ctc loss on a cslr dataset, we optimize an islr model using classification losses on a dictionary containing all glosses from the target cslr dataset. during inference, we apply a sliding window to a given sign video stream and process each video clip through the well-optimized islr model to obtain corresponding predictions. this approach aligns training and inference by utilizing short video clips as input for both. however, using a sliding window with a small stride may lead to multiple scans of the same sign, resulting in repetitive predictions. to mitigate this, we introduce an effective post-processing technique to eliminate duplicate predictions. we also consider co-articulations, which are the transitional movements of the body and hands between consecutive signs in a continuous video. since these movements are generally meaningless, we assign them to an additional background category, and predictions in this category are discarded during post-processing. our online method is illustrated in fig. 1b . in our methodology, we train the islr model using a sign dictionary. existing cslr datasets, such as phoenix-2014 @xcite , @xcite , and csl-daily @xcite , lack such dictionaries. however, a pre-trained cslr model utilizing ctc loss can effectively segment continuous sign videos into individual isolated signs @xcite @xcite . this process, known as ctc forced alignment, is a well-established technique in the speech community for accurately aligning transcripts to speech signals @xcite . therefore, we use the state-of-the-art cslr model, twostream-slr @xcite , as the sign segmentor for any cslr dataset. this approach allows us to create a sign dictionary that aligns with the glossary of the respective cslr dataset. during inference, a fixed-length sliding window may inadvertently include both a sign and its co-articulations. to better align the training and the inference, we generate a set of augmented signs from each isolated sign by trimming video segments surrounding it. this procedure also significantly enriches the training data. different signs typically exhibit various durations. personal habits of signers, e.g., signing speed, can also amplify this issue. this necessitates the model's adaptability to variations in sign durations, especially when a sliding window includes both a sign and co-articulations. we introduce a saliency loss, which compels the model to focus predominantly on the foreground signs while minimizing the influence of the co-articulations. the implementation is simple-we adopt an auxiliary classification loss on the pooled feature of the foreground parts. while our method is primarily designed for online cslr, it also shows promise for online sign language translation (slt) and enhancing offline cslr models. we start by implementing an additional gloss-to-text network, applying the wait-k policy @xcite tailored for simultaneous (online) machine translation. this allows for online slt by gradually feeding the gloss predictions generated by our online cslr model into the waitk gloss-to-text network. furthermore, our online cslr model can facilitate offline cslr models in performance. this is achieved by incorporating a lightweight adapter into our frozen online model and combining the adapter-generated features with those extracted by a pre-trained offline cslr model. our contributions can be summarized as follows: taking a step further, slt @xcite @xcite @xcite involves translating sign languages into spoken languages. this task is commonly approached as an nmt problem, employing a visual encoder followed by a seq2seq translation network. similar to cslr, online slt remains largely unexplored. sign spotting. given an isolated sign, the goal of sign spotting is to identify whether and where it has been signed in a continuous sign video @xcite . modern sign spotting works typically rely on extra cues, including external dictionaries @xcite , mouthings (albanie et al., 2020), or transformer attention @xcite . however, these cues can be either difficult to obtain (e.g., dictionaries) or unreliable (e.g., mouthings). sign spotting is typically used to enrich the training source for islr and few works validate the spotting task in the context of cslr. online speech recognition. practical online speech recognition systems have been studied in numerous works @xcite @xcite . in these studies, model architectures vary, including cnn (pratap et al., 2020), transformer @xcite , and a combination of them @xcite . additionally, multiple optimization frameworks are explored, such as ctc @xcite , seq2seq @xcite , or a hybrid of these @xcite . unlike online speech recognition, online cslr remains underexplored. this work takes the first step towards building a practical online cslr framework. ## method an overview of our online framework is shown in fig. 2 . we first build a sign dictionary with the aid of a sign segmentor, i.e., a pre-trained cslr model (sec. 3.1). then, we train an islr model on this dictionary, employing dedicated loss functions at both the instance and gloss levels (sec. 3.2). this is followed by a demonstration of online inference using the optimized islr model (sec. 3.3). finally, we present two extensions (sec. 3.4): (1) enabling online slt with a wait-k gloss-to-text network; (2) boosting the performance of an offline model using our online model. 11052 ## conclusion in this work, we develop a practical online cslr framework. first, we construct a sign dictionary that aligns with the glossary of a target dataset. to enrich the training data, we collect augmented signs by cropping clips around each sign. to enable online cslr, we train an islr model on the dic-tionary, using both standard classification loss and the introduced saliency loss. during inference, we perform online cslr by feeding each sliding window into the well-optimized islr model on the fly. a simple yet efficient post-processing algorithm is introduced to eliminate duplicate predictions. furthermore, two extensions are proposed for online slt and boosting offline cslr models, respectively. along with the extensions, our framework achieves sota performance across three benchmarks under various task settings.