TTIC ’s WMT - SLT 22 Sign Language Translation System (2022)

Abstract: We describe TTIC’s model submission to WMT-SLT 2022 task on sign language translation (Swiss-German Sign Language (DSGS) - German). Our model consists of an I3D backbone for image encoding and a Transformerbased encoder-decoder model for sequence modeling. The I3D is pre-trained with isolated sign recognition using the WLASL dataset. The model is based on RGB images alone and does not rely on the pre-extracted human pose. We explore a few different strategies for model training in this paper. Our system achieves 0.3 BLEU score and 0.195 Chrf score on the official test set.

## introduction sign language, a full-fledged natural language that conveys meaning through gestures, is the primary chief of communication among deaf people. sign language translation is a task for automatically translating sign languages into written languages. due to its widespread potential applications, it has recently received growing research interest @xcite . existing methods for sign language translation are primarily based on gloss, a transliteration system annotating sign language with symbols from written language. utilizing gloss usually boosts the performance of current translation systems by a large margin. in the widely used german sign language translation benchmark phoenix14t @xcite , state-of-the-art gloss-based models @xcite are roughly 15 points better (in bleu-4) than gloss-free models @xcite . however, gloss is more expensive to annotate than written language translation. there have been relatively few amounts of studies for glossfree sign language translation. specifically, @xcite ; @xcite utilize local visual features (e.g., hands) to enhance the translation performance. those systems require domain-specific training data (e.g., labeled handshape data used in orbay and akarun (2020)), which is not always accessible for the target sign language. the fusion of visual features at different scales also increases the complexity of the modeling pipeline. in this paper, we study a simple model for sign language translation between dsgs and german in a gloss-free setting. our model uses a 3d convolutional network for visual feature extraction and a transformer-based encoder-decoder for sequence modeling. it is built on raw rgb images rather than pose keypoints, thus avoiding potential mistakes from pose estimation and remaining fast in inference. we further study the impact of hyperparameters and different pretraining strategies on translation quality. without ensembling, our model achieves 0.3 bleu score and 0.195 chrf score on the official test set. ## method in this section, we describe our method for sign language translation. our model consists of an inflated 3d convnet (i3d) @xcite for visual encoding and a transformer-based encoder-decoder model @xcite for sequence modeling, which are described respectively below. i3d i3d @xcite is a 3d convolutional neural network proposed in action recognition. i3d has previously been explored in sign language processing @xcite @xcite and achieved competitive performance in isolated sign recognition @xcite . more formally, given a sequence of image frames i 1:t , the i3d model m v encodes them into a sequence of visual features f 1:t ′ : f 1:t ′ = m v (i 1:t ), where t and t ′ respectively denote the length of video and visual feature sequence. note due to the temporal stride in convolutional kernels of i3d, t ′ is not equal to t and is usually several factors smaller. to encourage the visual encoder m v to capture more signing-related visual cues (e.g., arm movement, handshape, and so on), we pretrain the i3d model with isolated sign recognition on wlasl, a large-scale dataset consisting of isolated american sign language (asl) signs. though asl and dsgs are two different sign languages, visual features regarding body movement are shared. empirically, we observed considerable gains in isolated sign pretraining. for computational efficiency, the pretrained i3d network m v is frozen in translation model training. m s is a standard sequenceto-sequence model widely used in machine translation @xcite @xcite . thus we only briefly review it here and a more detailed description can be found in @xcite . our sequence-to-sequence model m s includes a transformer encoder and transformer decoder, which are joined via attention. specifically, the transformer encoder transforms the visual features f v 1:t into e 1:t by injecting temporal information based on self-attention and positional embedding. the transformer decoder generates token sequence w 1:n in an auto-regressive manner while attending to the encoder output e 1:t through the attention mechanism. training loss we use cross-entropy loss for model training. more formally, given the translation pair (i 1:t , ŵ1:n ), suppose the model outputs probability vector p(•|i 1:t , ŵ1:n-1 ) at decoder step n. the loss is then computed as inference at test time, we use beam search for decoding image sequence i 1:t . the beam width and length penalty are hyperparameters tuned using the development set. ## experimental setup data we use focusnews and srf data to train our translation model. both focusnews and srf consist of dsgs-german pairs, which include 19 and 16 hours (10,136 and 7,071 sequences) of dsgs videos, respectively. the two datasets differ in multiple aspects. for example, focusnews are live signing from teleprompters by deaf signers based on news from 2008 to 2014, whereas srf dataset contains news videos from 2020 to 2021 which is interpreted into dsgs by hearing interpreters. both datasets are incorporated into training. note that frame rates in videos of focusnews and srf differ, we feed the raw videos in focusnews and srf without frame rate conversion. to pretrain the visual encoder, we use wlasl @xcite , a large-scale isolated sign dataset including ∼ 21k pairs of american sign language video clips and english words. training we use sentencepiece unigram tokenizer @xcite to tokenize the german translation. the number of subword units is tuned to 18,000 we use a 2-layer transformer with 512 hidden dimensions and 2048 hidden dimensions for both encoder and decoder. a dropout layer with a zeroing probability of 0.1 is added between the self-attention layer and the feedforward network. the model is trained with adam (kingma and ba, 2015) for 18k steps at a batch size of 32. the learning rate is linearly increased to 0.0008 for 2k steps and decayed to 0 in the remaining steps. the visual backbone i3d is pretrained on wlasl and frozen during translation model training. during isolated sign training, we initialize i3d from a model trained on the action recognition dataset kinetics @xcite . we use sgd with a 0.9 momentum value to train the model for 50 epochs at a batch size of 4. the initial learning rate is 0.001 and is halved if accuracy on the validation set does not increase for 3 epochs. before feeding into i3d, each isolated sign video is truncated to a 64-frame clip, which is padded with all-zero frames if the length is shorter than 64. each image frame is resized to 240 × 240. it is randomly cropped to 224 × 224 and horizontally flipped at a probability of 0.5 in training. at test time, we only center cropping to every image frame. evaluation we evaluate the system using bleu-{1,2,3,4} @xcite ) and rouge @xcite scores. ## experimental results in this section, we report results and conduct some analyses of the translation model on the development set.