Explore More Guidance: A Task-aware Instruction Network for Sign Language Translation Enhanced with Data Augmentation (2022)

Abstract: Sign language recognition and translation first uses a recognition module to generate glosses from sign language videos and then employs a translation module to translate glosses into spoken sentences. Most existing works focus on the recognition step, while paying less attention to sign language translation. In this work, we propose a task-aware instruction network, namely TIN-SLT, for sign language translation, by introducing the isntruction module and the learning-based feature fuse strategy into a Transformer network. In this way, the pre-trained model’s language ability can be well explored and utilized to further boost the translation performance. Moreover, by exploring the representation space of sign language glosses and target spoken language, we propose a multi-level data augmentation scheme to adjust the data distribution of the training set. We conduct extensive experiments on two challenging benchmark datasets, PHOENIX-2014-T and ASLG-PC12, on which our method outperforms former best solutions by 1.65 and 1.42 in terms of BLEU-4. Our code and trained networks will be available upon the publication of this work.

## introduction sign language recognition and translation aims to transform sign language videos into spoken languages, which builds a bridge for communication between deaf and normal people. considering the unique grammar of sign languages, current effective recognition and translation systems involve two steps: a tokenization module to generate glosses from sign language videos, and a translation module to translate the recognized glosses into spoken natural languages. previous works @xcite @xcite @xcite * equal contribution. # corresponding author: min chen. have proposed various solutions to address the first step, but paid less attention to the translation system. hence, this paper aims to solve the problem of sign language translation (slt) with the goal of translating multiple recognized independent glosses into a complete sentence. to do so, most existing works @xcite directly apply advanced techniques, e.g., seq2seq model @xcite or transformer @xcite , from neural machine translation to slt. however, different from the lingual translation task in neural machine translation, slt poses several unique challenges. first, it is hard to collect and annotate a large amount of sign language corpus. it is still an open question that how to explore more guidance and external information for slt task by incorporating the pre-trained language models based on masses of unlabeled corpus. second, since sign languages are developed independently from spoken languages with quite different linguistic features, the discrep-ancy of representation space between glosses and spoken sentences is significant, thus increasing the translation difficulty. to address the above issues, we propose a novel task-aware instruction network, called tin-slt for sign language translation, further enhanced with a multi-level data augmentation scheme. our tin-slt is capable of encoding pre-trained language model's ability into the translation model and also decreasing the discrepancy between the representation space of glosses and texts. to begin with, we leverage the extracted hidden features from the pre-trained model as extra information to guide the sign language translation. besides, we apply an instruction module to transform general token features into task-aware features. in this way, we can fully utilize the language skills originating from the external world, thus reducing the demand for sign language training data. next, to better inject the information from pretrained model into the slt model, we design a learning-based feature fusion strategy, which has been analyzed and validated to be effective compared with existing commonly-used fusion ways. finally, considering the large difference between the sign language glosses and texts in terms of the representation space, we propose a multilevel data augmentation scheme to enrich the coverage and variety of existing datasets. in summary, our contributions are threefold: (i) a novel tin-slt network to explore more guidance of pre-trained models, (ii) a learning-based feature fusion strategy, and (iii) a multi-level data augmentation scheme. extensive experiments on challenging benchmark datasets validate the superiority of our tin-slt over state-of-the-art approaches; see figure 1 for example results. methods for sign language translation. early approaches for slt rely on seq2seq model and attention mechanism @xcite , while facing the limitation of long-term dependencies. later, motivated by the ability of the transformer @xcite , many researchers utilize it to effectively improve slt performance. for example, the work in camgoz et al. (2020) tried to use transformer for both recognition and translation, and promote the joint optimization of sign language recognition and translation. the subsequent work @xcite proposed the stmc-transformer network which first uses stm-c networks @xcite to achieve better results for slr, and then exploits transformer for translation to obtain better slt performance. general neural machine translation. broadly speaking, sign language translation belongs to the field of neural machine translation, with the goal of carrying out automated text translation. earlier approaches deployed recurrent network @xcite , convolutional network @xcite , or transformer @xcite as encoder-decoder module. among them, transformer has achieved state-of-the-art results, but the translation performance still needs to be improved due to the limited training corpus. in addition, there are some explorations in bringing the pre-trained models into neural machine translation @xcite @xcite . ## challenges the goal of this work is to translate the recognized multiple independent glosses (network input) into a complete spoken sentence (expected output). compared with general neural machine translation tasks, slt faces two main challenges: limited annotated corpus: compared with natural languages, the data resources of sign languages are scarce @xcite . as a result, the slt models trained on limited data often suffer from the overfitting problem with poor generalization @xcite . discrepancy between glosses (input) and texts (output): figure 2 shows the representation space of sign glosses (yellow dots) and translated texts (red dots) using word2vec @xcite on two different datasets. we can observe that the representation space of sign glosses is clearly smaller than that of the target spoken language, thus increasing the difficulty of network learning. ## our approach to address the above challenges, we propose tin-slt by effectively introducing the pre-trained model into slt task and further designing a multi-level data augmentation scheme. figure 3 depicts the detailed network architecture. in the following subsections, we will firstly introduce the network architecture of tin-slt, followed by our solutions to address the above two challenges. ## conclusion in this paper, we proposed a task-aware instruction network for sign language translation. to address the problem of limited data for slt, we introduced a pre-trained model into transformer and designed an instruction module to adapt slt task. besides, due to the discrepancy between the representation space of sign glosses and spoken sentences, we proposed a multi-level data augmentation scheme. extensive experiments validate our superior performance compared with state-of-the-art approaches. while there is obvious improvement among most evaluation metrics, the complexity of our models is also increased, causing a longer training period. in the future, we would like to explore the possibility of designing a lightweight model to achieve real-time efficiency.