Prior Knowledge and Memory Enriched Transformer for Sign Language Translation (2022)

Abstract: This paper attacks the challenging problem of sign language translation (SLT), which involves not only visual and textual understanding but also additional prior knowledge learning (i.e. performing style, syntax). However, the majority of existing methods with vanilla encoder-decoder structures fail to sufficiently explore all of them. Based on this concern, we propose a novel method called Prior knowledge and memory Enriched Transformer (PET) for SLT, which incorporates the auxiliary information into vanilla transformer. Concretely, we develop gated interactive multi-head attention which associates the multimodal representation and global signing style with adaptive gated functions. One Part-of-Speech (POS) sequence generator relies on the associated information to predict the global syntactic structure, which is thereafter leveraged to guide the sentence generation. Besides, considering that the visual-textual context information, and additional auxiliary knowledge of a word may appear in more than one video, we design a multi-stream memory structure to obtain higher-quality translations, which stores the detailed correspondence between a word and its various relevant information, leading to a more comprehensive understanding for each word. We conduct extensive empirical studies on RWTH-PHOENIX-Weather-2014 dataset with both signer-dependent and signer-independent conditions. The quantitative and qualitative experimental results comprehensively reveal the effectiveness of PET.

## introduction recently, the combination of vision and language attracts increasing attention. sign language translation which aims to provide translated natural sentences for sign language videos is a valuable but challenging task in this topic (camgoz et al., 2018, † corresponding author 2020a,b; @xcite . since the visual and textual modalities are not aligned strictly in a weakly-supervised manner, the difficulties of sign language translation mainly lie in the multimodal representation learning of both modalities and the alignments between them. besides, additional prior knowledge (i.e. the performing style of different signers, the common syntactic structures of sentences) also has a strong influence on multimodal learning. encoder-decoder structures built upon long short-term memory unit @xcite ) (lstm) or transformer @xcite are widely used in end-to-end sign language translation, which directly generates natural sentences without intermediate products like gloss sequences. generally, the encoder extracts and encodes the sign language information, the decoder makes full use of the encoded results with crossmodal interaction. camgoz et al. (2018) first proposes the sign language translation task and utilizes lstms combined with attention mechanism (e.g. luong attention @xcite , badanau attention @xcite ) to solve it. due to the insufficient capacity to capture the long-range temporal correlations, @xcite replaces lstm with transformer, which could correlate any two-time steps of sequential features. the stacked attention blocks improve most of the metrics by a large margin. @xcite combines multiple articulatory channels with an-choring losses and proposes a novel multi-channel transformer architecture for sign language translation. @xcite employs video segment representation with multiple temporal granularities to develop a semantic pyramid network. in summary, many endeavors are devoted to the improvement of deep architectures for multimodal representation learning. however, the influences of additional prior knowledge are totally ignored. for example, as shown in fig. 1 , the natural sentence has its unique syntactic structure. motivated by the above observations, we propose a new method called prior knowledge and memory enriched transformer for sign language translation. specifically, we develop gated interactive multi-head attention which associates the multimodal representation and global signing style with adaptive gated functions. besides, we employ sentence templates that consist of pos tags to represent the syntactic structures of natural sentences, and accordingly, syntax learning is performed by directly inferring pos tags with the style-specific multimodal representation. the natural sentences are generated conditioned on such auxiliaries. furthermore, we find that the visual and textual context information, and additional auxiliary knowledge of a word may appear in more than one sign language video. for example, a word that comes up with different words may lead to various contextual visual perceptions, and the general gestural tendency of a word could support the decoding process. therefore, we design a multi-stream memory structure to store the full-spectrum correspondence between a word and its various relevant information in training data. the obtained memory contents are employed to aid in decoding. we conduct extensive empirical studies on the benchmark dataset, rwth-phoenix-weather-2014t (phoenix14t) @xcite , with both signer-dependent and signer-independent conditions. the quantitative and qualitative results comprehensively reveal the effectiveness and generalization of pet. the main contributions of this paper can be summarized as follows: ## approach fig. 2 shows the overall framework of prior knowledge and memory enriched transformer based on encoder-decoder structure. we develop gated interactive multi-head attention in all the attention blocks with adaptive gated control of signing style embeddings. in the decoder, we treat the sentence templates which consist of pos tags as the syntax-aware auxiliary for natural sentence generation. practically, two consecutive decoding blocks (syntactic and textual blocks) rely on the style-specific multimodal representation to predict the target words. furthermore, we design a multistream memory structure to enhance the comprehensive understanding for each word. ## experiments in this section, we present the experimental settings of sign language translation and report the results on the benchmark datasets. ## conclusion in this paper, we have proposed a new method called prior knowledge and memory enriched transformer for sign language translation. specifically, we develop the adaptive gated interaction which associates the multimodal representation and global signing style in all the attention blocks. one pos sequence generator relies on the associated information to predict the global syntactic structure, which is thereafter leveraged to guide the sentence generation. besides, considering that the visual and textual context information, and additional auxiliary knowledge of a word appear in more than one video, we design a memory structure to store the full-spectrum correspondence between a word and its various relevant information in the training data. the experimental results reveal the effectiveness and generalization of pet.