T ran SF ormer: Slow-Fast Transformer for Machine Translation (2023)

Abstract: Learning multiscale Transformer models has been evidenced as a viable approach to augmenting machine translation systems. Prior research has primarily focused on treating subwords as basic units in developing such systems. However, the incorporation of fine-grained character-level features into multiscale Transformer has not yet been explored. In this work, we present a S low- F ast two-stream learning model, referred to as Tran SF ormer, which utilizes a “slow” branch to deal with subword sequences and a “fast” branch to deal with longer character sequences. This model is efficient since the fast branch is very lightweight by reducing the model width, and yet provides useful fine-grained features for the slow branch. Our TranSFormer shows consistent BLEU improvements (larger than 1 BLEU point) on several machine translation benchmarks.

## introduction transformer @xcite has demonstrated strong performance across a range of natural language processing (nlp) tasks. recently, learning multiscale transformer models has been evidenced as a promising approach to improving standard transformer. previous research on this line can be broadly categorized into two streams: one learns local fine-grained features by using a fixed-length window @xcite @xcite , linguistic-inspired local patterns @xcite , and a hybrid approach that combines convolution and self-attention models @xcite or run in parallel @xcite ; the other learns sequence representations by considering multiple subword segmentation/merging schemas @xcite . despite the attractiveness of these approaches, previous work is based on an assumption that subwords are the basic units in sequence modeling, and therefore ignores smaller, more fine-grained character-level features. in fact, the benefits of using characters have long been appreciated, and character-based models have been discussed in several sub-fields of nlp, such as language modeling @xcite and machine translation @xcite @xcite . but there are still important problems one needs to address in multi-scale transformer. the first of these is the computational challenge of dealing with long sequences. for example, when we represent an english text as a character sequence, the length of this sequence is in general 5× longer than that of the subword sequence. we therefore need to consider this length difference in model design. the second problem is that, from a multiscale learning perspective, learning text representations with features at different levels is not just making use of the syntactic hierarchy of language. to better model the problem, we need some mechanism to describe the interactions among these different linguistic units. in this study, we aim to exploit the potential of character-level representations in multiscale sequence models while maintaining computational efficiency. drawing inspiration from the slowfast convolutional models in video classification @xcite , we propose the slow-fast transformer (transformer) model, which utilizes a fast, thin branch to learn fine-grained character-level features and a slow, wide branch to capture correlations among subword features. a cross-granularity attention layer is placed between the self-attention and feedforward sublayers to make exchanges of cross-granularity information. this enables the slow branch to be aware of fine-grained features while providing optimized high-level representations of the input sequence to the fast branch. we also make use of character-to-word boundary information to model the interactions among neighboring characters in a word. additionally, we develop a boundary-wise positional encoding method to better encode the positional information within words for the fast branch. through a series of extensive experiments on the wmt'14 english-german, wmt'17 chinese-english and wmt'16 english-romanian tasks, we demonstrate that transformer yields consistent performance gains while having a negligible increase in the number of parameters and computational cost. as a bonus, our trans-former is robust to errors caused by suboptimal tokenization or subword segmentation. ## related work multiscale transformer learning multiscale transformer is a promising way to acquire for further improvements in the machine translation task. a feasible way is to model global and local patterns to enhance transformer models @xcite @xcite . these work mainly modeled the localness within a fixed window size upon subword input features. apart from these, @xcite partitioned the input sequence according to phrase-level prior knowledge, and build attention mechanism upon phrases. similarly, @xcite proposed a multi-granularity self-attention mechanism, designed to allocate different attention heads to phrases of varying hierarchical structures. perhaps the most related work to ours is umst @xcite . they re-defined the sub-word, word and phrase scales specific to sequence generation, and modeling the correlations among scales. however, more fine-grained character-level scale is not explored in the previous work due to the serve challenge for encoding long character sequences. character-level nmt fully character-level neural machine translation originates from recurrent machine translation system in @xcite . they built a fully character-level encoder-decoder model, and utilize convolution layers to integrate information among nearby characters. @xcite show the potential of character-level models which can outperform subword-level models under fully optimization. this contributes to their greater flexibility in processing and segmenting the input and output sequences, though modeling such long sequences is time-consuming. more recently, several studies analyze the benefits of character-level systems in multilingual translation scenarios @xcite , low-resource translation and translating to typologically diverse languages @xcite . but these methods all simply view characters as basic units in language hierarchy, and it is still rare to see the effective use of multi-scale learning on character-based language features. ## method the proposed transformer follows a encoderdecoder paradigm (see figure 1 ) which involves two encoder branches operating at different input granularities. the original subword encoder, which has a large model capacity for fully learning correlations among input individuals, is defined as the slow branch. the other branch, designed to handle character-level representations using a thin encoder to efficiently capture correlations among characters, is referred to as the fast branch. our goal is to use the fast branch to learn a fine-grained but less precise representation to complement the slow branch. in the following sections, we will elaborate the core design of slow branch, fast branch and the cross-granularity attention, respectively. ## conclusions in this work, we comprehensively leverage the potential of character-level features in multiscale sequence models while preserving high computational efficiency. to accomplish this, we propose a slow-fast transformer architecture consisting of two branches in the encoder. the slow branch, akin to the vanilla transformer, handles subwordlevel features, while the fast branch captures finegrained correlations among characters. by leveraging the complementary features provided by the fast branch, our transformer demonstrates consistent improvements in bleu scores on three widelyused machine translation benchmarks. further indepth analyses demonstrate the effectiveness of the transformer and its potential as a universal multiscale learning framework.