Implicit Memory Transformer for Computationally Efficient Simultaneous Speech Translation (2023)

Abstract: Simultaneous speech translation is an essential communication task difficult for humans whereby a translation is generated concurrently with oncoming speech inputs. For such a streaming task, transformers using block processing to break an input sequence into segments have achieved state-of-the-art performance at a reduced cost. Current methods to allow information to propagate across segments, including left context and memory banks, have faltered as they are both insufficient representations and unnecessarily expensive to compute. In this paper, we propose an Implicit Memory Transformer that implicitly retains memory through a new left context method, removing the need to explicitly represent memory with memory banks. We generate the left context from the attention output of the previous segment and include it in the keys and values of the current segment’s attention calculation. Experiments on the MuST-C dataset show that the Implicit Memory Transformer provides a substantial speedup on the encoder forward pass with nearly identical translation quality when compared with the state-of-the-art approach that employs both left context and memory banks.

## introduction simultaneous speech translation (simulst) refers to the process of producing an output translation concurrently with an oncoming source speech input. for humans, performing accurate simulst is extremely difficult and becomes nearly impossible to perform over long periods of time. given the potential broad applications of simulst in industry and government sectors, there is a strong need for machine learning models to perform the task to a level above the capabilities of humans. one branch of machine learning models that have been effective in simulst is transformers @xcite using block processing, a process that breaks an input sequence into segments which the encoder processes sequentially and individually @xcite . as later segments may lose earlier information in a sentence (i.e., context fragmentation), techniques known as left context and memory banks have been introduced. the concept of left context was idealized with the transformer-xl @xcite , a model optimized for language modeling, which was later adapted for streaming automatic speech recognition (asr). the transformer-xl generated left context by saving the previous segment to each encoder layer, so the subsequent segment could include it in the attention calculation at the same encoder layer. memory banks were later introduced in the self-attention calculation of the augmented memory transformer @xcite , allowing it to outperform the transformer-xl in streaming asr and also be state-of-the-art in simulst @xcite . these memory banks were token summarizations of previous segments and helped retain explicit long-term dependencies. the augmented memory transformer also included the left context alongside the center (main) segment tokens with an additional right context, all of which add computational cost. we argue that the methods to generate and use left context and/or memory banks in both the transformer-xl and augmented memory transformer are naive, costing both models' performance at a given computational budget. in this paper, we propose a computationally efficient architecture, the implicit memory transformer, that implicitly retains memory through a novel left context generation method, thereby removing the need for memory banks entirely. briefly, the proposed left context method for a given encoder layer leverages the previous segment's attention output in the attention calculation of the current segment. our method for calculating left context is broadly applicable to any transformer model that utilizes block processing. the proposed implicit memory transformer is more computationally efficient than the augmented mem-ory transformer, reducing the cost of self-attention calculation, convolution layers, and feed-forward layers. we conduct our experiments on the english-german, english-french, and english-spanish language pairs of the must-c dataset @xcite and demonstrate a significant speedup over the augmented memory transformer for the forward pass of the encoder, with no reduction in the translation quality across all wait-k values. in the self-attention calculation for the encoder, memory banks, m i @xmath0, are added to the keys and values where n denotes the maximum number of memory banks for a given layer. each layer's memory banks summarize the previous segments and are theorized to allow the model to retain explicit long-term memory. each memory bank is created using the attention output of a summarization query, σ i @xmath1, included in the attention calculation. this summarization query is calculated by averaging the tokens in the current segment. for any given layer, the queries, keys, and values can be represented by the following equations: in each of the equations, w i q , w i k , and w i v are the query, key, and value projection matrices for layer i. the [.] operator concatenates l i n , c i n , r i n with σ i n or m i n . after the encoder processes each individual segment, they are concatenated before being provided to a simultaneous decoder @xcite . average lagging: average lagging is one prominent metric to determine the efficacy of a simulst model @xcite . it denotes in milliseconds the lag between the output translation and the input source sequence @xcite . bleu score: an equally important metric to evaluate a simulst model is the bleu score, which measures the translation similarity between the predicted output and the target output. the bleu score ranges from 0 to 1 and is often represented with percentages @xcite . ## experimental setup we conducted experiments on the english-german (en-de), english-french (en-fr), and english-spanish (en-es) language pairs from the must-c dataset @xcite . the data preparation scripts for the must-c dataset are provided in fairseq foot_0 @xcite , whereby kaldi is used to generate 80-dimensional log-mel filter bank features, and text is tokenized with a sentencepiece 10k unigram vocabulary. the statistics of the training, development, and test set (tst-common) for the english-german, english-french, and english-spanish language pairs of the must-c dataset are provided in ## conclusion achieving computationally efficient simultaneous speech translation (simulst) is critical to its deployment in practical real-time applications. however, even with the state-of-the-art simulst approach of the augmented memory transformer, its method of generating left context is computationally costly and ineffective, requiring the usage of memory banks to compensate for its shortcomings. as such, we propose an implicit memory transformer that utilizes an attention-based left context to provide the model with implicit memory. we found that the implicit memory transformer was able to achieve nearly identical performance to the augmented memory transformer at a significantly reduced computational cost.