Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks (2023)

Abstract: Transducer and Attention based Encoder-Decoder (AED) are two widely used frameworks for speech-to-text tasks. They are designed for different purposes and each has its own benefits and drawbacks for speech-to-text tasks. In order to leverage strengths of both modeling methods, we propose a solution by combining Transducer and Attention based Encoder-Decoder (TAED) for speech-to-text tasks. The new method leverages AED’s strength in non-monotonic sequence to sequence learning while retaining Transducer’s streaming property. In the proposed framework, Transducer and AED share the same speech encoder. The predictor in Transducer is replaced by the decoder in the AED model, and the outputs of the decoder are conditioned on the speech inputs instead of outputs from an unconditioned language model. The proposed solution ensures that the model is optimized by covering all possible read/write scenarios and creates a matched environment for streaming applications. We evaluate the proposed approach on the MuST-C dataset and the findings demonstrate that TAED performs significantly better than Transducer for offline automatic speech recognition (ASR) and speech-to-text translation (ST) tasks. In the streaming case, TAED outperforms Transducer in the ASR task and one ST direction while comparable results are achieved in another translation direction.

## introduction neural based end-to-end frameworks have achieved remarkable success in speech-to-text tasks, such as automatic speech recognition (asr) and speech-totext translation (st) @xcite . these frameworks include attention based encoder-decoder modeling (aed) @xcite , connectionist temporal classification (ctc) @xcite and transducer @xcite etc. they are designed with different purposes and have quite different behaviors, even though all of them could be used to solve the mapping problem from a speech input sequence to a text output sequence. aed handles the sequence-to-sequence learning by allowing the decoder to attend to parts of the source sequence. it provides a powerful and general solution that is not bound to the input/output modalities, lengths, or sequence orders. hence, it is widely used for asr @xcite @xcite @xcite , and st @xcite @xcite . ctc and its variant transducer are designed to handle monotonic alignment between the speech input sequence and text output sequence. a hard alignment is generated between speech features and target text tokens during decoding, in which every output token is associated or synchronized with an input speech feature. ctc and transducer have many desired properties for asr. for example, they fit into streaming applications naturally, and the input-synchronous decoding can help alleviate over-generation or under-generation issues within aed. @xcite ; @xcite show that transducer achieves better wer than aed in long utterance recognition, while aed outperforms transducer in the short utterance case. on the other hand, ctc and transducer are shown to be suboptimal in dealing with non-monotonic sequence mapping @xcite , though some initial attempts show encouraging progress @xcite . in this work, we propose a hybrid transducer and aed model (taed), which integrates both aed and transducer models into one framework to leverage strengths from both modeling methods. in taed, we share the speech encoder between aed and transducer. the predictor in transducer is replaced with the decoder in aed. the aed decoder output assists the transducer's joiner to predict the output tokens. transducer and aed models are treated equally and optimized jointly during training, while only transducer's joiner outputs are used during inference. we extend the taed model to streaming applications under the chunk-based synchronization scheme, which guarantees full coverage of read/write choices in the training set and removes the training and inference discrepancy. the relationship between streaming latency and aed alignment is studied, and a simple, fast aed alignment is proposed to achieve low latency with small quality degradation. the new approach is evaluated in asr and st tasks for offline and streaming settings. the results show the new method helps to achieve new state-of-the-art results on offline evaluation. the corresponding streaming extension also improves the quality significantly under a similar latency budget. to summarize, our contributions are below: 1. taed, the hybrid of transducer and aed modeling, is proposed for speech-to-text tasks 2. a chunk-based streaming synchronization scheme is adopted to remove the training and inference discrepancy for streaming applications 3. a simple, fast aed alignment is employed to balance taed latency and quality 4. the proposed method achieves sota results on both offline and streaming settings for asr and st tasks ## preliminary formally, we denote a speech-to-text task training sample as a (x, y) pair. @xmath0:t and @xmath1:u are the speech input features and target text tokens, respectively. t and u are the corresponding sequence lengths. y @xmath2. the objective function is to minimize the negative log likelihood log p(y|x, θ) over the training set in the streaming setting, the model generates predictions at timestamps denoted by @xmath3, • • • , t u , • • • , t u ), rather than waiting to the end of an utterance, where t @xmath4. we call the prediction timestamp sequence as an alignment a between speech x 1:t and token labels y 1:u . a u @xmath5:t and y 1:u . the streaming model parameter θ s is optimized through the offline modeling can be considered a special case of streaming modeling, i.e., the alignment is unique with all t @xmath6. the following two subsections briefly describe two modeling methods used in our hybrid approach. ## methods in this study, we choose the transformer-transducer (t-t) @xcite as the backbone in the proposed taed system. for the streaming setting, the speech encoder is based on the chunkwise implementation @xcite , which receives and computes new speech input data by chunk size n instead of one frame each time. ## comparison of streaming algorithms when to read new input and write new output is a fundamental question for the streaming algorithm. based on the choices of streaming read/write policies, they can roughly be separated into two families: pre-fixed and adaptive. the pre-fixed policy, such as wait-k @xcite , adopts a fixed scheme to read new input and write new output. on the other hand, the adaptive algorithms choose read/write policies dynamically based on the input speech data presented. the adaptive algorithms could be further separated into two categories based on input and output synchronization. the first category of adaptive streaming algorithms is based on the aed framework, including hard monotonic attention (hma) @xcite , milk @xcite , mocha @xcite , mma @xcite and continuous integrate-and-fire (cif) (dong and xu, 2020; chang and yi @xcite . those methods extract acoustic information from the encoder outputs via attention between the encoder and decoder. the acoustic information is fused with linguistic information, which is estimated from the decoded token history, within the decoder. there is no explicit alignment between the input and output sequence; in other words, the outputs are asynchronized for the inputs. as discussed in §2.1, aed models don't fit the streaming application easily, and approximations have been taken during training. for example, the alignmentdependent context vector extracted via attention between the encoder and decoder is usually replaced by a context vector expectation from alignments. it differs from inference, which is based on a specific alignment path sampled during decoding. hence a training and inference discrepancy is inevitable, potentially hurting the streaming performance. the second category of adaptive streaming methods is with synchronized inputs and outputs, in which every output token is associated with a speech input frame. this includes ctc, transducer, caat, and the proposed taed. they combine acoustic and linguistic information within the joiner if linguistic modeling is applied. specific read/write decisions are not required during training. this considers all alignments and is optimized via ctc loss or rnn-t loss. hence, there is no training and inference discrepancy. the detailed comparison of different methods is listed in ## related work given the advantages and weaknesses of aed and ctc/ transducer, many works have been done to 1,000 1,200 1,400 1,600 26 28 30 average lagging (ms) bleu caat (o) t-t (o) taed (o) caat t-t taed figure 4: bleu (↑) v.s. average lagging (↓) on the must-c @xmath7.4) 1,000 1,200 1,400 1,600 1,800 2,000 2,200 20 22 24 26 average lagging (ms) bleu caat (o) t-t (o) taed (o) caat t-t taed figure 5: bleu (↑) v.s. average lagging (↓) on the must-c @xmath8.2) transducer with attention @xcite , which is a transducer variant, also feeds the encoder outputs to the predictor. our method is different in two aspects. first, we treat the taed as a combination of two different models: transducer and aed. they are optimized with equal weights during training, while transducer with attention is optimized with rnn-t loss only. it is critical to achieve competitive results as shown in another solution is to combine those two methods through a two-pass approach @xcite @xcite . the first pass obtains a set of complete hypotheses using beam search. the second pass model rescores these hypotheses by combining likelihood scores from both models and returns the result with the highest score. an improvement along this line of research replaces the two-pass decoding with single-pass decoding, which integrates scores from ctc/transducer with aed during the beam search @xcite . however, sophisticated decoding algorithms are required due to the synchronization difference between two methods. they also lead to high computation cost and latency @xcite . furthermore, the two-pass approach doesn't fit streaming applications naturally. heuristics methods such as triggered decoding are employed @xcite . in our proposed solution, two models are tightly integrated with native streaming support, and taed predictions are synergistic results from two models. ## conclusion in this work, we propose a new framework to integrate transducer and aed models into one model. the new approach ensures that the optimization covers all read/write paths and removes the discrepancy between training and evaluation for streaming applications. taed achieves better results than the popular aed and transducer modelings in asr and st offline tasks. under the streaming scenario, the taed model consistently outperforms the transducer baseline in both the en asr task and @xmath9. it also excels the sota streaming st system (caat) in medium and large latency regions.