Label-Synchronous Neural Transducer for E 2 E Simultaneous Speech Translation (2024)

Abstract: While the neural transducer is popular for online speech recognition, simultaneous speech translation (SST) requires both streaming and re-ordering capabilities. This paper presents the LS-Transducer-SST, a label-synchronous neural transducer for SST, which naturally possesses these two properties. The LS-Transducer-SST dynamically decides when to emit translation tokens based on an Auto-regressive Integrate-and-Fire (AIF) mechanism. A latency-controllable AIF is also proposed, which can control the quality-latency trade-off either only during decoding, or it can be used in both decoding and training. The LS-Transducer-SST can naturally utilise monolingual text-only data via its prediction network which helps alleviate the key issue of data sparsity for E2E SST. During decoding, a chunk-based incremental joint decoding technique is designed to refine and expand the search space. Experiments on the Fisher-CallHome Spanish (Es-En) and MuST-C En-De data show that the LS-Transducer-SST gives a better quality-latency trade-off than existing popular methods. For example, the LS-Transducer-SST gives a 3.1/2.9 point BLEU increase (Es-En/En-De) relative to CAAT at a similar latency and a 1.4 s reduction in average lagging latency with similar BLEU scores relative to Wait-k.

## introduction simultaneous speech translation (sst) generates translations from input speech in a streaming fashion. conventional cascaded sst performs streaming automatic speech recognition (asr) followed by text-based simultaneous machine translation @xcite . recently, end-to-end (e2e) sst has become popular and has advantages, including lower latency @xcite . however, e2e sst is challenging since it requires taking into account word re-ordering between source and target languages during the streaming process @xcite . neural transducers, the dominant model for low-latency asr, find this difficult due to their monotonic nature. furthermore, e2e training results in severe data sparsity @xcite . current sst methods are normally based on the transformer @xcite attention-based encoder-decoder (aed) structure, which isn't naturally able to deal with streaming, as noted by @xcite . a popular approach to adapt the aed to sst is the wait-k policy @xcite , which is a fixed read-write policy that uses a fixed number of wait duration steps before translation. however, it can be too aggressive or conservative in different cases @xcite . alternative methods involve a flexible policy which lets the model decide how much input to read before generating the next translation token @xcite , such as monotonic multi-head attention (mma) @xcite and continuous integrate-and-fire (cif) @xcite based sst system. however, these flexible policies normally rely on a latency loss to adjust the quality-latency trade-off at training, unlike the fixed wait-k policy that can control latency at decoding only. to better address the challenges of e2e sst, this paper adapts the label-synchronous neural transducer @xcite developed for asr to sst and denotes the resulting technique the ls-transducer-sst. in the ls-transducer-sst, an auto-regressive integrate-and-fire (aif) mechanism uses accumulated frame-level weights to dynamically determine when to emit translation tokens, based on which a label-level target-side encoder representation is extracted auto-regressively using an attention mechanism. therefore, the ls-transducer-sst is naturally equipped with both streaming and re-ordering capabilities. in addition, the prediction network of the ls-transducer-sst works as a standard language model (lm) as its output is directly combined with the extracted en-coder representation at the label level. as a benefit, the e2e sst data sparsity issue can be alleviated because the prediction network can effectively utilise monolingual text-only data, which is normally easy to collect, for tasks such as pre-training or text-based adaptation. while the standard aif theoretically ensures low-latency output for the ls-transducer-sst, to better control the qualitylatency trade-off, a latency-controllable aif is proposed, which controls the latency by adjusting the decision threshold of the accumulated frame-level weights. furthermore, the latency-controllable aif allows the quality-latency trade-off to be controlled not only during training but also during decoding, enabling the ls-transducer-sst to combine the advantages of typical fixed and flexible sst policies. this paper focuses on low/medium-latency scenarios to keep the low-latency advantage of e2e sst. during decoding, to improve translation quality, a chunk-based incremental joint decoding is further proposed to refine and expand the search space. the proposed ls-transducer-sst was evaluated on fisher-callhome spanish (es-en) and must-c en-de corpora and gave an improved qualitylatency trade-off compared to existing popular sst methods. the main contributions are summarised below: ## proposed method: ls-transducer-sst the ls-transducer-sst, as shown in fig. 1 , has four key components: encoder, aif, prediction network, and joint network. the encoder and aif extract label-level target-side representations (h aif 1 , • • • , h aif l ) from the source-language speech in a streaming fashion, while aif controls the time steps at which the representations are emitted. in addition, the prediction network is an autoregressive structure, e.g. transformer lm structure, which generates representations (h pred 1 , • • • , h pred l ) based on previous translation tokens. since the representations obtained from aif and the prediction network have the same length, the joint network can directly add the logits obtained from them using linear fully-connected layers, and the prediction network performs as an explicit lm. hence, the output of the ls-transducer-sst is a 2-dimensional matrix r l•v (v is the vocabulary size), which can use a cross-entropy loss computed with target translations for training and resolve the issue of expensive training for the standard neural transducer, whose output is a 3-dimensional tensor. ## experimental results the ls-transducer-sst was compared to fixed and flexible policy models. to maintain the lowlatency advantage of e2e sst, we focus on the low and medium latency scenarios, i.e. @xmath0., 2020). ## conclusions this paper proposes the ls-transducer-sst, which naturally possesses streaming and re-ordering capabilities. by introducing the target-side ctc branch, the re-ordering capability of the label-synchronous neural transducer is enhanced and thus the aif mechanism can be used for sst. therefore, the ls-transducer-sst is a flexible policy method that dynamically decides when to emit translation tokens using the aif. a latency-controllable aif is further proposed to effectively control the latency at decoding or training, enabling the ls-transducer-sst to combine the advantages of typical fixed and flexible sst policies. in addition, the ls-transducer-sst provides a natural way to utilise monolingual textonly data, which helps alleviate the e2e sst data sparsity issue. during decoding, a chunk-based incremental joint decoding is further proposed to refine and expand the search space. with the focus on low and medium-latency scenarios, experiments showed that the ls-transducer-sst gives a better quality-latency trade-off than existing methods.