Learning Adaptive Segmentation Policy for End-to-End Simultaneous Translation (2022)

Abstract: End-to-end simultaneous speech-to-text translation aims to directly perform translation from streaming source speech to target text with high translation quality and low latency. A typical simultaneous translation (ST) system consists of a speech translation model and a policy module, which determines when to wait and when to translate. Thus the policy is crucial to balance translation quality and latency. Conventional methods usually adopt fixed policies, e.g. segmenting the source speech with a fixed length and generating translation. However, this method ignores contextual information and suffers from low translation quality. This paper proposes an adaptive segmentation policy for end-to-end ST. Inspired by human interpreters, the policy learns to segment the source streaming speech into meaningful units by considering both acoustic features and translation history, maintaining consistency between the segmentation and translation. Experimental results on English-German and Chinese-English show that our method achieves a good accuracy-latency trade-off over recently proposed state-of-the-art methods.

## introduction recent years have witnessed extensive studies and rapid progress of simultaneous translation (st). it aims to perform translation from source speech into the target language with high quality and low latency and is widely used in many scenarios, such as international conferences, press releases, etc. generally, the research of st falls into two categories: the cascaded method, and the end-to-end to overcome these limitations, the end-to-end method attempts to directly translate from source speech to target text, without explicitly transcribing the source speech @xcite @xcite . to balance the translation quality and latency, the key challenge lies in the segmentation policy that determines the translation boundaries of the speech frames. most of the previous work used fixed policies. some of them take fixed-length policy @xcite @xcite that splits speech at a fixed frequency, for example, to generate one target word every t s ms (figure 1 (a) ). other work adopts word-based policy that splits the speech into words and generates one target word whenever a new source word is detected, which calls for an auxiliary source word detector @xcite @xcite @xcite , see figure 1 (b). however, both the above methods are "hard" policies, which do not consider the contextual information and result in low translation quality @xcite . in this paper, we propose an adaptive segmentation policy for end-to-end simultaneous translation based on meaningful unit (mu). the idea is borrowed from human interpreters, who do interpretation based on a unit with clear meaning rather than fixed frame length or word. we model the speech segmentation policy as a binary classification that determines whether a speech segment is an mu. once an mu is detected, it is fed into an end-to-end speech translation model, as illustrated in figure 1 (c ). we propose a supervised training method, using both acoustic features and translation features to train the policy. besides, we propose an incremental decoding method to construct training data from speech and translation pairs. concretely, we first train a full speech translation model m st , and then gradually expand speech frames to simulate simultaneous translation. when the translation of the current speech segment is a prefix of the full speech translation, the segment is extracted as an mu. at inference time, we employ the same m st to maintain the consistency between segmentation and translation. our method is more flexible than fixed policies, as it dynamically detects meaningful units according to contextual information. experiments on two language pairs show that the proposed approach outperforms the strong baselines in balancing translation quality and latency. ## related work cascade simultaneous translation. to eliminate the impact of asr errors, most previous work of cascade st use golden transcript, rather than asr result, to explore different read/write policies in st. existing policies can be classified into two categories: 1) the fixed policy segments the source text based on fixed lengths @xcite . for example, wait-k @xcite is a typical fixed policy that first read k source words, then generates one target word immediately after receiving one subsequent source word. 2) the adaptive policy learns to segment the source text according to its context @xcite @xcite @xcite . it has been proven that the adaptive policy is more effective than the fixed policy in balancing translation quality and latency @xcite . ## adaptive speech segmentation policy the overall framework of our adaptive speech segmentation policy is illustrated in figure 2 . given a streaming speech s, we incrementally detect whether a speech clip @xmath0, 2, ...) is an mu, where @xmath1, and f is the detection interval. once an mu is detected, the speech translation model produces its translation y with the translation history y p force decoded as a translation prefix. meanwhile, y is displayed to users and added to the translation history y p to improve mu detection. in the following, we first introduce our mu detection model (section 3.1), then propose a method to construct mu training data (section 3.2). finally, we describe the training details in section 3.3. ## experiments we carry out experiments on english-german (en-de) and chinese-english (zh-en) simultaneous translation. we use sacrebleu @xcite to evaluate the translation performance and the acoustic average lagging (al) @xcite as the latency metric. the al measures the system lagging behind an "ideal policy", which produces translation at the same speed as the audio received. ## conclusion we present an adaptive speech segmentation policy for end-to-end simultaneous translation, which triggers translation with a meaningful speech unit de-tector. experiments across two language pairs show that our method outperforms state-of-the-art methods with constrained training corpus, suggesting the effectiveness of our adaptive policy. ablation studies reveal key factors that lead to its success, including tail-truncation, multi-modal segmentation, and speech-text pre-training. the statistics of the training data used in zh-en experiments is listed in