Fluent and Low-latency Simultaneous Speech-to-Speech Translation with Self-adaptive Training (2020)

Abstract: Simultaneous speech-to-speech translation is an extremely challenging but widely useful scenario that aims to generate target-language speech only a few seconds behind the source-language speech. In addition, we have to continuously translate a speech of multiple sentences, but all recent solutions merely focus on the single-sentence scenario. As a result, current approaches will accumulate more and more latencies in later sentences when the speaker talks faster and introduce unnatural pauses into translated speech when the speaker talks slower. To overcome these issues, we propose Self-Adaptive Translation which flexibly adjusts the length of translations to accommodate different source speech rates. At similar levels of translation quality (as measured by BLEU), our method generates more fluent target speech latency than the baseline, in both Zh<->En directions.

## introduction simultaneous speech-to-speech translation, which mimics the human interpreter's practice to translate the source speech into a different language with 3 to 5 seconds delay, has wide usage scenarios such as international conference meetings, traveling and negotiations as it provides more natural communication process than simultaneous speech-to-text translation. this task has been widely considered as one of the most challenging tasks in nlp with (but not limited to) following reasons: on one hand, the simultaneous translation is a hard task due to the word order difference between source and target languages, e.g., sov languages (german, japanese, etc.) and svo languages (english, chinese, etc.); on the other hand, simultaneous speech-to-speech translation escalates the challenge by considering the smooth cooperation between the modules of speech recognition, translation and speech synthesis. in order to achieve simultaneous speech-tospeech translation (ssst), to the best of our knowledge, most recent approaches @xcite dismantle the entire system into a three-step pipelines, streaming automatic speech recognition (asr) @xcite @xcite , simultaneous text-to-text translation (st2t) @xcite @xcite , and text-to-speech (tts) synthesis @xcite @xcite . most recent efforts mainly focus on st2t which is considered the key component to further reduce the translation latency and improve the translation quality for the entire pipeline. to achieve better translation quality and lower latency, there has been extensive research efforts which concentrate on the st2t by introducing more robust models @xcite , better policies @xcite @xcite , new decoding algorithms @xcite , or multimodal information @xcite . however, is it sufficient to only consider the effectiveness of st2t and ignore the interactions between other different components? furthermore, in practice, when we need to translate multiple sentences continuously, it is not only important to consider the cooperations between st2t and other speech-related modules, but also essential to take the effects between current and later sentence into consideration as shown in fig. 1 . unfortunately, all the aforementioned techniques ignore other speech modules and merely establish their systems and analysis on singlesentence scenario, which is not realistic. to achieve fluent and constant, low-latency ssst, we also need to consider speech speed difference between target and source speech. fig. 2 shows the speech rate distributions for both chinese and english in our speech corpus. the speech rate varies especially for different speakers. as shown in fig. 1 , when we have various source speech speed, the number of unnatural pauses and the latency vary dramatically. more specifically, when speaker talks slowly, tts often needs to make more pauses to wait for more tokens from st2t which usually does not output new translations with limited source information. these unnatural pauses lead to semantic and syntactic confusion @xcite . on the contrary, when speaker talks fast, the target speech synthesized from previous st2t models (e.g. wait-k) always introduce large latency which accumulates through the entire paragraph and causes significant delays.therefore, in realistic, the latency for the latter sentences are far more than the claimed latency in the original system. fig. 3 supports the above hypothesis when source side speech rate varies while using one wait-k translation model and itts model. to overcome the above problems, we propose self-adaptive translation (sat) for simultaneous speech-to-speech translation, which flexibly determines the length of translation based on different source speech rate. as it is shown in figure 6 , within this framework, when the speakers talk very fast, the model is encouraged to generate abbreviate but informative translation. hence, as a result of shorter translation, the previous translation speech can finish earlier and alleviate their effects to the latter ones. similarly, when the speakers have slower speech rate, the decoder will generate more meaningful tokens until a natural speech pause. the speech pauses can be understood as a natural boundary between sentences or phrase which does not introduce ambiguity to the translation. in conclusion, we make the following contributions: ## preliminaries in this section, we first introduce each component of three-step pipeline, which are streaming asr, simultaneous translation models and incremental tts techniques. ## self-adaptive translation to overcome the above practical problems, we propose self-adaptive translation (sat) technique to enable the ability of adjusting the length of the translation based on the demand of latency and fluency. we first demonstrate the problem of one naive solution. then, we introduce our training framework and talk about how to apply this technique in practice during inference time. ## paragraph-based boundary aware latency as mentioned frequently above, latency is another essential dimension for simultaneous speech-tospeech translation performance. however, the measurement of speech delay from source speech to each synthesized word in target speech is challenging and there is no direct metric that is suitable for simultaneous speech-to-speech translation. human interpreters use ear-voice-span (evs) @xcite to calculate translation delay for some landmark words from source speech to target source. however, this requires the target-to-source word correspondence. in practice, the translation model sometimes makes errors during translation which includes miss translation of some words or over translated some words that source does not include. thus, an automatic fully word-to-word alignment between target and source is hard to be accurate. human annotation is accurate but expensive and not practical. inspired by @xcite who proposed translation lag (tl) to ignore the semantic correspondence between words from target to source side and only calculate each target delay proportionally to each source words regardless the actually meaning of word in the task of simultaneous "speech-to-text" translation, we use a similar method to calculate the latency for each sentence. nevertheless, tl is only designed for singlesentence latency ,while we need to measure the latency of a paragraph of speech. thus, we pro-pose paragraph based boundary aware latency (pbal) to compute the latency of long speech simultaneous translation. in pbal, we first align the each sentence, make each word's correspondence within the sentence boundary. then we compute the time differences of the finished time between each target word's audio and its proportion corresponding source word's finish time in source side. in experiments, we determine the finish time of each source and target words by forced aligner @xcite and align the translation and source speech by using the corresponding streaming asr as a bridge. ## conclusions we proposed self-adaptive translation for simultaneous speech-to-speech translation which flexibly adjusts translation length to avoid latency accumulation and unnatural pauses. in both zh↔en directions, our method generates fluent and low latency target speeches with high translation quality.