End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation (2023)

Abstract: Conventional speech-to-text translation (ST) systems are trained on single-speaker utterances, and they may not generalize to real-life scenarios where the audio contains conversations by multiple speakers. In this paper, we tackle single-channel multi-speaker conversational ST with an end-to-end and multi-task training model, named Speaker-Turn Aware Conversational Speech Translation, that combines automatic speech recognition, speech translation and speaker turn detection using special tokens in a serialized labeling format. We run experiments on the Fisher-CALLHOME corpus, which we adapted by merging the two single-speaker channels into one multi-speaker channel, thus representing the more realistic and challenging scenario with multi-speaker turns and cross-talk. Experimental results across single- and multi-speaker conditions and against conventional ST systems, show that our model outperforms the reference systems on the multi-speaker condition, while attaining comparable performance on the single-speaker condition. We release scripts for data processing and model training.

## introduction speech translation (st) has seen wide adoption in commercial products and the research community @xcite due to its effectiveness in bridging language barriers. st aims to translate audio of source languages into text of the target languages. this problem was tackled by a cascaded approach that pipelines automatic speech recognition (asr) and machine translation (mt) over the last few decades @xcite @xcite . however, end-to-end speech translation (e2e-st) systems @xcite figure 1 : a two-speaker multi-turn conversational segment. previous work focuses on separated channels without considering cross-talks and speaker-turns (top). stac-st targets a more challenging scenario where multiple speakers converse with occasional cross-talks due to merged channels (bottom). despite significant recent advances in e2e-st @xcite , most st systems to date have focused on translating isolated speech utterances from monologue speech @xcite , read speech @xcite or prompted speech @xcite . being trained on single-turn utterances, these systems may lack the ability to handle real-life scenarios in which multiple speakers converse, and sometime overlap, in the same audio channel @xcite . in this work, we tackle the more challenging task of multi-speaker conversational st. we refer to it as multi-turn & multi-speaker (mt-ms), as opposed to single-turn, which most st systems implicitly assume. this is illustrated in figure 1 , where a "conversation" between two speakers recorded with separate channels (top) becomes more difficult to translate if the channels are merged (bottom), due to the introduction of speaker-turns and cross-talks. in particular, st with cross-talks and speaker-turns is difficult because speech content of different sentences is mixed up or switched. while mt-ms speech has been studied in asr @xcite , to the best of our knowledge, this is the first paper that investigates it in end-to-end st. we tackle mt-ms st with an approach we named speaker-turn aware conversational speech translation (stac-st). stac-st is a multi-task training framework that combines asr, st and speaker-turn detection using special tokens in a serialized labeling format. it is inspired by a recent speech foundation model, whisper @xcite , which jointly trains asr, x-to-english st, voice activity detection, and language identification with 680k hours of speech data using labeling-based multitask learning. our contributions are as follows: ## speaker-turn aware conversational speech translation (stac-st) this section describes our end-to-end multi-task learning model for multi-turn multi-speaker conversational st. ## experimental setup this section introduces the datasets and metrics we used for evaluation, as well as architecture and training details of stac-st. ## conclusions in this work, we present stac-st, an end-to-end system designed for single-channel multi-turn & multi-speaker speech translation that uses a multitask training framework to leverage both asr and st datasets. we demonstrate that stac-st generalizes to both standard pre-segmented st benchmarks and multi-turn conversational st, the latter being a more challenging scenario. stac-st is also shown to learn the task of speaker change detection, which helps multi-speaker st and asr. we investigate different aspects of stac-st, including the impact of model and data size, automatic segmentation for long-form conversational st, zero-shot multi-turn & multi-speaker st with-out specific training data. overall, this work sheds light on future work towards more robust conversational st systems that can handle speaker-turns and cross-talks. limitations 1. our primary test sets, fisher and call-home, have narrowly one translation direction (@xmath0. the only other public conversational st dataset we are aware of is mslt @xcite , but it only contains independent utterances, which is far from representing a realistic mt-ms use case. we call for more publicly available longform conversational st data under a friendly license. 2. due to the same limitation of publicly available datasets, we do only explore conversations between two speakers. 3. we segment the test sets based on human annotations. despite being the best choice for the mt-ms data in our study ( §5.3.1), it is not a realistic scenario for testing. we leave improving segmentation on noisy long-form conversational audio as future work. 4. we segment long-form audio files into up to 30s pieces following @xcite , but we do not use the preceding segments as context. we focus on improving translation quality of conversations by speaker-turn and cross-talk detection, yet using the context information could also help. in addition, within each mt-ms segment, the inter-utterance context could have already been leveraged @xcite . we leave analysis of the interand intra-segment context as future work. 5. we only test the transformer architecture as we focus on solving a challenging mt-ms st task with multi-task learning, which is orthogonal to the architecture choice. we leave exploring other architecture options, such as conformer @xcite , hyperconformer @xcite or conmer @xcite as future work.