Towards Real-World Streaming Speech Translation for Code-Switched Speech (2023)

Abstract: Code-switching (CS), i.e. mixing different languages in a single sentence, is a common phenomenon in communication and can be challenging in many Natural Language Processing (NLP) settings. Previous studies on CS speech have shown promising results for end-to-end speech translation (ST), but have been limited to offline scenarios and to translation to one of the languages present in the source monolingual transcription). In this paper, we focus on two essential yet unexplored areas for real-world CS speech translation: streaming settings, and translation to a third language (i.e., a language not included in the source). To this end, we extend the Fisher and Miami test and validation datasets to include new targets in Spanish and German. Using this data, we train a model for both offline and streaming ST and we establish baseline results for the two settings mentioned earlier.

## introduction speech technologies are one of the main applications of machine learning, and are currently deployed in many real-world scenarios. to ensure a adequate user experience, factors other than accuracy need to be taken into account. one of them is the ability to produce an output in real-time (streaming settings) with a low latency and another one is effectively handling the distinctive characteristics inherent in spoken language, like code-switching. code-switching (cs) is the phenomenon in which a speaker alternates between multiple languages in a single utterance. due to globalization @xcite , it is becoming increasingly prevalent in spoken language, not only in bilingual communities but also in monolingual communities. cs presents a challenge in various natural language processing (nlp) settings, such as automatic speech recognition (asr), machine translation (mt), and speech translation (st), due to the inherent complexity of dealing with two source languages, as well as the scarcity of cs training and test data @xcite . despite the relevance of st for cs speech task, the available literature on the subject is rather limited. @xcite investigate the task defined as monolingual transcription, i.e. transcribing a cs utterance using words of only one language, hence translating those words that are cs. their work proposes and compares different approaches to evaluate the stated task in japanese-english cs to english. other follow-up work takes a similar approach (see section 2). to date, however, certain essential topics, such as translation to a language not present in the cs source or streaming st, have yet to be explored, despite its critical importance for real-world usage. the primary challenge in translating to a third language stems from the unavailability of datasets with such characteristics. furthermore, streaming settings present further challenges: achieving a balance between latency, stability and accuracy is crucial for delivering a seamless user experience, as with any streaming task. besides, cs tasks may require more context than monolingual ones because of the added complexity of language mixing. thus, addressing the trade-offs between these metrics in cs streaming st may prove to be more intricate than with monolingual data. in our work, we resolve the two aforementioned challenges: first, the insufficiency of data and results for translation to a third language, and second, the absence of a baseline for streaming cs st. to alleviate the data scarcity in cs tasks, we extend fisher @xcite and bangor miami cs @xcite datasets (combined english and spanish source and english targets) by incorporating spanish and german targets in the test and validation sets. 1 these additions allow us to evaluate the performance of our models on monolingual transcription (translation to english or spanish), but also for the first time in cs st into a third language (german) setting baseline results. furthermore, this study is the first on streaming st for cs speech, and examines errors in transcripts generated by both offline and streaming models, considering different latency and flickering constraints, and different training techniques such as prefix-sampling. we show that prefix-sampling does not improve the model performance, and that errors in cs points appear in the same proportion streaming and offline st. our work sets baseline results and provides insight into the impact of cs on the performance of different models, and helping to identify potential points for future research that can contribute to the advancement of the field. to sum up, the main contributions of our work are: ## related work during the past few years, there has been an increasing interest in cs tasks. prior work has focused in @xcite @xcite and asr @xcite @xcite @xcite . however, the topic of cs in st has been relatively under-explored, and usually concentrating only on monolingual transcription @xcite @xcite , and relying on synthetically generated data @xcite . the first work on cs st was done by @xcite . the authors analyse different architectures and training configurations for japanese-english cs to english monolingual transcription. @xcite present a similar work but in a different language pair. the authors present a cs dataset with natural english-spanish cs text and speech sources and english text targets, gathering cs sentences in fisher and bangor miami datasets. with these data, they are able to evaluate asr and transformer encoder transformer decoder or embedded text & source @xmath0: proposed model architecture. the multimodal encoder supports training on both speech translation and text translation data. the tagging scheme is designed to allow generating either the (code-switched) transcript or a (monolingual) translation. st, although the st setting is actually monolingual transcription. the authors explore different architectures through a two-steps training: a pretraining on non-cs data and a fine-tuning on cs data. they find that end-to-end st models obtain higher accuracy than cascaded ones and that accuracy on cs test sets improves after the fine-tuning step without noticeably impacting performance on non-cs sets. later, hamed et al. (2022) present a corpus for egyptian arabic-english cs tasks. the dataset contains text and speech cs sources, and targets in monolingual english and egyptian arabic. by combining these sets the authors are able to study asr (from cs speech to cs text), as well as mt and st. however, because of the target languages, both the st and mt settings are actually monolingual transcription and a text-to-text variant of this task. finally, @xcite present last, a language-agnostic model for st and asr that aims to replace acoustic language id gated pipelines by a unique cs model. however, their work focuses on inter-sentential cs (when a cs happens just at sentence boundaries) using synthetic data. ## model we adopt the multimodal model design proposed by @xcite for speech translation (figure 1 ). this model supports speech transcription, speech translation, and text translation, and leverages paired data of all three tasks through multitask training. similar to @xcite , we extract speech representations using a pretrained wav2vec 2.0 base model @xcite 2 which results in 20ms per frame. to compute downsampled speech representations, wav2vec 2.0 applies a stack of three convolutional layers, resulting in 160ms per frame: each layer has a kernel of 3 and a stride of 2. to extract text representations for multitask text-to-text training, we simply use a 1024-dimensional embedding layer. next we attach an encoder-decoder transformer @xcite with pre-layer normalization, a hidden dimension of 1024, dropout of 0.1, five encoder layers and three decoder layers. the input to the encoder is either the downsampled speech representations, or the embedded source text. in the decoder, we use 1024-dimensional lstms @xcite instead of self-attention which obtained better results in preliminary investigations. the model is trained in a multi-task fashion, where we sum the losses of the transcription task, text translation task, speech translation task, as well as a ctc loss @xcite applied on top of the full encoder. tasks are weighted equally. importantly to our work, we use a shared decoder to perform either transcription or translation, with a language tag indicating the desired output language for st, or the @xmath1. note that the transcript will be equivalent to the translation in the source language for monolingual sentences, but a special token for transcripts is needed to account for cs sentences. to employ our model in a streaming setting, we use the re-translation technique @xcite . this technique retranslates the utterance to update its prior prediction as additional information is received. to control the trade-off between latency, flickering, and accuracy, we set a mask on the last k sub-words of the prior prediction, allowing the model to rewrite only that part of the output. therefore, a high k allows the model to rewrite the whole prediction, obtaining a high accuracy but poor latency and flickering scores, and on the contrary, setting @xmath2, hindering the accuracy but leading to no flickering and the lowest possible latency. section 5 contains experiments to obtain the appropriate k. bangor miami @xcite : the dataset contains recorded conversations between bilingual english/spanish speakers in casual settings, with a high proportion of naturally occurring code-switched speech. the recordings were obtained using small digital recorders worn on belts, resulting in low audio quality with background noise. we use the splits for cs st defined by @xcite . fisher @xcite : the dataset was collected for asr by pairing spanish speakers located in the us and canada through phone calls. although it is not a cs focused dataset, it contains a significant amount of cs utterances due to the speakers being in english-speaking contexts. the recording was done through phone recordings in 2004, which makes it a noisy asr dataset, although less noisy than miami. we use the splits for cs st defined by @xcite . covost (wang et al., 2020): a multilingual and diversified st datset based on the common voice project @xcite . this dataset includes language pairs from multiple languages into english, and it includes low resource languages. must-c (di gangi et al., 2019): a dataset for st research. it is a large-scale, multi-language dataset that includes speech recordings from english ted talks and corresponding human transcriptions and translations. the dataset covers translation from english to many languages. the recording context (ted talks) makes it a quality clean dataset. ## experiments our experiments follow four main directions: (1) finding a reasonable k to control re-translation flickering and latency, (2) studying the occurrence of errors around cs switching points, (3) analyzing the usefulness of prefix-sampling and (4) establishing baseline numbers for translation to a third language and for streaming tasks for cs speech, including transcription, monolingual translation, and translation. to evaluate our models we will use three different metrics. to measure the model accuracy we use bleu @xcite with sacrebleu @xcite and a beam size of 5. to evaluate the lag between model input and output we use average lag (al, @xcite ), and to measure the flickering we use normalized erasure @xcite ). additionally, we use wer to evaluate asr performance. ## conclusions in this work, we have tackled two open ends in cs st: translation to a third language and streaming settigns. to do so, we have trained offline and streaming models for direct translation and transcription of cs speech. furthermore, we have extended fisher and miami test and validation sets with new spanish and german targets. by doing this we have been able to analyse not only monolingual transcription, but also pure translation. we have observed a drop of up to 18 bleu points between the two settings, showcasing the importance of not relying on monolingual transcription when aiming for st models, as has been commonly done in previous work. given the greater complexity of translating to a third language as compared to monolingual translation, we think that incorporating additional data would be necessary to tackle the accuracy drop. however, since natural codeswitched data is limited and generating synthetic data is beyond the scope of this study, we leave this for future research. to summarize, our work presents new data, an in depth analysis of the impact of cs in the predictions, and results for streaming cs speech translation and translation to a third language, which can serve as a baseline for future work in a field that although relevant is still far from solved.