End-to-End Speech Translation for Code Switched Speech (2022)

Abstract: Code switching (CS) refers to the phenomenon of interchangeably using words and phrases from different languages. CS can pose significant accuracy challenges to NLP, due to the often monolingual nature of the underlying systems. In this work, we focus on CS in the context of English/Spanish conversations for the task of speech translation (ST), generating and evaluating both transcript and translation. To evaluate model performance on this task, we create a novel ST corpus derived from existing public data sets. We explore various ST architectures across two dimensions: cascaded (transcribe then translate) vs end-to-end (jointly transcribe and translate) and unidirectional (source -> target) vs bidirectional (source <-> target). We show that our ST architectures, and especially our bidirectional end-to-end architecture, perform well on CS speech, even when no CS training data is used.

## introduction over half of the world's population is estimated to be bilingual. 2 those that know multiple languages are prone to code switch, i.e., to interchangeably use words and phrases from two (or more) languages in situations such as casual dialog, while traveling abroad, or simply to use a word they find more fitting @xcite . in cs, the base language is referred to as the matrix language while the contributing language is called the embedded language @xcite , where speakers often use the matrix language the majority of the time. code switched language is challenging to both automatic speech recognition (asr) and machine translation (mt) -and therefore also to the composite task of speech translation (st). while a rich 1 we make instructions and extra data needed to construct our cs data set available at https://github.com/apple/ml-codeswitched-speech-translation 2 bbc: https://bbc.in/3jgwzz2 * work done as an intern. ## related work code-switching in nlp has seen a rise of interest in recent years, including a dedicated workshop starting in 2014 @xcite and still ongoing @xcite . cs in machine translation also has a long history @xcite @xcite @xcite , but has seen a rise of interest with the advent of large multilingual models such as mbart @xcite or mt5 @xcite @xcite . due to the lack of available cs data and the ease of single-word translation, most of these recent related mt works have synthetically created cs data for either training or testing by translating one or more of the words in a sentence @xcite @xcite . we differ from those works by using naturally occurring cs data (section 3) which models the real-world cs distribution rather than arbitrary language mixing. for spoken input, as present in asr and st, synthetically creating realistic cs data is more challenging than it is for mt. however, dedicated asr corpora that contain natural cs exist, including the bangor miami @xcite , seame @xcite , and the recent largescale asru 2019 task @xcite . these corpora generally do not contain translations of the asr annotations, since they were designed for the asr task only. however, there exist two exceptions, which we leverage to derive our st cs data set, described in section 3. there also exists a wide range of prior modeling work on cs in asr models, for a variety of strategies @xcite @xcite @xcite @xcite . however, the recently introduced large multilingual models for speech, such as wav2vec, wav2vec 2.0, @xcite ; @xcite and hubert @xcite , are still underexplored with regards to their cs performance. handling mixed languages also requires understanding what languages are being spoken. systems that support mixed language input therefore require some form of automatic lid -either as an explicit component on the utterance @xcite or word-level @xcite , or implicitly learned by the underlying model(s) via a multi-task learning setup @xcite @xcite . in our work, we leverage both, exploring utterance-level lid components as well as implicit learning of utterance and word level lid. in both mt and asr, prior publications have also included the study of intra-word mixing of languages @xcite , a phenomenon we do not explore in our work. finally, our work builds off of advances made by @xcite ; @xcite that show that combining large multilingual speech and text models provide consistent improvements. we differ however, by exploring st in the novel cs setting. ## conclusion in this work, we expand the st literature to explore code-switching, contributing a new task framework for st that extends the joint transcription and translation setup. to further progress, we built and open-sourced a new st corpus for cs from existing public datasets. we evaluated a range of models, showing that using bilingual joint decoders provides gains over using separate task decoders. we also showed that e2e systems provide better performance than their cascading counterparts on the cs task. overall, our work shows that st models can perform well on cs applications with both no fine-tuning and in low-resource settings, opening the door to new and exciting areas of future work. selected from running a search over {0.01, 0.005, 0.001, 0.0005, 0.0001, 0.0005}. we found that 0.0005 was best for all models, so we examined learning rates again between 0.0001 to 0.001 (by 0.0001) and found that they all performed similarly, thus we use 0.0005 in our experiments. for efficiency in batch size while training, we removed all instances whose audio length was longer than 20 seconds. we freeze the attention layers for the first 500 * (64 / batch size) steps, which is approximately the first epoch of training. we initially trained the models on only covost and must-c and found there was a large domain shift between these datasets and the comparatively noisier fisher and miami datasets. as domain shift was not the focus of this paper, we further trained the models on the fisher and miami monolingual training sets to reduce the effect of domain shift. as a sanity check of the effectiveness of our training, we also include scores in we evaluate using word error rate, character error rate, charcut, and bleu. as the models learn different punctuation techniques from a variety of sources, including must-c, covost, miami, and fisher, we remove all punctuation from the output before evaluating on the cs/mono test sets, in order to only measure scores on the content. for bleu, we use sacrebleu with parameters case.lc+numrefs.1+smooth.4.0+tok.13a.