C o V o S witch: Machine Translation of Synthetic Code-Switched Text Based on Intonation Units (2024)

Abstract: Multilingual code-switching research is often hindered by the lack and linguistically biased status of available datasets. To expand language representation, we synthesize code-switching data by replacing intonation units detected through PSST, a speech segmentation model fine-tuned from OpenAI’s Whisper, using a speech-to-text translation dataset, CoVoST 2. With our dataset, CoVoSwitch, spanning 13 languages, we evaluate the code-switching translation performance of two multilingual translation models, M2M-100 418M and NLLB-200 600M. We reveal that the inclusion of code-switching units results in higher translation performance than monolingual settings and that models are better at code-switching translation into English than non-English. Further, low-resource languages gain most from integration of code-switched units when translating into English but much less when translating into non-English. Translations into low-resource languages also perform worse than even raw code-switched inputs. We find that systems excel at copying English tokens but struggle with non-English tokens, that the off-target problem in monolingual settings is also relevant in code-switching settings, and that models hallucinate in code-switching translation by introducing words absent in both of the original source sentences. CoVoSwitch and code are available at https://github.com/sophiayk20/covoswitch.

## introduction code-switching (csw), otherwise known as codemixing, refers to the use of linguistic units from multiple languages in a conversation or utterance @xcite . in general, researching codeswitching comprehensively is a complicated task due to the lack of code-switched data. one solution is to use existing code-switching datasets 1 covoswitch is released as a huggingface dataset. https: //huggingface.co/datasets/sophiayk20/covoswitch . @xcite , but there is a limited number of such datasets and using them constrains research to the few language pairs that datasets are concentrated in, such as spanish-english or hindi-english @xcite . to alleviate the problem, previous work @xcite brought together multiple datasets, such as fisher @xcite and bangor miami @xcite . nevertheless, in the multilingual setting, collecting data from multiple sources mixes different degrees of codeswitching and blocks parallel understanding across languages. alternatively, most works have introduced synthetic datasets @xcite . these have been based on linguistic theories, such as the matrix language frame (mlf) model @xcite and the equivalence constraint @xcite . applying the equivalence constraint requires the use of constituency parsers. @xcite utilized the stanford parser @xcite and the berkeley neural parser @xcite . however, as of now, the stanford parser supports arabic, chinese, english, french, german, and spanish, while the berkeley neural parser supports arabic, basque, english, french, german, hebrew, hungarian, korean, polish, and swedish. this presents a bottleneck in the number of languages that can be used for research and impedes the creation of code-switching data for unsupported or low-resource languages such as tamil. synthetic datasets have also introduced codeswitching mainly based on words. these include random replacements based on words @xcite @xcite and replacements based on connected components of aligned words @xcite . however, word-based switching may not completely reflect the code-switching phenomenon. recent research @xcite demon-strated that code-switching is more common across intonation units than within as a result of looser syntactic relationships and that intonation units should therefore serve as new replacement units instead of words. this constraint is referred to as the intonation unit boundary constraint. to expand language representation, experiment with intonation units as basis units of codeswitching, and reflect both linguistic and prosodic constraints, we synthesize data by following the matrix language frame model and the intonation unit boundary constraint. we keep english as the matrix language and embed segments from non-english languages by replacing english intonation units of utterances from covost 2 @xcite , a speech-to-text translation (s2tt) dataset, detected with psst @xcite , an english prosodic speech segmentation model fine-tuned from openai's speech recognition model whisper @xcite . utilizing s2tt datasets is advantageous for several reasons. first, they include transcripts for both languages and audio files for one language in each pair, which allows the simultaneous incorporation of text and speech features in code-switching data creation. moreover, recent datasets cover a multitude of high-resource and low-resource languages, which enables the inclusion of diverse language pairs for synthetic codeswitching data. meanwhile, we observe that while recent works @xcite have demonstrated the translation performance of multilingual large language models with billions of parameters such as xglm-7.5b and bloomz-7b1 on codeswitching data, performance of multilingual neural machine translation (mnmt) models with millions of parameters remains relatively underexplored. we therefore measure the zero-shot code-switching translation performance of m2m-100 418m @xcite and @xcite , capable of multilingual translation for 100 and 200 languages respectively, on our synthetic dataset. our contributions are summarized as follows: we (1) apply a single synthetic data generation method to different language pairs, including lowresource languages such as tamil, based on a single dataset and thereby eliminate differences that emerge from the discrepancies in data generation methodology, (2) release a new code-switching dataset, covoswitch, with similar code-switching levels across 13 languages, and (3) compare trans2 synthetic data generation ## machine translation experimental setup models. we use the huggingface pre-trained model checkpoints facebook/m2m100_418m and facebook/nllb-200-distilled-600m for the m2m-100 418m and nllb-200 600m models. these two models were chosen for their exceptional multilingual capabilities, with m2m-100 intended for non-english centric translation and nllb-200 designed to improve translation performance in low-resource languages. both support all languages covered by our synthetic dataset. translation settings. we experiment with four translation settings for each of the english and non-english language pairs. first is @xmath0, in which code-switched text is translated into english. this setting was examined in previous research @xcite ), but we also experiment with @xmath1. we compare these two codeswitching translation settings to two monolingual translation settings, @xmath2, where x is a non-english language and en is english. baselines. our baselines are twofold. first, we compare code-switching translations with monolingual translations and interpret deltas from monolingual baselines as the gains or losses from introducing code-switching units. we set our second baseline in consideration of our synthetic code-switched inputs. because synthetic code-switched inputs already contain segments from reference texts, evaluation scores for these may be higher than translations of solely monolingual texts. in light of this, we consider deltas from raw code-switched inputs the performance of systems in translating code-switched text. evaluation metrics. we measure the performance of translation models with the following automatic metrics: chrf++ @xcite at the character level, spbleu @xcite at the languageagnostic subword level tokenized through sentence-piece @xcite , and comet @xcite at the detokenized representation level. spbleu and chrf++ measure similarity between reference translation and system translation, while comet predicts human judgments of system translations based on a neural model. we use the flores-200 (costa-jussà et al., 2022) tokenizer available through sacrebleu @xcite for spbleu and unbabel/wmt22-comet-da @xcite for comet calculation. we supplement chrf++, spbleu, and comet with copy and replacement rates to examine whether translation systems can perform implicit language identification to copy or replace tokens as appropriate. as in @xcite @xcite , we define copy rate as the rate at which the target tokens already present in code-switched input is successfully transferred over to the machine translation system output. we define replacement rate as the rate at which the system successfully converts non-target input tokens to target tokens. it follows that lower replacement rates indicate less translated outputs. all experiments are conducted on a single nvidia l4 gpu. ## conclusion in this work, we present covoswitch, a codeswitching dataset created by replacing intonation units detected by psst, a speech segmentation model fine-tuned from whisper, on covost 2, a speech-to-text translation dataset. using cov-oswitch, we examine the performance of two mnmt models with millions of parameters, m2m-100 418m and nllb-200 600m, and compare code-switching translations against monolingual translations and high-resource languages against low-resource languages. we discover that the introduction of code-switching units results in higher performing translations compared to monolingual settings and that models are better at codeswitching translation into english than into non-english. meanwhile, low-resource languages gain most from monolingual baselines compared to other languages in @xmath3. systems also exhibit poor translation abilities in low-resource @xmath4. additionally, we find that models struggle to copy non-english tokens, identify the off-target problem in code-switching settings, and confirm that models hallucinate in codeswitching translation by creating words nonexistent in the original source sentences. by releasing cov-oswitch, we aim to support the inclusion of a wider variety of languages in code-switching research.