Code-Switched Text Synthesis in Unseen Language Pairs (2023)

Abstract: Existing efforts on text synthesis for code-switching mostly require training on code-switched texts in the target language pairs, limiting the deployment of the models to cases lacking code-switched data. In this work, we study the problem of synthesizing code-switched texts for language pairs absent from the training data. We introduce GLOSS, a model built on top of a pre-trained multilingual machine translation model (PMMTM) with an additional code-switching module. This module, either an adapter or extra prefixes, learns code-switching patterns from code-switched data during training, while the primary component of GLOSS, i.e., the PMMTM, is frozen. The design of only adjusting the code-switching module prevents our model from overfitting to the constrained training data for code-switching. Hence, GLOSS exhibits the ability to generalize and synthesize code-switched texts across a broader spectrum of language pairs. Additionally, we develop a self-training algorithm on target language pairs further to enhance the reliability of GLOSS. Automatic evaluations on four language pairs show that GLOSS achieves at least 55% relative BLEU and METEOR scores improvements compared to strong baselines. Human evaluations on two language pairs further validate the success of GLOSS.

## introduction code-switching, the linguistic phenomenon of using more than one language within a single utterance or conversation, 1 is a common expression of multilingualism in informal text and speech @xcite . to accommodate the needs of multicultural and multilingual societies and individuals, there is a growing interest in investigating models dedicated to code-switching training en-es a superconductor levitates on a magnetic track un superconductor levita sobre a magnetic track inference model there is no such thing as a perfect budget 不可能有完美的budget es gibt kein perfect budget en-hi its height is only 1982 meters इसकी ऊ ँ चाई क े वल 1982 meter है ## problem formulation our goal is to synthesize code-switched (cs) texts for language pairs where their cs examples are never provided during training. given a monolingual input sentence x e in language l e and an assigned language l m (l m ̸ = l e ), we aim to generate a sentence x m,e that mixes l m and l e , while preserving the semantic meaning of x e . 4 we consider the setting where the assigned language l m in the testing time is different from those in the training time. more formally, as illustrated in figure 1 , the training set consists of n language pairs (l en , l mn )(@xmath0, 2, ..., n }), while training corpus for code-switching ada pter its height is only 1982 meters encoder इसकी hi_in decoder encoder decoder stage 1 : pre-trained multilingual machine translation model stage 2 : fine-tune on training data for code-switching इसकी its height is only 1982 meters encoder इसकी hi_in decoder इसकी prefix prefix hi_in es_xx fr_xx zh_cn …. en_xx ja_xx …. q k v adapter adapter q k v transformer block transformer block en-es en-hi en-fr sample the testing set includes target language pairs where l mt / ∈ {l m 1 , ..., l m n }, ∀t. this scenario reflects real-world situations where code-switched data is more readily available for certain language pairs, such as spanish-english and hindi-english, while it is less accessible for others, such as bengali-english and swahili-english. ## method we introduce gloss, a generalized code-switched text synthesizer that tackles the two specific challenges raised by our problem setting: (1) the model needs to generate texts across many languages, some are not even in the cs training data; to address the first challenge, we begin by obtaining a pre-trained multilingual machine translation model (pmmtm) using multilingual machine translation data, which covers all languages that would be used for final cs text synthesis ( §3.1). 5 the remaining challenge is how to make pmmtm a code-switched text synthesizer with only limited language coverage of training data. we propose to augment an additional codeswitching module onto pmmtm, thereby creating gloss ( §3.2). this additional code-switching module is trained on our limited cs data while keeping pmmtm parameters fixed. instead of finetuning the entire pmmtm, this modularized design improves systematic generalization @xcite , where pmmtm focuses on generating translated sentences and the code-switching module concentrates on "mixing" languages. this approach allows gloss to be more adaptable and less prone to overfitting during the fine-tuning process on cs data. finally, we present a self-training framework that enables gloss to more stably generate cs texts in target language pairs ( §3.3). ## human evaluation to further verify the quality of our method, we conduct the human evaluation for hindi-english and chinese-english code-switched (cs) text using sentences in english as the source language. ## related work early approaches @xcite @xcite on code-switched (cs) text synthesis were built based on various linguistic theories, such functional head constraints @xcite , matrix-language theory @xcite , and equivalence-constraint theory @xcite . to turn linguistic theories into computational models, @xcite ; pratapa and choudhury (2021) leverage trained constituency parser to extract parses of translation pairs and create cs sentences by mixing translation pairs following the syntactic constraints derived from the theories. however, constraints cannot be postulated as a universal rule for all cs scenarios, especially for languages that are syntactically divergent (berk-seligson, 1986), such as english and chinese, since they have word alignments with an inverted order @xcite . owing to the limitation, more and more recent works start to build cs text synthesizers in a data-driven way. @xcite train a sequence generative adversarial model on real cs text to generate chinese-english cs sentences. @xcite build a cs text synthesizer using the generative adversarial network, while several follow-up works @xcite @xcite using different generative model techniques are also presented. more studies have been introduced to improve the synthesis quality such that we cannot exhaust them in this short summary. we refer readers to the recent survey @xcite . although many of these efforts had some success, the above-mentioned methods can only generate cs text in the same language pair sets used in training. given the difficulties of acquiring cs data, this requirement hinders the scalability of these models to support more language pairs. hence, in this paper, we take a step forward to explore the possibility of zero-shot transfer generalization in cs text synthesis and present gloss that can generate reasonable outputs. ## conclusion in this paper, we develop a novel generalized codeswitched text synthesizer, which can even generate code-switched sentences where the corresponding code-switched training data is unavailable. we introduce gloss that is built on top of a pre-trained multilingual machine translation model and augmented with an adapter or prefixes. the modularized design of learning specific parameters for mixing languages from a translated distribution helps the overall system generalization, hence, fulfilling our goal. extensive experiments verify our methods' effectiveness qualitatively and quantitatively. in the future, we plan to investigate how our synthesizer performs on downstream tasks such as conversational understanding under a code-switched scenario.