From Machine Translation to Code-Switching: Generating High-Quality Code-Switched Text (2021)

Abstract: Generating code-switched text is a problem of growing interest, especially given the scarcity of corpora containing large volumes of real code-switched text. In this work, we adapt a state-of-the-art neural machine translation model to generate Hindi-English code-switched sentences starting from monolingual Hindi sentences. We outline a carefully designed curriculum of pretraining steps, including the use of synthetic code-switched text, that enable the model to generate high-quality code-switched text. Using text generated from our model as data augmentation, we show significant reductions in perplexity on a language modeling task, compared to using text from other generative models of CS text. We also show improvements using our text for a downstream code-switched natural language inference task. Our generated text is further subjected to a rigorous evaluation using a human evaluation study and a range of objective metrics, where we show performance comparable (and sometimes even superior) to code-switched text obtained via crowd workers who are native Hindi speakers.

## introduction code-switching (cs) refers to the linguistic phenomenon of using more than one language within a single sentence or conversation. cs appears naturally in conversational speech among multilingual speakers. the main challenge with building models for conversational cs text is that we do not have access to large amounts of cs text that is conversational in style. one might consider using social media text that contains cs and is more readily available. however, the latter is quite different from conversational cs text in its vocabulary (e.g., due to the frequent use of abbreviated slang terms, hashtags and mentions), in its sentence structure (e.g., due to character limits in tweets) and in its word forms (e.g., due to transliteration being commonly employed in social media posts). this motivates the need for a generative model of realistic cs text that can be sampled to subsequently train models for cs text. in this work, we tackle the problem of generating high-quality cs text using only limited amounts of real cs text during training. we also assume access to large amounts of monolingual text in the component languages and parallel text in both languages, which is a reasonable assumption to make for many of the world's languages. we focus on hindi-english cs text where the matrix (dominant) language is hindi and the embedded language is english. 1 rather than train a generative model, we treat this problem as a translation task where the source and target languages are monolingual hindi text and hindi-english cs text, respectively. we also use the monolingual hindi text to construct synthetic cs sentences using simple techniques. we show that synthetic cs text, albeit being naive in its construction, plays an important role in improving our model's ability to capture cs patterns. we draw inspiration from the large body of recent work on unsupervised machine translation @xcite to design our model, which will henceforth be referred to as translation for code-switching, or tcs. tcs, once trained, will convert a monolingual hindi sentence into a hindi-english cs sentence. tcs makes effective use of parallel text when it is available and uses backtranslation-based objective functions with monolingual text. below, we summarize our main contributions: ## related work early approaches of language modeling for codeswitched text included class-based n-gram models @xcite , factored language models that exploited a large number of syntactic and semantic features @xcite , and recurrent neural language models @xcite for cs text. all these approaches relied on access to real cs text to train the language models. towards alleviating this dependence on real cs text, there has been prior work on learning code-switched language models from bilingual data @xcite and a more recent direction that explores the possibility of generating synthetic cs sentences. @xcite presents a technique to generate synthetic cs text that grammatically adheres to a linguistic theory of code-switching known as the equivalence constraint (ec) theory @xcite . @xcite proposed a bilingual attention language model for cs text trained solely using a parallel corpus. another recent line of work has explored neural generative models for cs text. @xcite use a sequence generative adversarial network (seqgan @xcite ) trained on real cs text to generate sentences that are used to aid language model training. another gan-based method proposed by @xcite aims to predict the probability of switching at each token. winata et al. (2018) and winata et al. (2019) use a sequence-to-sequence model enabled with a copy mechanism (pointer network @xcite ) to generate cs data by leveraging parallel monolingual translations from a limited source of cs data. @xcite proposed a hierarchical variational autoencoder-based model tailored for code-switching that takes into account both syntactic information and language switching signals via the use of language tags. (we present a comparison of tcs with both @xcite and @xcite in section 5.2.1.) in a departure from using generative models for cs text, we view this problem as one of sequence transduction where we train a model to convert a monolingual sentence into its cs counterpart. @xcite ; @xcite use gan-based models to modify monolingual sentences into cs sentences, while we treat this problem of cs generation as a translation task and draw inspiration from the growing body of recent work on neural unsupervised machine translation models @xcite to build an effective model of cs text. the idea of using translation models for codeswitching has been explored in early work @xcite @xcite . concurrent with our work, there have been efforts towards building translation models from english to cs text @xcite and cs text to english @xcite . while these works focus on translating from the embedded language (english) to the cs text or vice-versa, our approach starts with sentences in the matrix language (hindi) which is the more dominant language in the cs text. also, ours is the first work, to our knowledge, to repurpose an unsupervised neural machine translation model to translate monolingual sentences into cs text. powerful pretrained models like mbart @xcite have been used for codemixed translation tasks in concurrent work @xcite . we will further explore the use of synthetic text with such models as part of future work. ## our approach figure 1 shows the overall architecture of our model. this is largely motivated by prior work on unsupervised neural machine translation @xcite . the model comprises of three layers of stacked transformer @xcite encoder and decoder layers, two of which are shared and the remaining layer is private to each language. monolingual hindi (i.e. the source language) has its own private encoder and decoder layers (denoted by enc p 0 and dec p 0 , respectively) while english and hindi-english cs text jointly make use of the remaining private encoder and decoder layers (denoted by enc p 1 and dec p 1 , respectively). in our model, the target language is either english or cs text. ideally, we would like enc p 1 and dec p 1 to be trained only using cs text. however, due to the paucity of cs text, we also use text in the embedded language (i.e. english) to train these layers. next, we outline the three main training steps of tcs. d e c x 3 d s c w z 5 l m k 5 m x x j 5 k b y q 3 t m 9 o r n n f b e l n y i 0 u e + 4 4 p f s / z d v m z 2 l p m v t u s e k 5 n b a f c v / n f 7 5 u 5 h v l t t n j g 0 4 c d w r k z b r v r / j r 7 k y 8 c s b e l p n w l d e j s z m y h y j l y a r t r e p m + a t 7 h p a b i q t a d 9 j 8 y r n d m 0 y x 9 i j l x o g 0 z 3 9 x p c z q e h d 4 j n p y q r 7 x h u q 0 r z v g 7 7 s t i j b o e m z 8 + u e 9 r f k m 6 p b i t c s u c j q d a x h x w v r k e z 8 p x t h c t w c w 4 i 6 p p a k e d 8 v u u f m w d l y 6 u b y t y 5 f s k 1 1 y q f x y q i 7 i h a m s o u h w j n v r v a 2 a v w 9 x 7 h u 7 / p 1 q w 6 o a l f i n 7 k c v 7 t l t c @xmath2 ## description of datasets 4.1 a new hindi-english cs dataset we introduce a new hindi-english cs dataset, that we will refer to as all-cs. it is partitioned into two subsets, movie-cs and treebank-cs, based on their respective sources. movie-cs consists of conversational hindi-english cs text extracted from 30 contemporary bollywood scripts that were publicly available. 3 the hindi words in these sentences were all romanized with potentially multiple non-canonical forms existing for the same hindi token. we employed a professional annotation company to convert the romanized hindi words into their respective backtransliterated forms rendered in devanagari script. we also asked the annotators to provide monolingual hindi translations for all these sentences. using these monolingual hindi sentences as a starting point, we additionally crowdsourced for cs sentences via amazon's mechanical turk (mturk) @xcite . turkers were asked to convert a monolingual hindi sentence into a natural-sounding cs variant that was semantically identical. each turker had to work on five hindi sentences. we developed a web interface using which turkers could easily copy parts of the hindi sentence they wanted to retain and splice in english segments. more details about this interface, the crowdsourcing task and worker statistics are available in appendix a. all-cs comprises a second subset of cs sentences, treebank-cs, that was crowdsourcing using mturk. we extracted 5292 monolingual hindi sentences (with sentence lengths less than or equal to 15 words) from the publicly available hindi dependency treebank that contains dependency parses. 4 these annotations parse each hindi sentence into chunks, where a chunk is defined as a minimal, non recursive phrase. turkers were asked to convert at least one hindi chunk into english. this was done in an attempt to elicit longer spans of english segments within each sentence. figure 2 shows the sentence length distributions for movie-cs and treebank-cs, along with histograms accumulating english segments of different lengths in both subsets. we clearly see a larger fraction of english segments with lengths within the range [2-6] in treebank-cs compared to movie-cs. that quantifies the amount of mixing between languages (0 denotes a purely monolingual corpus and 1 denotes equal mixing from both languages) and i-index measures the fraction of switching points in the corpus. we observe treebank-cs exhibits higher m-index and i-index values compared to movie-cs indicating more code-switching overall. all-cs also contains a non-trivial number of named entities (nes) which are replaced by an ne tag in all our language modeling experiments. ## experiments and results first, we investigate various training curricula to train tcs and identify the best training strategy by evaluating bleu scores on the test set of all-cs ( §5.1). next, we compare the output from tcs with synthetic cs text generated by other methods ( §5.2). we approach this via language modeling ( §5.2.1), human evaluations ( §5.2.2) and two downstream tasks-natural language inference and sentiment analysis-involving real cs text ( §5.2.3). apart from these tasks, we also present four different objective evaluation metrics to evaluate synthetic cs text: bertscore, accuracy of a bert-based classifier and two diversity scores ( §5.3). ## conclusions in this work, we present a neural translation model for cs text that transduces monolingual hindi sentences into realistic hindi-english cs text. text generated by our model is evaluated using a number of different objective metrics, along with lm, nli and sentiment analysis tasks, and a detailed human evaluation study. the role of synthetic data in training such models merits a more detailed investigation which we leave for future work. a mturk task details figure 4 depicts the portal used to collect data using amazon's mechanical turk platform. the collection was done in two rounds, first for movie-cs and then for treebank-cs. with treebank-cs, the sentences were first divided into chunks and the turkers were provided with a sentence grouped into chunks as shown in figure 4 . they were required to switch at least one chunk in the sentence entirely to english so as to ensure a longer span of english words in the resulting cs sentence. a suggestion box converted transliterated hindi words into devanagari and also provided english suggestions to aid the workers in completing their task. with movie-cs, since there were no chunk labels associated with the sentences, they were tokenized into words. on mturk, we selected workers with hit approval rate of 90% and location restricted to countries with significant hindi speakers -australia, bahrain, canada, india, kuwait, malaysia, mauritius, myanmar, nepal, netherlands, new zealand, oman, pakistan, qatar, saudi arabia, singapore, south africa, sri lanka, thailand, united arab emirates, united kingdom, united states of america. it was clearly specified in the guidelines that the task must be attempted by native hindi speakers. each response was manually checked before approving. turkers were paid @xmath02.25-@xmath12/hr @xcite .