C o C oa: An Encoder-Decoder Model for Controllable Code-switched Generation (2022)

Abstract: Code-switching has seen growing interest in recent years as an important multilingual NLP phenomenon. Generating code-switched text for data augmentation has been sufficiently well-explored. However, there is no prior work on generating code-switched text with fine-grained control on the degree of code-switching and the lexical choices used to convey formality. We present CoCoa, an encoder-decoder translation model that converts monolingual Hindi text to Hindi-English code-switched text with both encoder-side and decoder-side interventions to achieve fine-grained controllable generation. CoCoa can be invoked at test-time to synthesize code-switched text that is simultaneously faithful to syntactic and lexical attributes relevant to code-switching. CoCoa outputs were subjected to rigorous subjective and objective evaluations. Human evaluations establish that our outputs are of superior quality while being faithful to desired attributes. We show significantly improved BLEU scores when compared with human-generated code-switched references. Compared to competitive baselines, we show 10% reduction in perplexity on a language modeling task and also demonstrate clear improvements on a downstream code-switched sentiment analysis task.

## introduction bilingual speakers form a significant portion (current estimates of 43% 1 ) of the world's population. to cater to the human-computer interaction needs of this user segment, natural language generation (nlg) and natural language understanding (nlu) tasks for code-switching (cs) are receiving increasing amount of attention from the research community @xcite . code-switching in a sentence typically involves switching between a matrix language l1 and an embedded language l2. a key challenge in cs that has not been previously addressed by generation models is to explicitly control for syntactic and lexical diversity @xcite . such controllable nlg models would help build robust computational models for cs text. we draw attention to three specific dimensions of diversity that are commonly observed in cs text. in this work, we aim to generate text that specifically spans these three cs dimensions. 1. language mix ratio: the first syntactic dimension of diversity refers to the varying number of l1 and l2 words in a cs sentence that depends on a number of factors like language pair, socioeconomic context, etc. for instance, @xcite find that among immigrant bengalis in the uk, first-generation immigrants tend to use bengali and english as l1 and l2 respectively, while the order is reversed for younger generation immigrants. code-mixing index (cmi) @xcite is commonly used to quantify the ratio of l1 vs. l2. ## ablations and model variants comparison with tagging. figure 3b shows how tagging compares to the scaled vector approach for encoder-side control. on increasing the number of bins (k) from 3 to 8 for tagging, cmi corr increases from 0.77 to 0.89, making it more comparable to the cmi corr of 0.92 using the scaled approach. as we increase the number of bins, k-class cmi acc drops for both the tagging and scaled vector approaches, with the latter showing a small but consistent improvement across bins. ## the diverse all-cs dataset we release a new dataset diverse-allcs which is an extension of allcs with more natural diversity in cmi and spi. 9 we select 1928 sentences from allcs-test with a sentence length between 5 and 15 tokens. to mimic the natural distribution of cmi/spi scores, we refer to allcs-train and divide cmi/spi values into two bins, resulting in four cmi, spi combinations. we use cocoa m,cmi,spi to generate model outputs and sample proportional to the number of allcs sentences assigned to each cmi/spi combination. after removing duplicates, we have 5380 unique cs sentences that were subjected to human evaluations. ## discussion and conclusion we list a few observations from our experiments. 1. we find that decoder-side control is more brittle and requires interventions like multi-task training to improve on naturalness and consistency of the generated outputs. encoder-side control, in comparison, is more robust. the trade-off is that the latter requires parallel data that is more tedious to acquire, while decoder-side control does not. in conclusion, we establish cocoa as a state-ofthe-art controllable code-switched generation tool with test-time controls that outperforms existing unconstrained baselines.