GCM : A Toolkit for Generating Synthetic Code-mixed Text (2021)

Abstract: Code-mixing is common in multilingual communities around the world, and processing it is challenging due to the lack of labeled and unlabeled data. We describe a tool that can automatically generate code-mixed data given parallel data in two languages. We implement two linguistic theories of code-mixing, the Equivalence Constraint theory and the Matrix Language theory to generate all possible code-mixed sentences in the language-pair, followed by sampling of the generated data to generate natural code-mixed sentences. The toolkit provides three modes: a batch mode, an interactive library mode and a web-interface to address the needs of researchers, linguists and language experts. The toolkit can be used to generate unlabeled text data for pre-trained models, as well as visualize linguistic theories of code-mixing. We plan to release the toolkit as open source and extend it by adding more implementations of linguistic theories, visualization techniques and better sampling techniques. We expect that the release of this toolkit will help facilitate more research in code-mixing in diverse language pairs.

## introduction code-mixing, which is the alternation between two or more languages in a single conversation or utterance is prevalent in multilingual communities all over the world. processing code-mixed language is challenging due to the lack of labeled as well as unlabeled data available for training nlp models. since code-mixing is a spoken language phenomenon, it is more likely to occur in informal written text, such as social media and chat data. such data may not as easily available as monolingual data for building models, and may also exhibit other issues such as cross-transcription and nonstandard spellings. to alleviate this problem and train language models that can use unlabeled data for pre-training, we see the generation of synthetic code-mixed data as a promising direction. various linguistic theories have been proposed that can determine how languages are mixed together, and in prior work we presented the first computational implementation @xcite of the matrix-language @xcite and equivalence constraint theories @xcite . we also showed that generating synthetic data using our computational implementations improved word embeddings leading to better downstream performance on sentiment analysis and pos tagging @xcite , as well as rnn language models @xcite . the multilingual bert @xcite model finetuned with synthetic code-mixed data outperformed all prior techniques on the gluecos benchmark @xcite for code-switching, which spans 11 nlp tasks in two language pairs. the approach of generating synthetic code-mixed data has gained traction following our work, with other approaches including using generative adversarial networks @xcite , an encoder-decoder framework with transfer learning @xcite , using parallel data with a small amount of real code-mixed data to learn code-mixing patterns @xcite and a novel two-level variational autoencoder approach @xcite . in this work, we present a tool gcm that can automatically generate synthetic code-mixed data given parallel data or a machine translation system between the languages that are being mixed. our tool is intended for use by nlp practitioners who would like to generate training data to train models that can handle code-mixing, as well as linguists and language experts who would like to visualize how code-mixing occurs between languages given different linguistic theories. the toolkit provides three modes -a batch mode, that can run the data generation pipeline on servers, an interactive mode, that can be used for quick prototyping as well as a web interface that can be used to visualize codemixed sentence generation. the gcm tool will be released as open source and we plan to improve it by adding more implementations of linguistic theories, visualization techniques and better algorithms for sampling. we expect that the release of this toolkit will spur research in code-mixing in diverse language pairs and enable many nlp applications that would not be possible to build due to the lack of code-mixed data. ## method in this section we discuss the linguistic theories that we implement in the tool and the pipeline we use for generating code-mixed (hereafter referred to as cm) sentences. ## system overview we provide three modes in the gcm tool: a batch mode, an interactive library mode and a webinterface to address the needs of nlp practitioners, researchers, linguists and language experts: ## conclusion and future work generating synthetic cm data has become a promising direction in research on code-mixing, due to the lack of available data and has proved to be successful in improving various cm nlp tasks. in this paper, we describe a tool for generating synthetic cm data given parallel data in two languages, or a translator between two languages. we implement two theories of code-mixing, the equivalence constraint (ec) theory and the matrix-language (ml) theory to generate cm data, followed by a sampling stage to sample sentences that are close to real code-mixing in naturalness. the gcm tool operates in three modes -a batch mode, which is meant for large scale generation of data, a library mode, which is meant to be customizable and extensible and a web ui, which is meant as a visu-alization tool for linguists and language experts. we plan to release the gcm tool as open source code and add more implementations of linguistic theories, generation techniques and sampling techniques. we believe that this tool will help address some of the problems of data scarcity in cm languages, as well as help evaluate linguistic theories for different language pairs and we expect that the release of this toolkit will spur research in diverse code-mixed language pairs.