Enhancing Code-mixed Text Generation Using Synthetic Data Filtering in Neural Machine Translation (2023)

Abstract: Code-Mixing, the act of mixing two or more languages, is a common communicative phenomenon in multi-lingual societies. The lack of quality in code-mixed data is a bottleneck for NLP systems. On the other hand, Monolingual systems perform well due to ample high-quality data. To bridge the gap, creating coherent translations of monolingual sentences to their code-mixed counterparts can improve accuracy in code-mixed settings for NLP downstream tasks. In this paper, we propose a neural machine translation approach to generate high-quality code-mixed sentences by leveraging human judgements. We train filters based on human judgements to identify natural code-mixed sentences from a larger synthetically generated code-mixed corpus, resulting in a three-way silver parallel corpus between monolingual English, monolingual Indian language and code-mixed English with an Indian language. Using these corpora, we fine-tune multi-lingual encoder-decoder models viz, mT5 and mBART, for the translation task. Our results indicate that our approach of using filtered data for training outperforms the current systems for code-mixed generation in Hindi-English. Apart from Hindi-English, the approach performs well when applied to Telugu, a low-resource language, to generate Telugu-English code-mixed sentences.

## introduction code-mixing (cm) is a phenomenon of mixing two or more languages in an utterance of a speech or text @xcite . this form of communication is prevalent in multi-lingual communities owing to socio and psycho-linguistic reasons. with the advent of social media, code-mixing has become a common phenomenon of communication on social platforms like facebook, twitter, reddit, etc. the extensive use of code-mixing has led to interesting computational multi-lingual nlp research directions. linguistic research on code-mixing has proposed multiple theories for generating code-mixed sentences. the equivalence constraint (ec) theory, introduced by @xcite , posits that code-switching occurs when there is functional equivalence between the source and target languages, indicating similarity in meaning, pragmatics, or discourse function. the matrix language frame (mlf) theory, proposed by @xcite , suggests that bilingual individuals incorporate words or phrases from a non-dominant language into a dominant language or "matrix language" structure. recently, pre-trained models @xcite have become the state-ofart models for multi-lingual language analysis and generation systems. availability of large monolingual text corpora from sources like news, wikipedia, books, has enabled researchers to train large language models at scale. however, building natural language processing (nlp) systems for code-mixed text or speech has become challenging due to its resource poor nature. while codemixed text is prevalent in various online platforms, such text often co-exists with monolingual data. thus, identifying code-mixed sentences and building large-scale corpus is challenging. recently researchers have used multiple approaches to translate between monolingual and their code-mixed counterparts. gcm @xcite proposed an open-source toolkit which leverages ec and mlf theories of code-mixing to generate multiple synthetic code-mixed sentences for a given set of parallel monolingual sentences. however, a limitation of gcm is that the generated code-mixed sentence need not always be a natural sentence. @xcite proposed neural machine translation methods to generate a code-mixed sentence given a monolingual input, where the synthetic data is created using clausal substitutions based on mlf theory. all the previously proposed approaches to create synthetic code-mixed data to train machine translation systems have not considered the quality of the synthetic code-mixed data. in this paper, we propose a novel approach for generating natural code-mixed sentences, by finetuning multi-lingual encoder-decoder models. the main focus of the paper is to train these models with good quality code-mixed sentences and their monolingual counterparts. in order to create this silver parallel corpus, we use code-mixed quality filters that are created from the human judgements on minimal gold-standard text. the main contributions of this paper are summarized as below : while synthetic data has been used to train machine translation models to generate code-mixed, the quality of those synthetically generated sentences has not been analyzed. we hypothesize that controlling the quality of synthetic code-mix sentences before using them to train translation models can lead to more natural code-mix sentences, and can even be compute efficient. to the best of our knowledge, this is the first work to use human judgements for quality of code-mixed text to create silver parallel data, and use the data to train neural machine translation models for code-mixed text generation. ## methodology in our methodology, as illustrated in figure 1 , we propose models for generating code-mixed text which trained using a silver parallel corpus created by filtering a large synthetic code-mixed corpus. for training the quality filters, we leverage human annotations capturing the quality of code-mixed sentences (sec.3.1.1) and distributions in humangenerated code-mixed sentences (sec.3.1.2). using the trained filters we create silver parallel corpus (sec.3.2). we use the filtered sentences to train machine translation models that will enable generation of code-mixed text (sec.3.3). in this study we experiment with two language pairs -hindi-english and telugu-english. ## experimental setup in this section, we present our experiments to examine the effectiveness of each filter and its contribution towards generating high-quality codemixed sentences. the experimental setup is described in detail, followed by a comprehensive analysis of the results obtained. in our experimental setup, we performed finetuning of pre-trained language models, namely mt5 and mbart, for code-mixed text generation in hindi-english and telugu-english. the input to these models consists of the concatenation of two corresponding monolingual sentences, and the output is a code-mixed sentence. for each language pair, we fine-tuned each model on four different training datasets created using random sampler, polynomial, bert, and probabilistic filters, respectively. using a random sampler as a baseline, our objective was to evaluate the model's performance by using the same hyperparameters and an equal number of samples for both random sampler and other filters. we fine-tuned mt5 and mbart for hindi-english and telugu-english code-mixed text generation using appropriate hyperparameters. for mt5, we trained with a batch size of 64 and a learning rate of 2e-3, while for mbart, we used a batch size of 32 and a learning rate of 3e-6. we used the default ada-w optimizer (kingma and ba, 2014) for training both models, and selected the hyperparameters to minimize the validation dataset loss. ## results and analysis the datasets used for evaluation include codemixed sentences that are sourced from various social media platforms (calcs, mrinaldhar, senti-dataset) as well as sentences that are generated by humans(all-cs), and those that are transcribed from speech (dialoguedataset). our models were able to achieve quality results on a variety of codemixed datasets, despite the differences in sampling and characteristics between the training (synthetically generated) and testing sets. this suggests that our models are robust and can be applied to a wide range of datasets with varying characteristics and highlights the effectiveness of our models. in a similar experiment proposed by @xcite , models when trained on synthetic data and tested on the all-cs test dataset achieved a bleu score of 17.73. however, our mt5 model trained with data after applying probabilistic filtering outperformed it, achieving a much higher score of 30.02. this significant improvement highlights the importance of using probabilistic models for code-mixed language translation, as it allows for better modeling of the underlying language patterns and improves the overall performance of the figure 3: example illustrating telugu-english codemixed text generation using multiple filters the mbart model with bert filtering achieved the highest rouge-l scores, while the mt5 model with probabilistic filtering achieved the highest bleu scores, for hindi test datasets. for telugu test datasets, models with probabilistic filtering achived higher rouge-l and bleu scores. the bleu and rouge-l scores demonstrate how filtering plays a significant role in the model's performance. all testing datasets are human-generated code-mixed sentences, highlighting how the filters aid in generating codemixed sentences that closely resemble humangenerated ones. the probabilistic filter uses the feature distribution of human-generated code-mixed sentences to improve the model's performance. this filter was initially created using the hindi-english dataset but was also applied to the telugu-english dataset, demonstrating its language-independent nature. the good results obtained from telugu-english code-mixed test generation highlight the power and effectiveness of this filtering mecha-nism. ## conclusion in this work, we present a novel approach to create high-quality silver parallel data for code-mixed data. the primary focus of our approach is to select natural code-mix sentences from a larger synthetically generated code-mixed corpus. leveraging human knowledge, we train filters to select high-quality code-mixed sentences. using the filtered sentences, we fine-tune mllms for machine translation task. our filtering-based neural machine translation approach for code-mixed sentence generation shows promising results across various datasets, and different language pairs -hindi-english and telugu-english. the finetuning of pre-trained models such as mt5 and mbart has enabled us to generate high-quality code-mixed sentences with minimal gold-standard corpus. we also experimented with the probabilistic filter method, which does not need human annotations for quality but relies on human generated code-mixed sentences. the probabilistic filter is effective and language-independent, as probabilistic filter either matches or outperforms other filters proposed in the study. it can easily be extended to other languages, unlike other mechanisms that require human effort. our study has implications for the generation of natural code-mixed sentences at scale -which can improve downstream task performances.