Towards Code-Mixed H inglish Dialogue Generation (2021)

Abstract: Code-mixed language plays a crucial role in communication in multilingual societies. Though the recent growth of web users has greatly boosted the use of such mixed languages, the current generation of dialog systems is primarily monolingual. This increase in usage of code-mixed language has prompted dialog systems in a similar language. We present our work in Code-Mixed Dialog Generation, an unexplored task in code-mixed languages, generating utterances in code-mixed language rather than a single language that is more often just English. We present a new synthetic corpus in code-mix for dialogs, CM-DailyDialog, by converting an existing English-only dialog corpus to a mixed Hindi-English corpus. We then propose a baseline approach where we show the effectiveness of using mBART like multilingual sequence-to-sequence transformers for code-mixed dialog generation. Our best performing dialog models can conduct coherent conversations in Hindi-English mixed language as evaluated by human and automatic metrics setting new benchmarks for the Code-Mixed Dialog Generation task.

## introduction due to the popularity of different social media and messaging platforms over the last decade, there has been a significant increase in internet users, mainly from multilingual societies. multilingual speakers regularly combine languages in what is commonly called code-mixing or code-switching while communicating with other multilingual speakers. this has resulted in a substantial influx of mixed language data in the form of comments, conversations, and other forms of communication. traditional natural language processing tasks like tokenization and tagging, semantic processing, machine translation, and text generation face new and interesting challenges due to this language mixing. dialog systems have been of great interest amongst the natural language processing community for widespread applications. these systems are broadly categorized into three categories: taskoriented dialog system @xcite , open-ended conversational systems @xcite , and interactive question answering system. traditional dialog systems have mostly relied on a rule or template-based approach @xcite . the success of deep neural networks with a considerable amount of training data has led towards end-to-end trained sequence-to-sequence (seq2seq) @xcite models that enhance the generality and diversity of the text generated. recent advances in attention-based mechanisms @xcite and transformers @xcite have shown significant performance improvement and shifted the communities' approach and interest in training larger models. however, all of these prior works use monolingual data and specifically english. the increasing use of code-mixed languages and ubiquitous nature of multilingual speakers call for catering to the needs of such users in a multilingual fashion with a dialog system, the need for a code-mixed dialog generational system. a recent study by @xcite shows that in a real-life setting, people prefer chatbots that engage in codemixed language. code-mixing in informal contexts like newsgroups, tweets, comments and blogs has made it difficult to define a uniform structure to the language. however, linguists have formulated various hypotheses @xcite @xcite and constraints @xcite @xcite that can define a general rule for code-mixing. with the rise of large pretrained language mod-els like @xcite @xcite ), there's been a lot of improvement in machine translation and multilingual models. prior works @xcite show the effectiveness of large multilingual pretrained language models like mbert @xcite and xlm @xcite on code-mixed data. our work attempts to utilize these large seq2seq pre-trained multilingual transformer-based models for the code-mixed dialog generation task. specifically, our contributions are as follows. we finetune the mbart model for the dialog generation in two ways to generate coherent dialog, as seen from both automatic and human evaluation metrics. ## related work code-mixing refers to the interleaving of words belonging to different languages. code-mixing is most common in multilingual cultures, and it is becoming more common as the number of people using social media and messaging services on the internet grows @xcite . this has lead to a rising research interest in recent years, and several tasks have been conducted as part of codeswitching workshops @xcite . there has been a lot of advancements in solving different code-mixed tasks like language identification @xcite , named-entity recognition @xcite , question answering @xcite , part-of-speech tagging @xcite , and information retrieval @xcite . very recently, a code-mixed version of the glue benchmark was proposed by @xcite which introduced a common benchmark for all the tasks. ## system overview this section describes the benchmark dataset for code-mixed dialog generation: cm-dailydialog, the english to hinglish translation model used to generate this dataset, and our mbart based dialog generation model. ## experimental setup in this section, we describe our experimental setup for both our mbart-en cm model and the dialog model described in section 3.1.1 & 3.2 respectively. our proposed approach is written in pytorch @xcite , and the mbart model weights and architecture used are from the hug-gingface's transformer @xcite package. we use only the mbart-cc-25 weights in all our modeling. all the mbart based models were trained using the adamw optimizer with weight decay. we used all the default hyperparameters except the number of training epochs. we finetuned therefore, there is no baseline for us to compare our model to and we report our numbers as it is. as discussed in section 3.1, we process our datasets from english to code-mixed and then into triplets. we split our processed cm-dailydialog dataset into 8:1:1 splits for training, validation, and test set and use the additional code-mixed nli dataset in conjunction with the cm-dailydialog dataset to see any performance improvement with the increased data. we evaluate both our mbartdialog and mbart-dialog + models on bleu and perplexity metric and report our scores in the we also show the performance of our mbarten cm model on different datasets for the monolingual english to hinglish translation task using metrics like sacrebleu(reported as bleu) and code-mixing index in ## results ## conclusion we introduce a new benchmark dataset for codemixed dialog generation, cm-dailydialog, a codemixed version of the dailydialog. our work pro-poses using multilingual transformers (mbart) and demonstrates how they help in code-mixed dialog generation. we also introduce a new monolingual english to code-mixed machine translation model using mbart. with our comprehensive experiments, we show the effectiveness of our approach in terms of machine translation and dialog generation and set new benchmarks in the code-mixed dialog generation task. the manual error analysis illustrates the quality of the new dataset, although it is synthetically generated. in terms of both automatic and human evaluation metrics, we show that the dialog generated from our model is of high quality. as part of the future work, we would like to improve our machine translation model to improve our cm-dailydialog data that further boosts our dialog generation. another huge scope of improvement is in the vocabulary discrepancy in different datasets, and we wish to resolve this to further boost our modeling and performance.