Detecting Entailment in Code-Mixed H indi- E nglish Conversations (2020)

Abstract: The presence of large-scale corpora for Natural Language Inference (NLI) has spurred deep learning research in this area, though much of this research has focused solely on monolingual data. Code-mixing is the intertwined usage of multiple languages, and is commonly seen in informal conversations among polyglots. Given the rising importance of dialogue agents, it is imperative that they understand code-mixing, but the scarcity of code-mixed Natural Language Understanding (NLU) datasets has precluded research in this area. The dataset by Khanuja et. al. for detecting conversational entailment in code-mixed Hindi-English text is the first of its kind. We investigate the effectiveness of language modeling, data augmentation, translation, and architectural approaches to address the code-mixed, conversational, and low-resource aspects of this dataset. We obtain an 8.09% increase in test set accuracy over the current state of the art.

## introduction natural language inference (nli) is a widely researched nlp task which involves determining if a premise entails or contradicts a hypothesis. the performance of machine learning models on this task has important implications for other natural language understanding tasks such as question answering, semantic search and text summarization. while large corpora such as snli @xcite and multinli @xcite are available for monolingual and cross-lingual nli, @xcite introduce the first nli dataset with hindi-english (hinglish) text. we refer to this dataset as cs-nli. code-mixing is a phenomenon prevalent in multilingual communities @xcite . it poses a number of interesting challenges for nlp applications, such as the mixing of units from multiple grammar systems, morphological * equal contribution differences between monolingual and code-mixed text due to the intermixing of affixes, and nonstandard transliteration between the writing systems involved. in cs-nli, hindi is present in a non-standard romanized form. multilingual speakers most often code-mix in informal settings such as social media, in-person, and telephonic conversations, due to which there is a dearth of clean, large-scale code-mixed corpora such as wikipedia articles and books that can be used for pre-training, making this a low-resource task. @xcite leverage bollywood movie scripts containing hinglish text to create cs-nli, with conversations as premises. the creation of hypotheses based on dialogue-like premises transforms the task from one of textual entailment to one of conversational entailment. the inclusion of scripts from multiple movies makes this data inherently noisy due to non-standard romanization of hindi, the variation in dialects across movies and differing grammar styles among hinglish speakers. in this work, we explore and analyze a variety of techniques to leverage existing pre-trained models such as bert @xcite for processing code-mixed and conversational text. we present a comparison of linguistic, data augmentation and architectural approaches to conversational entailment in code-mixed text. we show multiple techniques that interestingly give similar results, while also beating the current state of the art foot_0 . the code for the approaches described in this paper will be made available on github foot_1 . ## related work nli for monolingual and cross-lingual text is a well-researched task that has been addressed using a variety of techniques including neural networks, symbolic logic and knowledge bases @xcite . the use of transformer models such as bert and roberta @xcite , pre-trained on large monolingual corpora, has advanced the state of the art on the snli and multinli datasets. while unsupervised pre-training of deep learning models has been shown to improve performance on a variety of nlp tasks, the limited amount of data available precludes large-scale pre-training on code-mixed text. multilingual bert (mbert) @xcite is pre-trained on monolingual wikipedia corpora from 104 languages, including hindi in its original devanagari script. xlm-roberta (xlm-r) @xcite is trained on the commoncrawl corpus, which includes romanized hindi text, making this model the closest one to being pre-trained on hinglish. ## methodology given the success of pre-trained models on other nli tasks, we tackle this task by fine-tuning bert, mbert and xlm-r for sentence-pair classification. due to the scarcity of examples in cs-nli, we focus our efforts on the modification and augmentation of the data used to fine-tune these models. in this section, we describe techniques to address the code-mixed, low-resource, and conversational aspects of the task. ## experimental setup in the majority of our approaches, we fine-tune bert, mbert, mod-mbert (110m parameters), and xlm-r (550m parameters) for 1 to 6 epochs on an nvidia geforce gtx 1070 gpu. we experiment with batch sizes of 8,16, and 32, and learning rates between 1e-5 and 5e-5, and report results using a batch size of 8 and learning rate of 1e-5. ## results and analysis on fine-tuning the bert models on cs-nli, we observe a large variation in the results based on the subset of data used for evaluating the model, as demonstrated in table-2. to address this variation, we perform eight-fold cross validation with early stopping, and report the mean and standard deviation of the accuracies across eight splits. these results are shown in in this section, we provide qualitative and quantitative analysis of our approaches. the qualitative analysis is performed on the cross-validation splits. ## conclusion our results show that there is a long way to go in nlp for code-mixed language tasks. even using standard techniques such as multilingual language modeling and data augmentation, our results are still behind an equivalent task in a high resource environment. although this dataset contains higher level challenges such as sarcasm detection that are not yet solved even in high-resource languages, even phenomena such as negation, role swapping and paraphrasing become challenging due to code-mixing. code-mixed language pairs can be thought of as a separate language @xcite , and perhaps large-scale pre-training on code-mixed data would be able to push the boundaries of codemixed interpretation, as has been the case with high-resource languages.