MUTANT : A Multi-sentential Code-mixed H inglish Dataset (2023)

Abstract: The multi-sentential long sequence textual data unfolds several interesting research directions pertaining to natural language processing and generation. Though we observe several high-quality long-sequence datasets for English and other monolingual languages, there is no significant effort in building such resources for code-mixed languages such as Hinglish (code-mixing of Hindi-English). In this paper, we propose a novel task of identifying multi-sentential code-mixed text (MCT) from multilingual articles. As a use case, we leverage multilingual articles from two different data sources and build a first-of-its-kind multi-sentential code-mixed Hinglish dataset i.e., MUTANT. We propose a token-level language-aware pipeline and extend the existing metrics measuring the degree of code-mixing to a multi-sentential framework and automatically identify MCT in the multilingual articles. The MUTANT dataset comprises 67k articles with 85k identified Hinglish MCTs. To facilitate future research directions, we will make the dataset and the code publicly available upon publication.

## introduction over the years, we have seen enormous downstream applications of multi-sentential datasets in the areas such as question-answering @xcite , summarization @xcite , machine translation @xcite , etc. the existing state-of-the-art methods prove challenging to scale effectively and efficiently on multi-sentential long sequence text @xcite , which unplugs several exciting research avenues. unfortunately, to a large extent, the majority of the research on multi-sentential data is dominated by a few popular monolingual languages such as english, chinese, and spanish. due to this, code-mixed languages (among other low-resource and under-explored languages) suffer from non-existent works in the aforementioned areas of interest. we posit that due to several inherent challenges, the nlp community hold back on building multisentential datasets for the low-resource and codemixed languages. one of the most significant bottlenecks in building such resources is the unavailability of mct on traditional and widely popular data sources such as social media platforms where the short-length and noisy code-mixed text is available in abundance. it presents several challenges such as the difficulty in curating a large-scale multisentential dataset at ease. another major challenge is the lack of metrics to measure the degree of codemixing in the multi-sentential framework. the existing metrics such as code-mixing index @xcite and multilingual-index @xcite already suffers from major limitations @xcite in the short-length text format. in such a scenario, it gets mystifying to build a retrieval pipeline to identify mct and we need to depend heavily on the expertise of human annotators which is a time and cost-demanding exercise. in this work, we address both of these challenges. as a representative use case, we base our work on hinglish, a popular code-mixed language in the indian subcontinent. but the insights from our exploration could be extended to other code-mixed language pairs. to address the first challenge, we identify two non-traditional multilingual data sources 1 i.e., political speeches and press releases along with hindi daily news articles (discussed in detail in section 3). figure 1 shows example hinglish mcts from two multilingual data sources. to address the second challenge, we propose a token-level languageaware pipeline and extend a widely popular metric (i.e., code-mixing index) measuring the degree of code-mixing in a multi-sentential framework. we demonstrate the effectiveness of the proposed pipeline with a minimal task-specific annotation which significantly reduces the overall human effort (discussed in detail in section 4). eventually, we build a novel multi-sentential dataset for the hinglish language with 85k mcts identified from 67k articles. in it should be noted that an article a can have multiple non-overlapping unique mcts i.e. @xmath5, m 2 , ..., m q } where @xmath6. ## multilingual and multi-sentential data sources over the years, we observe several interesting and diverse code-mixed data sources such as twitter, facebook, movie transcripts, etc. social media sites have acted as the cornerstone of the codemixed data collection pipelines due to the ease of availability of large-scale data. nonetheless, they present several challenges such as noisy data, short text, abusive, and multimodal data. given the requirements of mutant (i.e. multi-sentential and high-quality data), we refrain from using social media sites in this work. here, we focus on two major data sources: ## experimental setup problem definition: given a multilingual article a comprising of q multi-sentential text spans (mst) i.e. @xmath7, m 2 , ..., m q }, we predict a binary outcome l cm for each mst figure 2 shows the architecture of the mct identification pipeline. next, we discuss the various components of this pipeline in detail. ## mutant: a multi-sentential code-mixed hinglish dataset we evaluate the performance of mct identification pipeline and the five dual mec threshold generalization techniques using the three subsets of the sand dataset: d speech , d news , and d speech + d news . we report the following metric scores on each of the seven data sources: 1. accuracy: we compute accuracy as the ratio of the total correct prediction of mct and non-mct to the total number of mst. we multiply this ratio by 100 and report the accuracy percentage. a high accuracy % is preferred. 2. false mct rate (fmr): we define fmr as the ratio of incorrectly identified mct to the total number of actual monolingual mst. we report the fmr% and a low fmr% is preferred. 3. diversity@10 (d@10): we define d@10 as the percentage of articles in data source d i having more than 10% correctly identified mct. a high d@10 score is preferred. we report the results in tables 6, 7, 8. the meanbased threshold generalization techniques (la, ga, and alg) consistently show poor performance on all the metrics. given the nature of the problem, we prefer a low rate of misidentification of mono- ## analysis and discussion in this section, we qualitatively evaluate the mu-tant dataset by employing two human evaluators, different from the one used for the sand to avoid any biases in the evaluation. both evalua ## conclusion in this paper, we present a novel task of identifying mct from multilingual documents. we propose an mct identification pipeline by extending cmi to the multi-sentential framework and leveraging the pipeline we build a dataset for the hinglish language. we highlight several challenges in building such resources and our insights will be useful to future works in code-mixed and low-resource languages.