H in GE : A Dataset for Generation and Evaluation of Code-Mixed H inglish Text (2021)

Abstract: Text generation is a highly active area of research in the computational linguistic community. The evaluation of the generated text is a challenging task and multiple theories and metrics have been proposed over the years. Unfortunately, text generation and evaluation are relatively understudied due to the scarcity of high-quality resources in code-mixed languages where the words and phrases from multiple languages are mixed in a single utterance of text and speech. To address this challenge, we present a corpus (HinGE) for a widely popular code-mixed language Hinglish (code-mixing of Hindi and English languages). HinGE has Hinglish sentences generated by humans as well as two rule-based algorithms corresponding to the parallel Hindi-English sentences. In addition, we demonstrate the in- efficacy of widely-used evaluation metrics on the code-mixed data. The HinGE dataset will facilitate the progress of natural language generation research in code-mixed languages.

## introduction code-mixing is the mixing of two or more languages in a single utterance of speech or text. a commonly observed communication pattern for a multilingual speaker is to mix words and phrases from multiple languages. code-mixing is widespread across various language pairs, such as spanish-english, hindi-english, and bengali-english. recently, we observe a boom in the availability of code-mixed data with the inflation of the social media platforms such as twitter and facebook. in past, we witness magnitude of work to address standard code-mixing natural language understanding (nlu) tasks such as language identification @xcite @xcite , pos tagging @xcite , named entity recognition @xcite , and dependency pars-ing @xcite along with sentence classification tasks like sentiment analysis @xcite , stance detection @xcite , and sarcasm detection @xcite . unlike code-mixed nlu, natural language generation (nlg) of code-mixed text is highly understudied. resource scarcity adds to the challenge of building efficient solutions for code-mixed nlg tasks. evaluation of the code-mixed nlg tasks also lacks standalone resources, theories, and metrics. recently, we observe a growing interest in the code-mixed text generation task. to generate the code-mixed data various techniques have been employed such as matrix frame language theory @xcite @xcite , equivalent constraint theory @xcite , pointer-generator network @xcite @xcite , generative adversarial networks (gans) @xcite , etc. the majority of the available datasets @xcite @xcite employed in code-mixed nlg contains noisy codemixed text collected from social media platforms such as twitter. these datasets also lack the sanity check for the quality of sentences, making the systems developed on these datasets vulnerable to real-world applicability. to address the challenge of scarcity of high-quality resources for the codemixed nlg tasks, we propose hinge dataset foot_0 that will facilitate the community to build robust systems. the dataset contains sentences generated by humans as well as two rule-based algorithms. in in addition to the code-mixed nlg, the evaluation of the generated code-mixed text is a challenging task. the widely popular metrics for monolin- dataset characteristics banerjee et al. (2018) srivastava and singh (2020) gupta et al. (2020) hinge parallel source sentences only es only es human-generated code-mixed sentences multiple human-generated code-mixed sentences for a parallel sentence machine-generated code-mixed sentences multiple machine-generated code-mixed sentences for a parallel sentence human ratings for the quality of generated code-mixed sentences dataset size 6733 ueu, 6549 uhu 13,738 1,561,840 ps, 252,330 mghs 1,976 ps, 4,803 hghs, 3,952 mghs the scarcity of high-quality code-mixed datasets limits current research in various nlu tasks such as text generation and summarization. to address this challenge, we create a human-generated corpus of hinglish sentences corresponding to parallel monolingual english and hindi sentences. we use the iit bombay english-hindi parallel corpus (hereafter 'iit-b corpus') @xcite . the iit-b corpus has 1,561,840 parallel sentence pairs in english and hindi. the english sentences are written in the roman script, and the hindi sentences are written in the devanagari script. we randomly select 5,000 sentence pairs, in which the number of tokens in both the sentences is more than five to create a human-generated parallel corpus. to create the gold-standard dataset, we employ five human annotators. each annotator has expertlevel proficiency in writing, speaking, and understanding english and hindi languages. the objective of the annotation is to generate at least two unique hinglish code-mixed sentences corresponding to the parallel english and hindi sentence pairs. the annotators can also generate more than two code-mixed sentences for each sentence pair. we shuffle, pre-process, and share the sentence pairs with the annotators to generate the corresponding hinglish sentences. a single annotator annotates each sentence pair. we assign 1,000 unique sentence pairs to each annotator with the following annotation guidelines: post annotation, we remove the sentence pairs marked as "#" or are missing an annotation. note that due to the complexity of generating codemixed sentences, such as the inability to iden-tify two unique hinglish sentences, complex sentence structuring, and usage of difficult words in monolingual sentences, annotators do not provide two unique sentences hinglish sentences for each monolingual sentence pair. we obtain 1,978 sentence pairs with two or more unique hinglish sentences. on average, 2.5 code-mixed sentences are created for each hindi-english sentence pair. figure 1 shows an example of two code-mixed sentences generated by the annotator for a given sentence pair. qualitative evaluation of the human-generated hinglish text: to qualitatively evaluate the generated sentences, we adapt the evaluation strategy described in @xcite . we randomly sample 100 hinglish sentences generated by humans along with the source parallel monolingual english and hindi sentences. we employ two human annotators foot_1 for the qualitative evaluation. we ask the annotators to rate each hinglish sentence on two metrics: the average dcm scores by the two human annotators are 8.72 and 8.65. the average ra scores are 8.65 and 8.37. the high average scores demonstrate good quality code-mixed sentence generation. in addition to the human-generated code-mixed text, we also generate the hinglish sentence synthetically by following the embedded-matrix theory @xcite . we propose two rule-based hinglish text generation systems leveraging the parallel monolingual english and hindi sentences. in both systems, we use hindi as the matrix language and english as the embedded language. the matrix language imparts structure to the code-mixed text with tokens embedded from embedded language. we also use several linguistic resources in both generation systems. these resources include: we curate 77,805 pairs of english words and the corresponding hindi meanings from two sources 3,4 to construct an english-hindi dictionary. we further extend the english-hindi dictionary by incorporating parallel sentences in the iit-b corpus. we leverage vecmap's shared representation to identify the closest word in english and the corresponding hindi sentence. in addition, we also leverage giza++ to align the parallel sentences resulting in aligned english-hindi tokens. both of these steps extend the initial dictionary from 77,809 to 1,52,821 words and meaning pairs. we use this extended dictionary in the following two code-mixed text generation systems: it should be noted that the code-mixed sentences generated using the wac and pac procedures have added noise (spelling variations, grammatical inconsistencies, etc.) and are not an exact representation of the high-quality code-mixed text generated by humans. this noisy machine-generation of the code-mixed text is intentional such that the efficacy of the evaluation metrics could be studied (see section 4) for the wide spectrum of sentences and should not be limited to good quality code-mixed sentences. ## a study on evaluation of code-mixed text generation in this section, we evaluate machine-generated code-mixed text. this study demonstrates severe limitations of five widely popular nlp metrics in evaluating code-mixed text generation performance. we leverage the following met (2019b)), (iv) word error rate (wer, levenshtein (1966)), and (v) translator error rate (ter, snover et al. (2006)). higher bleu, nist, or bs values and lower wer or ter values represent better generation performance. we conduct two experiments to evaluate the machine-generated hinglish text: each generated sentence is rated by two human evaluators. all the evaluators have expertlevel proficiency in the english and hindi languages. for this task, we divide the human ratings into three buckets: ## limitations and opportunities in this section, we present a discussion on the various inherent limitations associated with the proposed hinge dataset. we also discuss the various opportunities for the computational linguistic community to build efficient systems and metrics for code-mixed languages. some of the major limitations with the dataset are: even with the presence of the above limitations, the hinge dataset could be effectively used for various purposes such as: ## conclusion in this paper, we present a high-quality dataset (hinge) for the text-generation and evaluation task in the code-mixed hinglish language. the codemixed sentences in the hinge dataset are generated by humans and rule-based algorithms. we demonstrate the poor evaluation capabilities of five widely popular metrics on the code-mixed data. along with the human-generated sentences, the machinegenerated sentences (as described in section 3) and the human ratings of these code-mixed sentences could facilitate building the highly scalable and robust evaluation metrics and strategies for the code-mixed text. the multiple human-generated sentences corresponding to a pair of parallel monolingual sentences will pave the way in designing natural language generation systems robust to adversaries and linguistic diversities such as spelling variation and matrix language.