PACMAN : PA rallel C ode M ixed d A ta generatio N for POS tagging (2022)

Abstract: Code-mixing or Code-switching is the mixing of languages in the same context, predominantly observed in multilingual societies. The existing code-mixed datasets are small and primarily contain social media text that does not adhere to standard spelling and grammar. Computational models built on such data fail to generalise on unseen code-mixed data. To address the unavailability of quality code-mixed annotated datasets, we explore the combined task of generating annotated code mixed data, and building computational models from this generated data, specifically for code-mixed Part-Of-Speech (POS) tagging. We introduce PACMAN(PArallel CodeMixed dAta generatioN) - a synthetically generated code-mixed POS tagged dataset, with above 50K samples, which is the largest annotated code-mixed dataset. We build POS taggers using classical machine learning and deep learning based techniques on the generated data to report an F1-score of 98% (8% above current State-of-the-art (SOTA)). To determine the efficacy of our data, we compare it against the existing benchmark in code-mixed POS tagging. PACMAN outperforms the benchmark, ratifying that our dataset and, subsequently, our POS tagging models are generalised and capable of handling even natural code-mixed and monolingual data.

## introduction code-mixing or code-switching is primarily observed and archetypal in multilingual societies across the globe @xcite @xcite . although linguists differentiate between code-mixing @xcite and code-switching @xcite , we use cm to mean both. cm has recently garnered significant interest for researchers because of its gradual emergence as the prima lingua for social media posts, blogs, chats, and messages. the first computational work on cm text was explored by @xcite for english-spanish pos tagging. they used individual pos taggers for english and spanish and heuristics, machine learning, and word-level language information (wli) to find the optimal tag for each word. subsequent work on pos tagging for cm text like @xcite ; @xcite use similar approaches, using language information and a combination tagger on social media cm text. they emphasize that wli is a mandatory ingredient for cm pos tagging, necessitating the use of a combination tagger. recent work by @xcite applies a set of hand-crafted features (including wli) to pos tag cm text, with higher accuracy, and without using a combination tagger. this establishes that cm computational models can be independently built without using models from constituent languages. code-mixing is abundantly observed in our daily lives through personal chats, messages etc., apart from social media. since code-mixing is majorly used on social media, cm datasets are built primarily on social media data, which is domain-specific, noisy, and has non-standard spellings and grammar, especially for languages written in scripts other than roman @xcite . this makes annotation of such data a separate challenge in itself. additionally, extracting genuine cm data from social media is a difficult task, accounting for: (i) inadequate code-mixed datasets @xcite , (ii) small sizes of available cm datasets and (iii) low code-mixing ratio (cmr) (discussed in section 2) of such datasets @xcite . to address these existing bottlenecks of cm language research, through this work, we explore the following fundamental question: can we generate high-quality annotated cm data and build computational models with it, comparable to natural cm data and its subsequent computational models? to address the above question, we explore the dual task of annotated cm data generation and building computational models from this generated data, specifically for cm pos tagging. we introduce pacman (parallel codemixed data generation) for cm pos tagging. through pac-man, our motivation is to address the gaps in the existing cm datasets and computational models. first, we generate annotated cm pos tagged data for hinglish, through an alignment, annotation and replacement strategy from parallel hindi-english corpus. the generated data has standard spelling and grammar, unlike social media datasets. further, this alleviates the strenuous and difficult task of annotating social media cm data. cm datasets usually have sizes equal to or less than a meagre 1000 samples @xcite . we report a corpus of above 50k annotated pos tagged cm samples, which are 100% code-mixed. although our data is generated for hinglish, we claim that our technique can be used to generate annotated data for any cm language pair, provided a parallel corpus and pos taggers are available for both languages. to our knowledge, our work is a first-of-a-kind, with respect to the generation of cm annotated data. we discuss our data generation pipeline in detail in section 3. second, we build pos taggers using both machine learning (ml) and deep learning (dl) based techniques on pacman data. singh et al. (2018) reports better accuracy with ml techniques than dl models. through this work, we illustrate that this is predominantly due to the inadequacy of data, typically required to build better deep learning models. our dl model outperforms our ml model by 1.5%. we would like to assert here that we build our models independently on cm data itself and not using a combination tagging strategy. we also analyse the contribution of wli on the cm pos tagging task by infusing language information into the models. the results obtained reveal that with sizeable data, instances where language information is useful, dwindle to a small fraction, as we obtain similar accuracy without language information. we discuss our pos taggers in detail in section 4. third, in order to gauge the quality of our data and models, we test them against the existing benchmark in cm pos tagging @xcite , which is based on social media text. our models outperform the sota benchmark by 10%, despite having a higher cmr. our results also show that our models built on pacman data are able to handle social media text as well as monolingual text. this is largely because of the size of our dataset, standard spelling and grammar integral to pacman, and uniform distribution of words and pos tags across hindi and english. this makes our models more generalized and equipped to handle different types of cm data. we discuss qualitative analysis of our data in section 5, error analysis in section 6 and conclude in section 7. ## code-mixing terminology in this section we define a few terms akin to codemixing, that we use in later sections of the paper: ## data generation pipeline code-mixed language research has recently received a lot of attention in the nlp community. despite the recent interest, @xcite report that there is a scarcity of datasets in the domain, and existing datasets are very small ( 1000 samples per dataset on average). besides, as cm is prevalent on social media, the existing cm datasets contain only social media text, which have nonstandard spelling and grammar and are difficult to annotate. in order to bridge this gap, in the present work, we endeavour to generate an exhaustive corpus of annotated code-mixed data, that is generic across domains and has standardised spelling and grammar, unlike social media data. hindi-english parallel corpus data extraction and cleaning final set of parallel sentences pacman pos tagged dataset stanza pos tagger fast aligner replacement strategy back-transliteration aligned hi-en and en-hi sentences pos tagged hindi and english sentences parallel hindi sentences parallel english sentences we introduce pacman (parallel codemixed data generation a one-of-a-kind data generation framework for cm pos tagging. it is a synthetically generated data framework, curated via an alignment, annotation and replacement based strategy. our approach is similar to the one applied by @xcite , but with the added objective of uniform distribution of words across constituent languages of the cm language pair. currently, pacman is implemented for hinglish, but using the same strategy will work for any cm language pair. the subsequent subsections present the details involved in creating the pacman dataset. ## part-of-speech tagging in this section, we discuss the tagset used for pos annotation, details and performance of the sequence labeling models applied on the pacman dataset for word-level pos prediction. in addition, we also analyse the contribution of wli for the cm pos labeling task. ## pacman: qualitative evaluation in this section, we gauge the quality of our dataset and models against the existing benchmark for cm pos tagging i.e., linguistic code-switching evaluation (lince), reported by @xcite . since pacman is synthetically generated, it is crucial that its efficacy is tested against a benchmark social media dataset (lince), which is considered natural cm data. to this end, we devise a set of intricately designed experimental scenarios for this comparative investigation. ## pacman: error analysis with the motivation of bridging gaps in existing cm datasets and computational models we generate pacman, using an alignment, annotation and replacement strategy from parallel hindi-english corpus. although the generated data is clean, formalised, and yields impressive results, our analysis shows that the usage of probabilistic tools and resources adversely affects the quality of the generated dataset, although such cases are scarce. we highlight these inaccuracies with examples, for every stage of our data generation pipeline. inaccuracies in the parallel corpus: we observed a few errors in the hindi parallel sentences in the corpus like: 1. multiple devanāgari forms of same hindi words e.g., / -hai (is); / -nahi (no) 2. hindi translations of english words being merely their devanāgari forms e.g., ú (default) , ú º (database) this results in incorrect non-standard words in the dataset, from the transliterations (he (hai), nahin (nahi), difolt (default), databes (database)). inaccurate alignment: due to inherent constraints in fast align, sometimes the alignment results rendered are inaccurate. this leads to incorrect word replacements and redundant words, resulting in erroneous data. in the example below it can be observed that the word dhvani (sound) is redundant and incorrect, due to the inaccurate alignment of the words (dhvani) and event. ## conclusion and future-work in this paper, we address some of the existing limitations in the datasets and computational models for code-mixed languages, specifically for the cm pos tagging task. we propose a first-of-its-kind work in generating cm annotated data. we introduce the pacman dataset, generated using an alignment, annotation and replacement strategy from hindi-english parallel corpus. we claim that pacman is the largest cm annotated dataset (around 50k samples). although the generated dataset is for hinglish, the strategy can be transferred to any cm language pair, having available parallel corpus and pos taggers. the pacman data adheres to standard spelling and grammar, unlike social media data, primarily used in cm research work. the use of both hindi and english as matrix languages, ensures uniform distribution of words in both languages, and the potential to understand monolingual contexts. to establish the effectiveness of the dataset, we build both ml (crf) and dl (bilstm) based sequential labeling models on pacman data. unlike previous work, our dl model outperforms the ml model by 1.5%. we analyse the effect of word-level language information on the cm pos tagging task, which reveals that, with a larger dataset, cases where wli is crucial, are minuscule, thus elevating the need for wli in cm pos labeling. we validate our dataset against the existing benchmark dataset for cm pos tagging. our best model outperforms the sota benchmark by 10% and in all computational scenarios, signifying that our dataset is more generalised and capable of handling a wide spectrum of cm data as well as monolingual data. scrutiny of errors observed in our dataset manifests that inaccuracies in probabilistic tools and resources used as a part of the data generation pipeline, adversely affect the quality of the dataset, although such cases are scarce. in future, we endeavour to generate more data in order to build transformer-based models on pac-man, and use matrix language information to prevent errors actuated by structural differences between hindi and english.