C odemixed NLP : An Extensible and Open NLP Toolkit for Code-Mixing (2021)

Abstract: The NLP community has witnessed steep progress in a variety of tasks across the realms of monolingual and multilingual language processing recently. These successes, in conjunction with the proliferating mixed language interactions on social media, have boosted interest in modeling code-mixed texts. In this work, we present CodemixedNLP, an open-source library with the goals of bringing together the advances in code-mixed NLP and opening it up to a wider machine learning community. The library consists of tools to develop and benchmark versatile model architectures that are tailored for mixed texts, methods to expand training sets, techniques to quantify mixing styles, and fine-tuned state-of-the-art models for 7 tasks in Hinglish. We believe this work has the potential to foster a distributed yet collaborative and sustainable ecosystem in an otherwise dispersed space of code-mixing research. The toolkit is designed to be simple, easily extensible, and resourceful to both researchers as well as practitioners. Demo: http://k-ikkees.pc.cs.cmu.edu:5000 and Library: https://github.com/murali1996/CodemixedNLP

## introduction code-mixing refers to fluid alteration between two or more languages in a given utterance. this phenomenon is ubiquitous and more natural in multilingual communities, and is highly prevalent in social media platforms. developing tools that can comprehend mixed texts can have a multitude of advantages, ranging from socially responsible nlp applications such as moderating abusive content in social media to improve naturalness of ubiquitous technologies such as conversational ai assistants and further to develop socio-cultural studies around human cognition, such as why and when people code-mix. nlp tools for monolingual and multilingual language processing have rapidly progressed in the past few years; thanks to the transformer-based models such as multilingual bert @xcite & xlm-roberta @xcite , and their pretraining techniques. on various mixed datasets, recent studies have shown that adopting multilingual pretrained models can perform better than their previous deep learning counterparts @xcite @xcite @xcite . while this looks promising for multilingual, the same is not translated to code-mixing. hence, a critical investigation is required to understand generalizable modeling strategies to enhance performance on mixed texts @xcite @xcite . at the same time, practitioners who require an off-the-shelf tool into their downstream mixed text application (eg. in this work, we address some of these shortcomings by creating an extensible and open-source toolkit for a variety of semantic and syntactic nlp applications in mixed languages. our toolkit offers- we believe the fine grained plug and play interfacing of the toolkit can serve a multitude of purposes in both academia and industry. such fine control over the individual components of the model can enable accelerated experimentation in training different model architectures, such as multi-tasking, representation-fusion, and language-informed modeling. this in-turn helps our understanding of utilizing pretrained transformer-models for mixed datasets. in addition, our toolkit also offers computation of metrics to quantify code-mixing such as code-mixing index, language entropy, etc., which can be utilized to find peculiarities of low-performing subsets. like a curse in disguise, though code-mixing is widely prevalent and available on social media, it is accompanied with non-standard spellings, mixed scripts and ill-formed sentences are common in code-mixing. to combat this, our toolkit offers techniques to augment the training sets with multiple views of each input corresponding to the above problems. among many potential applications, we first demonstrate our toolkit's utility in benchmarking ( § 3). in addition, we publish state-of-the-art models for different nlp tasks in hinglish and wrap them into a command line / deployable web interface ( § 4). our toolkit is easily extensiblepractitioners can incorporate new pretrained as well as fine-tuned models, include text processors such as tokenizers, transliterators and translators, and add wrappers on existing methods for downstream nlp applications. ## toolkit our toolkit is organized into components as depicted in figure 1 . in a nutshell, an end-to-end model architecture consists of one or more encoder components, a component for combining encodings, and one or more adaptor plus task components. ## experiments among many potential research applications of our toolkit, in this section, we demonstrate one benchmarking. ## demo we fine-tune and publish transformer-based models for 7 tasks in hinglish. we include 3 task types-(1) semantic (sentiment classification, hate speech and aggression identification), (2) syntactic (ner, pos and language identification), and (3) generation (@xmath0. we present some examples of utilizing these models in figure 2 . ## conclusion in this work, we presented a unified toolkit for modeling code-mixed texts. additionally, the toolkit contains various functionalities such as data augmentation, code-mixing quantification, and ready-to-use fine-tuned models for 7 different nlp tasks in hinglish. our toolkit is simple enough for practitioners to integrate new features as well as develop wrappers around its existing functionalities. we believe this contribution facilitates a sustainable and extensible ecosystem of models by adding novel pretraining techniques tailored for mixed texts, text normalization techniques to counter spelling variations, error analysis tools to identify peculiarities in incorrect predictions and so on.