GLUEC o S : An Evaluation Benchmark for Code-Switched NLP (2020)

Abstract: Code-switching is the use of more than one language in the same conversation or utterance. Recently, multilingual contextual embedding models, trained on multiple monolingual corpora, have shown promising results on cross-lingual and multilingual tasks. We present an evaluation benchmark, GLUECoS, for code-switched languages, that spans several NLP tasks in English-Hindi and English-Spanish. Specifically, our evaluation benchmark includes Language Identification from text, POS tagging, Named Entity Recognition, Sentiment Analysis, Question Answering and a new task for code-switching, Natural Language Inference. We present results on all these tasks using cross-lingual word embedding models and multilingual models. In addition, we fine-tune multilingual models on artificially generated code-switched data. Although multilingual models perform significantly better than cross-lingual models, our results show that in most tasks, across both language pairs, multilingual models fine-tuned on code-switched data perform best, showing that multilingual models can be further optimized for code-switching tasks.

## introduction code-switching, or code-mixing, is the use of more than one language in the same utterance or conversation and is prevalent in multilingual societies all over the world. it is a spoken phenomenon and is found most often in informal chat and social media on the internet. processing, understanding, and generating code-mixed text and speech has become an important area of research. recently, contextual word embedding models trained on a large amount of text data have shown state-of-the-art results in a variety of nlp tasks. models such as bert @xcite and its multilingual version, mbert, rely on large amounts of unlabeled monolingual text data to build monolingual and multilingual models that can be used for downstream tasks involving limited labelled data. @xcite propose a generalized language evaluation benchmark (glue) to evaluate embedding models on a wide variety of language understanding tasks. this benchmark has spurred research in monolingual transfer learning settings. data and annotated resources are scarce for codeswitched languages, even if one or both languages being mixed are high resource. due to this, there is a lack of standardized datasets in code-switched languages other than those used in shared tasks in a few language pairs. although models using synthetic code-switched data and cross-lingual embedding techniques have been proposed for codeswitching @xcite , there has not been a comprehensive evaluation of embedding models across different types of tasks. furthermore, there have been claims that multilingual models such as mbert are competent in zero-shot cross lingual transfer and code-switched settings. though comprehensively validated by @xcite in the case of zero-shot transfer, the probing in codeswitched settings was limited to one dataset of one task, namely pos tagging. to address all these issues and inspired by the glue @xcite benchmark, we propose gluecos, a language understanding evaluation framework for code-switched nlp. we include five tasks from previously conducted evaluations and shared tasks, and propose a sixth, natural language inference task for code-switching, using a new dataset foot_0 @xcite . we include tasks varying in complexity ranging from wordlevel tasks [language identification (lid); named entity recognition (ner)], syntactic tasks [pos tagging], semantic tasks [sentiment analysis; question answering] and finally a natural language inference task. where available, we include multiple datasets for each task in english-spanish and english-hindi. we choose these language pairs, not only due to the relative abundance of publicly available datasets, but also because they represent variations in types of code-switching, language families, and scripts between the languages being mixed. we test various cross-lingual and multilingual models on all of these tasks. in addition, we also test models trained with synthetic codeswitched data. lastly, we fine-tune the best performing multilingual model with synthetic codeswitched data and show that in most cases, its performance exceeds the multilingual model, highlighting that multilingual models can be further optimized for code-switched settings. the main contributions of our work are as follows: the rest of the paper is organized as follows. we relate our work to prior work to situate our contributions. we introduce the tasks and datasets used for gluecos motivating the choices we make. we describe the experimental setup, with details of the models used for baseline evaluations. we present the results of testing all the models on the benchmark and analyze the results. we conclude with a direction for future work and highlight our main findings. ## relation to prior work the idea of a generalized benchmark for codeswitching is inspired by glue @xcite , which has spurred research in natural language understanding in english, to an extent that a set of harder tasks have been curated in a follow-up benchmark, superglue @xcite once models beat the human baseline for glue. the motivation behind glue is to evaluate models in a multi-task learning framework across several tasks, so that tasks with less training data can benefit from others. although our current work does not include models evaluated in a multi-task setting, we plan to implement this in subsequent versions of the benchmark. there have been shared tasks conducted in the past as part of code-switching workshops co-located with notable nlp conferences. the first and second workshops on computational approaches to code switching @xcite conducted a shared task on language identification for several language pairs @xcite . the third workshop @xcite included a shared task on named entity recognition for the english-spanish and modern standard arabic-egyptian arabic language pairs @xcite . the forum for information retrieval evaluation (fire) aims to meet new challenges in multilingual information access and has conducted several shared tasks on code-switching. these include tasks on transliterated search, @xcite code-mixed entity extraction (rao and devi, 2016) and mixed script information retrieval @xcite . other notable shared tasks include the tool contest on pos tagging for code-mixed indian social media at icon 2016 @xcite , sentiment analysis for indian languages (code-mixed) at @xcite and the code-mixed question answering challenge @xcite . each of the shared tasks mentioned above attracted several participants and have led to follow up research in these problems. however, all tasks have focused on a single nlp problem and so far, there has not been an evaluation of models across several code-switched nlp tasks. our objective with proposing gluecos is to address this gap, and determine which models best generalize across different tasks, languages and datasets. due to the lack of standardized datasets, we choose to create our own train-test-validation splits for some tasks. also, we use an off-the-shelf transliterator and language detector, where necessary, details of which can be found in appendix a. in cases where the datasets have been a part of shared tasks, we report the highest scores obtained in each task as the state of the art (sota) for the dataset. however, note that we report this to situate our results in context of the same, and these cannot be directly compared, since each task's sota is obtained by varied training architecture, suited to perform well in one particular task alone. ## experimental setup we use standard architectures for solving each of the tasks mentioned above (refer to appendix b). we experiment with several existing cross lingual word embeddings that have been shown to perform well on cross lingual tasks. we also experiment with the multilingual bert (mbert) model released by @xcite . in a survey on cross lingual word embeddings, @xcite establish that various embedding methods optimize for similar objectives given that the supervision data involved in training them is similar. based on this, we choose the following representative embedding methods that vary in the amount of supervision involved in training them. ## results and analysis tables 3-8 show the results of using the embedding techniques described above for each task and dataset. mbert provides a large increase in accuracy as compared to cross-lingual techniques, and in most cases, the modified mbert technique performs best. we do not experiment with baseline or cross-lingual embedding techniques for nli, since we find that mbert surpasses the other techniques for all other tasks. for nli, as in the other cases, we find that modified mbert performs better than mbert. we hypothesize that this happens because code-switched languages are not just a union of two monolingual languages. the distributions and usage of words in code-switched languages differ from their monolingual counterparts, and can only be captured with real code-switched data, or synthetically generated data that closely mimics real data. @xcite point out how all crosslingual word embedding methods optimize for bilingual lexicon induction. each model is trained using different language pairs and different training and evaluation dictionaries, leading to it overfitting to the task it is optimizing for and failing in other cross-lingual scenarios. also, the loss function in training cross-lingual word embeddings has a component where w1 in one language predicts the context of its aligned word w2 in the other language. however, in the case of code-switching, w1 appear overall, the cross-lingual and mbert models perform better for english-spanish as compared to english-hindi. this could be due to several reasons. we find that for most tasks, modified mbert performs better than mbert. in cases where this is not true (qa en-hi; fg en-hi), the difference in accuracy between the two models is small. this could be attributed to errors made by the transliterator or corpus differences, but in general we observe that the modified en-hi mbert model does not significantly outperform the base mbert model. given the promising results obtained by modified mbert, it would be interesting to pre-train a language model for code-switched data which is trained on the monolingual corpora of languages involved and fine-tuned on gcm as proposed, to compare against fine-tuning mbert itself, which is trained on multiple languages. we find that accuracies vary across tasks in the gluecos benchmark, and except in the case of lid, code-switched nlp is far from solved. this is particularly stark in the case of sentiment and nli, which are three and two way classification tasks respectively. modified mbert performs only a little over chance, which shows that we are still in the early days of solving nli for code-switched languages, and also indicates that our models are far from truly being able to understand code-switched language. ## conclusion in this paper, we introduce the first evaluation benchmark for code-switching, gluecos. the benchmark contains datasets in english-hindi and english-spanish for six nlp tasks -lid, pos tagging, ner, sentiment analysis, question answering and a new code-switched natural language inference task. we test various embedding techniques across all tasks and datasets and find that multilingual bert outperforms cross-lingual embedding techniques on all tasks. we also find that for most datasets, a modified version of mbert that has been fine-tuned on synthetically generated code-switched data with a small amount of real code-switched data performs best. this indicates that while multilingual models do go a long way in solving code-switched nlp, they can be improved further by using real and synthetic code-switched data, since the distributions in code-switched languages differ from the two languages being mixed. in this work, we use standard architectures to solve each nlp task individually and vary the embeddings used. in future work, we would like to experiment with a multi-task setup wherein tasks with less training data can significantly benefit from those having abundant labelled data, since most code-switched datasets are often small and difficult to annotate. we experiment with datasets having varied amounts of code-switching and from different domains and show that some tasks, such as lid and pos tagging are relatively easier to solve, while tasks such as qa and nli have low accuracies. we would like to add more diverse tasks and language pairs to the gluecos benchmark in a future version. all the datasets used in the gluecos benchmark are publicly available, and we plan to make the nli dataset available for research use. we hope that this will encourage researchers to test multilingual, cross-lingual and code-switched embedding techniques and models on this benchmark.