Are Multilingual Models Effective in Code-Switching? (2021)

Abstract: Multilingual language models have shown decent performance in multilingual and cross-lingual natural language understanding tasks. However, the power of these multilingual models in code-switching tasks has not been fully explored. In this paper, we study the effectiveness of multilingual language models to understand their capability and adaptability to the mixed-language setting by considering the inference speed, performance, and number of parameters to measure their practicality. We conduct experiments in three language pairs on named entity recognition and part-of-speech tagging and compare them with existing methods, such as using bilingual embeddings and multilingual meta-embeddings. Our findings suggest that pre-trained multilingual models do not necessarily guarantee high-quality representations on code-switching, while using meta-embeddings achieves similar results with significantly fewer parameters.

## introduction learning representation for code-switching has become a crucial area of research to support a greater variety of language speakers in natural language processing (nlp) applications, such as dialogue system and natural language understanding (nlu). code-switching is a phenomenon in which a person speaks more than one language in a conversation, and its usage is prevalent in multilingual communities. yet, despite the enormous number of studies in multilingual nlp, only very few focus on code-switching. recently, contextualized language models, such as mbert @xcite and xlm-r @xcite have achieved stateof-the-art results on monolingual and cross-lingual tasks in nlu benchmarks @xcite @xcite @xcite . however, the effectiveness of these multilingual language models on code-switching tasks remains unknown. several approaches have been explored in code-switching representation learning in nlu. character-level representations have been utilized to address the out-of-vocabulary issue in codeswitched text @xcite , while external handcrafted resources such as gazetteers list are usually used to mitigate the low-resource issue in code-switching @xcite ; however, this approach is very limited because it relies on the size of the dictionary and it is language-dependent. in another line of research, meta-embeddings have been used in code-switching by combining multiple word embeddings from different languages @xcite . this method shows the effectiveness of mixing word representations in closely related languages to form languageagnostic representations, and is considered very effective in spanish-english code-switched named entity recognition tasks, and significantly outperforming mbert @xcite with fewer parameters. while more advanced multilingual language models @xcite than multilingual bert @xcite have been proposed, their effectiveness is still unknown in codeswitching tasks. thus, we investigate their effectiveness in the code-switching domain and compare them with the existing works. here, we would like to answer the following research question, "which models are effective in representing code-switching text, and why?." in this paper, we evaluate the representation quality of monolingual and bilingual word embeddings, multilingual meta-embeddings, and multilingual language models on five downstream tasks on named entity recognition (ner) and part-ofspeech tagging (pos) in hindi-english, spanish-english, and modern standard arabic-egyptian. we study the effectiveness of each model by considering three criteria: performance, speed, and the number of parameters that are essential for practical applications. here, we set up the experimental setting to be as language-agnostic as possible; thus, it does not include any handcrafted features. our findings suggest that multilingual pretrained language models, such as xlm-r base , achieves similar or sometimes better results than the hierarchical meta-embeddings (hme) @xcite ) model on code-switching. on the other hand, the meta-embeddings use word and subword pre-trained embeddings that are trained using significantly less data than mbert and xlm-r base and can achieve on par performance to theirs. thus, we conjecture that the masked language model is not be the best training objective for representing code-switching text. interestingly, we found that xlm-r large can improve the performance by a great margin, but with a substantial cost in the training and inference time, with 13x more parameters than hme-ensemble for only around a 2% improvement. the main contributions of our work are as follows: ## representation models in this section, we describe multilingual models that we explore in the context of code-switching. figure 1 shows the architectures for a word embeddings model, a multilingual language model, and the multilingual meta-embeddings (mme), and hme models. ## experiments in this section, we describe the details of the datasets we use and how the models are trained. ## conclusion in this paper, we study multilingual language models' effectiveness so as to understand their capability and adaptability to the mixed-language setting. we conduct experiments on named entity recognition and part-of-speech tagging on various language pairs. we find that a pre-trained multilingual model does not necessarily guarantee highquality representations on code-switching, while the hierarchical meta-embeddings (hme) model achieve similar results to mbert and xlm-r base but with significantly fewer parameters. interestingly, we find that xlm-r large has better performance by a great margin, but with a substantial cost in the training and inference time, using 13x more parameters than hme-ensemble for only a 2% improvement.