From E nglish to Code-Switching: Transfer Learning with Strong Morphological Clues (2020)

Abstract: Linguistic Code-switching (CS) is still an understudied phenomenon in natural language processing. The NLP community has mostly focused on monolingual and multi-lingual scenarios, but little attention has been given to CS in particular. This is partly because of the lack of resources and annotated data, despite its increasing occurrence in social media platforms. In this paper, we aim at adapting monolingual models to code-switched text in various tasks. Specifically, we transfer English knowledge from a pre-trained ELMo model to different code-switched language pairs (i.e., Nepali-English, Spanish-English, and Hindi-English) using the task of language identification. Our method, CS-ELMo, is an extension of ELMo with a simple yet effective position-aware attention mechanism inside its character convolutions. We show the effectiveness of this transfer learning step by outperforming multilingual BERT and homologous CS-unaware ELMo models and establishing a new state of the art in CS tasks, such as NER and POS tagging. Our technique can be expanded to more English-paired code-switched languages, providing more resources to the CS community.

## introduction although linguistic code-switching (cs) is a common phenomenon among multilingual speakers, it is still considered an understudied area in natural language processing. the lack of annotated data combined with the high diversity of languages in which this phenomenon can occur makes it difficult to strive for progress in cs-related tasks. even though cs is largely captured in social media platforms, it is still expensive to annotate a sufficient amount of data for many tasks and languages. additionally, not all the languages have the same incidence and predominance, making annotations impractical and expensive for every combination spanish-english tweet original: @mrolvera06 other @t11gre other go too cavenders ne y tambien ve a @eltoroboots ne other english: @mrolvera06 @t11gre go to cavenders and also go to @eltoroboots figure 1 : examples of code-switched tweets and their translations from the cs lid corpora for hindi-english, nepali-english and spanish-english. the lid labels ne and other in subscripts refer to named entities and punctuation, emojis or usernames, respectively (they are part of the lid tagset). english text appears in italics and other languages are underlined. in this paper, we study the cs phenomenon using english as a starting language to adapt our models to multiple code-switched languages, such as nepali-english, hindi-english and spanish-english. in the first part, we focus on the task of language identification (lid) at the token level using elmo @xcite as our reference for english knowledge. our hypothesis is that english pre-trained models should be able to recognize whether a word belongs to english or not when such models are fine-tuned with codeswitched text. to accomplish that, we introduce cs-elmo, an extended version of elmo that contains a position-aware hierarchical attention mechanism over elmo's character n-gram representations. these enhanced representations allow the model to see the location where particular n-grams occur within a word (e.g., affixes or lemmas) and to associate such behaviors with one language or another. 1 with the help of this mechanism, our models consistently outperform the state of the art on lid for nepali-english @xcite , spanish-english @xcite , and hindi-english @xcite . moreover, we conduct experiments that emphasize the importance of the position-aware hierarchical attention and the different effects that it can have based on the similarities of the code-switched languages. in the second part, we demonstrate the effectiveness of our cs-elmo models by further fine-tuning them on tasks such as ner and pos tagging. specifically, we show that the resulting models significantly outperform multilingual bert and their homologous elmo models directly trained for ner and pos tagging. our models establish a new state of the art for hindi-english pos tagging @xcite and spanish-english ner @xcite . our contributions can be summarized as follows: 1) we use transfer learning from models trained on a high-resource language (i.e., english) and effectively adapt them to the code-switching setting for multiple language pairs on the task of language identification; 2) we show the effectiveness of transferring a model trained for lid to downstream code-switching nlp tasks, such as ner and pos tagging, by establishing a new state of the art; 3) we provide empirical evidence on the importance of the enhanced character n-gram mechanism, which aligns with the intuition of strong morphological clues in the core of elmo (i.e., its convolutional layers); and 4) our cs-elmo model is self-contained, which allows us to release it for other researchers to explore and replicate this technique on other code-switched languages. foot_1 ## related work transfer learning has become more practical in the last years, making possible to apply very large neural networks to tasks where annotated data is limited @xcite @xcite . cs-related tasks are good candidates for such applications, since they are usually framed as low-resource problems. however, previous research on sequence labeling for code-switching mainly focused on traditional ml techniques because they performed better than deep learning models trained from scratch on limited data @xcite . nonetheless, some researchers have recently shown promising results by using pre-trained monolingual embeddings for tasks such as ner @xcite and pos tagging @xcite . other efforts include the use of multilingual sub-word embeddings like fasttext @xcite for lid @xcite , and cross-lingual sentence embeddings for text classification like laser @xcite @xcite , which is capable of handling code-switched sentences. these results show the potential of pre-trained knowledge and they motivate our efforts to further explore transfer learning in code-switching settings. our work is based on elmo @xcite , a large pre-trained language model that has not been applied to cs tasks before. we also use attention @xcite within elmo's convolutions to adapt it to code-switched text. even though attention is an effective and successful mechanism in other nlp tasks, the code-switching literature barely covers such technique @xcite . @xcite use a different attention method for ner, which is based on a gated cell that learns to choose appropriate monolingual embeddings according to the input text. @xcite proposed multilingual meta embeddings (mme) combined with self-attention @xcite . their method establishes a state of the art on spanish-english ner by heavily relying on monolingual embeddings for every language in the code-switched text. our model outperforms theirs by only fine-tuning a generic cs-aware model, without relying on task-specific designs. another contribution of our work are position embeddings, which have not been considered for code-switching either. these embeddings, combined with cnns, have proved useful in computer vision @xcite ; they help to localize non-spatial features extracted by convolutional networks within an image. we apply the same prin-ciple to code-switching: we argue that character n-grams without position information may not be enough for a model to learn the actual morphological aspects of the languages (e.g., affixes or lemmas). we empirically validate those aspects and discuss the incidence of such mechanism in our experiments. ## methodology elmo is a character-based language model that provides deep contextualized word representations @xcite . we choose elmo for this study for the following reasons: 1) it has been trained on a large amount of english data as a general-purpose language model and this aligns with the idea of having english knowledge as starting point; 2) it extracts morphological information out of character sequences, which is essential for our case since certain character n-grams can reveal whether a word belongs to one language or another; and 3) it generates powerful word representations that account for multiple meanings depending on the context. nevertheless, some aspects of the standard elmo architecture could be improved to take into account more linguistic properties. in section 3.1, we discuss these aspects and propose the position-aware hierarchical attention mechanism inside elmo. in section 3.2 and section 3.3, we describe our overall sequence labeling model and the training details, respectively. ## experiments we describe our experiments for lid in section 5.1, including insights of the optimized models. in section 5.2, the optimized lid models are further fine-tuned on downstream nlp tasks, such as ner and pos tagging, to show the effectiveness of our preliminary cs adaptation step. we test for statistical significance across our incremental experiments following @xcite , and we report p-values below 0.02 for lid. we discuss hyperparameters and fine-tuning details in appendix d. ## conclusion and future work we present a transfer learning method from english to code-switched languages using the lid task. our method enables large pre-trained models, such as elmo, to be adapted to code-switching settings while taking advantage of the pre-trained knowledge. we establish new state of the art on lid for nepali-english, spanish-english, and hindi-english. additionally, we show the effectiveness of our cs-elmo model by further fine-tuning it for ner and pos tagging. we outperform multilingual bert and homologous elmo models on spanish-english ner and hindi-enlgish pos tagging. in our ongoing research, we are investigating the expansion of this technique to language pairs where english may not be involved. appendix for "from english to code-switching: transfer learning with strong morphological clues"