Contextual Embeddings for A rabic- E nglish Code-Switched Data (2020)

Abstract: Globalization has caused the rise of the code-switching phenomenon among multilingual societies. In Arab countries, code-switching between Arabic and English has become frequent, especially through social media platforms. Consequently, research in Natural Language Processing (NLP) systems increased to tackle such a phenomenon. One of the significant challenges of developing code-switched NLP systems is the lack of data itself. In this paper, we propose an open source trained bilingual contextual word embedding models of FLAIR, BERT, and ELECTRA. We also propose a novel contextual word embedding model called KERMIT, which can efficiently map Arabic and English words inside one vector space in terms of data usage. We applied intrinsic and extrinsic evaluation methods to compare the performance of the models. Our results show that FLAIR and FastText achieve the highest results in the sentiment analysis task. However, KERMIT is the best-achieving model on the intrinsic evaluation and named entity recognition. Also, it outperforms the other transformer-based models on question answering task.

## introduction nowadays, due to globalization, bilingual communities tend to code-switch in everyday communication. code-switching (cs) is defined as the embedding of linguistic units such as phrases, words, and morphemes of one language into an utterance of another language @xcite . this linguistic behavior occurs on both spoken and written scales. primary language appears the most inside an utterance, while secondary language is the language of embedded words or phrases inside an utterance. the phenomenon of code-switching has been increasingly reported in linguistic studies in the past years as more people tend to code-switch. furthermore, this phenomenon has become popular in arab countries, where people code-switch between different dialects or their dialect and foreign languages. for instance, it is common to mix between arabic and french in tunisia and egyptian arabic and english in egypt. cs behavior is common in online interaction, especially among social media users, generating vast amounts of cs data @xcite . for example, on twitter, the multilingual users are more active than the monolingual ones @xcite . thus, it is essential nowadays to process and understand the natural language of humans. this is performed by applying several natural language processing (nlp) techniques while using different deep learning approaches. lately, cs data started to gain attention in different nlp tasks @xcite @xcite . one of the powerful developments in the nlp field is word embeddings; they represent words as vectors in a continuous space. this leads to having semantically similar words grouped near each other. it has been proven that adding word embedding instead of the traditional bag-of-words approach to different nlp tasks is very efficient @xcite . recently, bilingual word embeddings gained much attention to embedding in the same space words from two languages. most of the standard bilingual word embedding techniques are intended to be trained and work on monolingual texts, not on a mix of two languages. thus, they are not the ideal option to learn embeddings for code-switched tasks @xcite . as our focus in this work is on the arabic-english cs data, previously @xcite developed a classical word embedding model called bi-cs. this model produced classical word embeddings by training on cs arabic-english corpus. it was trained on monolingual data and a small amount of cs data, which acted as a gluing force bringing the monolingual embeddings closer in the vector space. however, classical word embeddings compute static vectors for each word. in polysemous words that depend on the context, classical word embeddings fail to model these words. contextual embeddings, on the other hand, compute context-dependent nature vectors for words. this work aims to train bilingual contextual embedding models using state-of-the-art embedding types generated from our collected codeswitched arabic-english corpus. the models we created were using flair, bert, and electra. we also proposed a new contextual word embedding model called kermit based on the previous work @xcite , capable of mapping both arabic and english words inside one vector space efficiently in terms of data usage. all our trained and proposed models are available as an open source foot_0 . to compare our bilingual embedding models, we applied intrinsic and extrinsic evaluation methods. in the intrinsic evaluation, the models are tested on their ability to assign near-by vectors to similar tokens. this was achieved by calculating the cosine similarity between the tokens using berts core @xcite . in the extrinsic evaluation, we evaluated our models on three downstream nlp tasks; named entity recognition, sentiment analysis, and question answering on arabic-english cs text. as a result of the intrinsic evaluation, the best model is kermit, which is capable of modeling higher cosine-similarity with similar code-switched words and lower for unrelated ones. in addition, the system we implemented using kermit is the best-achieving model for the ner task on arabic-english data. regarding the sentiment analysis, flair and fasttext achieved higher results compared to all other models. this shows that the character embedding outperformed the different types of embedding in some tasks on cs data. results also show that the kermit model achieved the highest results on question answering task across the transformer-based models. the paper is organized as follows. section 2 presents some related work for word embedding models. in section 3, the process of data collection and corpus creation is illustrated. section 4 explains the different types of embedding models we trained. section 5 discusses the different evaluation methods and their results. finally, section 7 concludes the paper and presents future work. ## related work some studies implemented different word embedding models to deal with various arabic nlp tasks. for example, @xcite implemented arabic pre-trained word embedding models. their first version contained six distinct models built using either continuous bag-of-words (cbow) or skip-gram (sg) techniques for the different text domains. they measured the similarity of word vectors of a subset of sentiment words and named entities to evaluate their models. then they applied the clustering technique to see if, in each set, the words with the same polarity will be clustered together or no. they also used semeval-2017 semantic textual similarity to check how equivalent paired snippets of text. another set of word embedding models for the arabic language is presented in @xcite . the models are implemented using cbow, sg, and glove techniques, and they are generated from a set of arabic tweets. they proposed a new way to measure the similarity of the arabic words to evaluate the performance of their proposed models. besides, they tested the performance in the multi-class sentiment analysis classification task. bilingual models have been explored as a link to bridge the gap between languages in cs word embedding and building models for low-resource languages. several bilingual models have been proposed to align cross-lingual data, they used different alignment techniques such as word-level @xcite , sentence-level @xcite , both word and sentence level @xcite and document-level alignments @xcite . besides @xcite presented a comparison between four different cross-lingual embeddings models @xcite @xcite , they vary in terms of the amount of supervision. @xcite compared between the three bilingual models @xcite @xcite for enhancing the downstream tasks of sentiment analysis and pos tagging on english-spanish cs data. moreover, they proposed an approach for training cs text generated by @xcite using skip-grams. in @xcite , they proposed several arabic-english cross-lingual word embedding models trained on pairs of arabic-english parallel sentences. it is essential to mention that training cross-lingual and multilingual embeddings require monolingual data. the set of syntactic structures and semantic associations of code-switched text are not shown in monolingual sentences. thus, learning cs text analysis using cross-lingual and multilingual embeddings is not optimal. code-switched embedding should be trained using cs text @xcite . @xcite proposed an approach to employ the bert model and generative adversarial net model for cs text generation. the developed system is capable of generating full cs sentences by training on low resourced cs data. through this process, the bert model learns contextual embedding representation. they evaluated their generated data in an asr system on mandarin-english cs data. a multi-encoder-decoder transformer model was proposed in @xcite for cs data. this model involves two language-specific encoder modules, each trained on monolingual data. these two modules are then integrated and trained on low resourced cs data. eventually, this module is capable of representing contextual embedding. related to our work for arabic-english cs data, @xcite compared different bilingual embeddings @xcite @xcite having different cross-lingual supervision. they also proposed two extensions, one of which depends on monolingual and small cs corpora, and another one combines the first two approaches. they evaluated the effect of using different embeddings in language modeling. however, the proposed embeddings are classical ones and do not consider the context of the words. ## data collection to train the word embedding models using arabic-english code-switched text, we collected the data using three different techniques/sources and created two corpora. the first corpus cs train is composed of 105 million tokens. the first source of data was using the cs corpus of @xcite collected from social media platforms. the second technique is generating 30 million arabic-english cs tokens by translating monolingual modern standard arabic corpus into arabic-english cs data. we iterated automatically over the corpus and translated tokens according to a set of linguistic constraints using an open-sourced neural machine translation api foot_1 . we followed the linguistic constraints inferred from evaluating real arabic-english code-switched data in @xcite . they defined trigger words such as ( , the), ( , in) and ( , and), which are arabic words preceding a cs point. the final part of the corpus is composed from the monolingual arabic news-wire data. to augment the size of the training data, we created another corpus cs train ++ by adding to the initial one cs train 20 million tokens from the same arabic monolingual news-wire data-set. also, we translated them following the same linguistic constraints to form the other 19 million cs tokens. we trained several bilingual contextual embedding models to compare them and have an efficient model for arabic-english cs data used in several nlp tasks. we used several state-of-the-art techniques and the type of contextual embeddings for building our models. we started by creating a baseline model using a stack of arabic pre-trained pooled flair embedding and pre-trained fasttext. the other models we built are using flair, bert, and electra. we also proposed a new model called the kermit. ## evaluation & results to evaluate our models, we have used intrinsic and extrinsic methods. we first trained the models using the cs train and then using cs train ++ . for clarity, all models with a suffix ++ have been trained using cs train ++ . ## conclusion & future work word embedding is one of the cores of all nlp tasks. pre-training word embeddings on extensive unlabelled data and using it on downstream tasks can save a lot of data annotation. however, it is a problem to find a large data-set when it comes to low resourced languages such as code-switched arabic-english data. in this paper, we propose a solution to create a large corpus for pre-training word embeddings through several sources. we used this corpus for training different bilingual contextual embedding models: flair, bert, and electra. we also proposed a new model kermit. this model is trained as both a discriminator and a generator where each task has a different variant of the input. the different variants of the data helped the kermit model to train efficiently on the collected corpus. we have pre-trained and evaluated all the models on three downstream tasks. the character based-model flair, along with fasttext, outperformed all models in the task of sentiment analysis. however, ker-mit got better results in the ner task and outperformed the other transformer-based models in the question-answering task. besides, we evaluated the performance of the models by calculating the cosine similarities between different tokens, and kermit captured the highest similarities. in the future, more comparisons with available models could be done. we can pre-train variants of the model by keeping the monolingual data without cs text and compare the performance in different tasks. the models will be evaluated in different tasks using various annotated data-sets. an evaluation technique should be applied on the generated cs data to check how it simulates the cs behaviour.