Much Gracias: Semi-supervised Code-switch Detection for S panish- E nglish: How far can we get? (2021)

Abstract: Because of globalization, it is becoming more and more common to use multiple languages in a single utterance, also called code-switching. This results in special linguistic structures and, therefore, poses many challenges for Natural Language Processing. Existing models for language identification in code-switched data are all supervised, requiring annotated training data which is only available for a limited number of language pairs. In this paper, we explore semi-supervised approaches, that exploit out-of-domain mono-lingual training data. We experiment with word uni-grams, word n-grams, character n-grams, Viterbi Decoding, Latent Dirichlet Allocation, Support Vector Machine and Logistic Regression. The Viterbi model was the best semi-supervised model, scoring a weighted F1 score of 92.23%, whereas a fully supervised state-of-the-art BERT-based model scored 98.43%.

## introduction social platforms have been the cradle of the internet, driving vast amounts of communication among people from all over the world. as a consequence, the way people communicate in written text has changed, as now it is common to use, for example, abbreviations of words, emoticons, references to other users and use multiple languages within the same utterance. an annotated example sentence of this is the following tweet: word el online exercise de hoy : label es en en es es other this phenomenon has caught particular interest in both sociolinguistics and natural language processing (nlp) @xcite . classifying the language labels on the word level (i.e. code-switch detection) has shown to be beneficial to improve performance on downstream nlp tasks, like dependency parsing @xcite or lexical normalization @xcite . previous work has shown that high performances can be achieved for this task for many language pairs @xcite . however, to the best of our knowledge, most previous work focused on supervised settings, restraining their usefulness to language pairs for which annotated datasets exist. recent efforts to unify existing datasets have collected annotation for 4 @xcite and 2 @xcite language pairs, which confirms that annotated data is not available for most language pairs. in supervised settings, recent transformer models @xcite have reached a new state-of-the-art @xcite , outperforming bi-lstms and traditional machine learning methods used earlier @xcite . @xcite tackled this task in a semi-supervised setup as well, where they used character n-gram language models trained on monolingual data to predict perplexity on the target word for classification. they show that this obtains a micro-average f1 score of 92.9%, compared to 95.6% with a supervised crf-model. to overcome this limitation, we focus on exploiting only mono-lingual datasets for performing word-level language identification in code-switched data. we refer to this setup as semi-supervised, since we have no data annotated for the task at hand (code-switch detection). this enables the possibility to easily train models for new language pairs, and leads to the research question: how do semi-supervised models compare and perform in the task of language identification in english-spanish code-switched data? (rq1). since supervised methods have the advantage of learning from annotated data, the second research question is: how much can we reduce the gap in performance between the aforementioned semisupervised models and a supervised state-of-the-art model? (rq2). previous work in similar setups have automatically generated code-switched data from monolingual datasets @xcite . we consider this approach to be orthogonal to ours, and @xcite exploit mono-lingual in-domain data, syntactic parsers and parallel sentences. ## datasets in this section, we will first describe the manually annotated code-switched data that we use for evaluating our models, then we describe the monolingual data that we will use as "training" data. it should be noted that this is not real training data, as it is not annotated for the task at hand (thus the setting is semi-supervised). ## results to evaluate the performance of our models, we use weighted f1 score foot_6 . as found in we use multilingual bert and all default settings. results in when comparing common confusions of our best semi-supervised model (viterbi) to the output of machamp, we found that there was more confusion in the viterbi model about other, where 213 words were classified as lang1 and 60 as lang2 instead, compared to just 3 and 1 in machamp. full confusion matrices can be found in the appendix. the majority voting ensembling models do not lead to improved performance. however, the oracle ensemble, which always picks the correct label if it is available from one of the models, shows that there is potential in improving the selection method for ensembling. ## discussion when inspecting the performances of the models per class (see also regarding the lda model, its low performance can be explained by the results of @xcite , which show that for the task of language filtering, the performance of lda decreases when the dominating language decreases under 70% of the whole text. this is also the case in our experiments, where the test data had a 54% english and 46% spanish ratio. furthermore, the amount of evidence per sample is rather low compared to the normal use of lda (it is commonly used on the document level). for character n-grams, we observed that the more we increased the value of n, the better results we got, up until @xmath0. the higher order n-grams performed better with around 12% difference in validation weighted f1 score, as we can capture groups of letters that are representative for a language, e.g. 'tion' in english and 'cion' in spanish. this model achieves good results also because it addresses the problem of misspelled words. for word n-grams, using tri-grams resulted in worse predictions than using bi-grams with around 11% difference in validation weighted f1 score. for lda, svm and logistic regression models we tried to vectorize data with countvectorizer from scikit learn, which gives the termfrequency for each n-gram in a word. however, tfidfvectorizer performed approximately 1% better in lda and logistic regression and 4% for svm in validation data. this was then the preferred vectorizer in all models, as it helps decreasing the impact of very frequent character n-grams that are not expressing much value and gives more importance to less frequent character n-grams. the fact that the oracle model has a 3% higher weighted f1 score than the best model (in validation data), suggests that there is room for improvement for the ensemble model with other methods than majority voting. improvements on the single models could be achieved by using bigger monolingual datasets of the same size or selecting a corpus that is more similar to the test set (social media-like posts), which is not as easy to query as wikipedia articles. the overall performance of the models can also be slightly improved by a more complex method for the other class (the existing rule-based method scored an f1 of 96.76, see the training efficiency of the viterbi model and the supervised model were measured in a windows sub-system for linux environment on an i7-7700k processor with 16gb ram. we ran the machamp model in this environment and it completed in 53,990 seconds. in comparison, the viterbi training completed in 1,805 seconds, which is an improvement of almost 30 times faster than the machamp model. ## conclusion in this study we evaluated different types of models, namely word uni-grams, word n-grams, character n-grams, viterbi decoding, latent dirichlet allocation, support vector machine and logistic regression, for the task of semi-supervised language identification in english-spanish codeswitched data. we found that most of the models achieved promising results, however, the viterbi model performed the best with a weighted f1 score of 95.76% on validation data and 92.23% on test data (rq1). using this model, one can potentially train cs-detection for many more language pairs as previously possible. furthermore, since the majority voting did not lead to improvements, we experimented with an oracle model, which showed that by combining results form our models, the best score we could achieve is 98.47% on validation data. even though the results were good, our models still underperformed compared to the supervised machamp model, that scored 99.24% weighted f1 score on validation data and 98.43% on test data (rq2). there is also a clear take away that, by using simpler, faster approaches like ours and when top performance is not crucial, one can avoid the extensive process of human-annotation and long training time that are needed by finetuning these large transformer models on supervised data.