Code-Switched Language Identification is Harder Than You Think (2024)

Abstract: Code switching (CS) is a very common phenomenon in written and spoken communication, but is handled poorly by many NLP applications. Looking to the application of building CS corpora, we explore CS language identification for corpus building. We make the task more realistic by scaling it to more languages and considering models with simpler architectures for faster inference. We also reformulate the task as a sentence-level multi-label tagging problem to make it more tractable. Having defined the task, we investigate three reasonable architectures for this task and define metrics which better reflect desired performance. We present empirical evidence that no current approach is adequate, and finally provide recommendations for future work in this area.

## introduction code switching (cs), or the use of one or more languages within the same utterance @xcite , is a very common phenomenon in written and spoken communication @xcite . however, many natural language processing (nlp) applications currently struggle to deal with it effectively @xcite ). an obvious first step in building better systems for cs is gathering the data necessary for training effective models, something which is currently lacking for cs text @xcite . a fundamental part of this process is identifying cs in the first place. in this paper, we look at cs language identification (lid) for text and the challenges in getting cs lid systems to work at scale. previous shared tasks on cs lid have produced systems which achieve impressive results @xcite , albeit limited to two languages. we seek to extend cs lid systems to work in a realistic setting as part of a corpus building pipeline by scaling up both the number of languages covered and the speed of inference. our intended use case is mining web text to build cs corpora which can then be used as training data in applications aimed at handling cs. we therefore reformulate cs lid as a multilabel task where the aim is to assign a set of language labels to each sentence, rather than a wordlevel or document-level tagging task as in previous work (section 3). we experiment with highcoverage lid systems (200+ languages) which are simple enough to scale easily, and investigate three different models as reasonable baseline approaches to the task (section 4). we test on wide range of cs and single-label lid test sets aiming to cover as many languages as possible (section 5), and we choose metrics that better reflect true performance in our multi-label setting than those commonly used for single-label lid (section 6). we find that even the best-performing models are still inadequate for identifying cs text at scale (section 7), due to the inherent difficulty of defining cs and detecting the intended language(s) in realistic settings. finally, we make recommendations for future work in this area based on our findings (section 8). to aid future research, we provide code to obtain and transform training and test data, to train all models, and to calculate evaluation metrics. foot_0 2 previous work lid has been an active topic of research for a long time in nlp @xcite . much of the most recent research on this topic has been towards covering more and more languages, with some models claiming to cover over a thousand @xcite @xcite @xcite . however, nearly all general-purpose lid systems assume that text is entirely monolingual (e.g. @xcite or occasionally that any different languages present occur in discrete chunks (e.g. @xcite . this leads to pipelines where cs text is ignored or discarded. previous work on multiple-label lid specifically can be split into two main sub-tasks: multilingual lid, where the expected input is a document containing discrete monolingual chunks in different languages; and cs lid, where the expected input is a sentence or short text containing cs text. the former task has a longer history and its intended application is to segment web text @xcite @xcite . the latter task has received more attention recently including several shared tasks on cs lid, where the aim was wordlevel tagging of cs text given a known pair of languages @xcite . however, both tasks have a limited application to web-scale text because they assume that the input is only in a small number of known languages and tend to reply on computationally-expensive, highcapacity models like transformers @xcite or large language models (llms) for classification. we argue that these are not realistic for filtering web crawls since inference is too slow and expensive. finally, we note that despite the wide range of approaches towards monolingual lid @xcite , lid algorithms are still found to perform poorly in practice compared to test performance, particularly for low resource languages @xcite . this shows that even the simpler task of monolingual high-coverage lid remains a challenging problem. ## task definition we define our task as follows: given a short input text (around sentence length), return a set of codes corresponding to the language(s) it contains. following nllb team et al. (2022), we output modified iso 639-3 language codes encoding both the language variety and the script: for example, eng_latn means english written in latin text. this way of framing the task differs from most previous work on cs lid by assigning tags on the sentence-level, rather than on the word level as in @xcite ; @xcite . less granular labeling like this speeds up inference and so is more practical when using lid to build corpora from web-scale text. in addition, we felt that labelling on the sentence-level avoided some of the ambiguity when labelling at the word level. our model covers many more languages than the previous shared tasks in cs lid (201 rather than just two) so the search space becomes much larger and less tractable at the word level. in addition, the shared tasks included extra tags aside from the two included languages, covering categories such as named entities, 'foreign words', and nonlinguistic content like emojis. we wished to avoid this complication since it was not relevant to our aim of dataset building. ## models we compare the performance of three models for cs lid: openlid, a pre-existing single-label lid model adapted to a multi-label setting @xcite , multilid, a novel lid model, and franc, a high-coverage lid package. 2 the first two models are trained on the same data (openlid) to help isolate the effect of the change in architecture. we employ franc as a comparison point, since it allocates prediction scores in a different way and covers more languages than the other two. in this way, we aim to measure the performance of three reasonable approaches to cs lid, explore their limitations, and so guide further research. ## test sets our aim when choosing test sets was to cover as many cs language pairs as possible, despite the limited number of easily-accessible cs test sets. we were further hampered by the fact that the open-lid training data does not include indian languages written using roman characters which are some of the most common languages to include in cs test sets @xcite @xcite . nonetheless, we source six cs test sets which include eight languages, plus a high-coverage monolingual test set. we describe all test datasets below and include fuller instructions on how to obtain them in appendix a. most of the datasets we use are annotated with language tags at the token level. to fit with our task, we convert these to sentence-level tags by relabelling the sentence as cs if two language labels are present, monolingual if only one is present, and discarding the sentence if it has no language labels (e.g. the sentence only contains named entities or emojis). turkish-english dataset yirmibeşoglu and eryigit (2018) created a cs turkish-english dataset as part of their work on detecting cs for this language pair. the data is sourced from twitter and the ekşi sözlük online forum, then labelled at the token level as either turkish or english. after recombining sentences, the dataset consists of 376 lines of data and 98.9% of the sentences are labelled as cs. indonesian-english dataset @xcite created a cs indonesian-english dataset from twitter data, where each token in each tweet is annotated with a language tag. after pre-processing, the dataset consists of 825 lines of data and 93.5% of the sentences are labelled as cs. basco basque-spanish corpus this corpus contains spanish and basque sentences sourced from a collection of text samples used in training bilingual chatbots @xcite . these sentences were shown to volunteers who were asked to provide a realistic alternative text with the same meaning in euskañol (basque-spanish cs). the created sentences were checked for validity by a team of annotators. we process this corpus into our test set by extracting all spanish, basque, and euskañol utterances present in the final corpus and labelling them using the provided utterance-level language labels. after processing, the dataset consists of 2304 lines of data, of which 59.8% are labelled as cs. ## measuring performance the most common metrics for single-label, multiclass problems are precision and recall (defined in appendix b). however, whilst these metrics give some insight into the functioning of our models, we found them too easy to misinterpret in a multi-label setting. the first reason for this is that precision and recall are undefined when there are no true positive examples of a predicted class in the dataset. this was very common given our high-coverage models, but precision and recall could not detect this key performance issue. secondly, neither precision nor recall account for true negatives, a key indicator for our application of building web corpora since avoiding spurious labels helps prevent noisy datasets. as a consequence of these findings, we decided that precision and recall were not suitable for use as main metrics. instead, we chose three alternative metrics as a better reflection of the desired downstream performance: exact match ratio, hamming loss, and false positive rate (fpr). these metrics allow direct comparison between our different datasets and are easy to interpret correctly even in a multi-label setup with many classes such as ours. we define and discuss each metric below. exact match ratio this metric is simply that for each sentence i in our dataset of length n , we count a correct match if all the predicted labels (ŷ) match the gold labels (y): the higher the metric, the better. the exact match ratio has the advantage of being easy to understand, but it is a strict measure of success and does not reward partial matches. hamming loss we therefore also report hamming loss which allows us to both give credit for partial matches and to penalise predicting too many labels. it can be understood as the fraction of wrong labels among the total number of labels, and the smaller the value of the loss the better. more precisely, let l be the number of classes (languages), y i,l ( ŷi,l ) signify the boolean that the i th example (prediction) is assigned the l th language label, and ⊕ denote exclusive-or: false positive rate finally, we report the macroaverage of false positive rate (fpr) with respect to each language class, or the ratio of number of examples incorrectly identified as a particular language (false positives, f p ) to the total number of ground truth negatives (true negatives plus false positives, t n + f p ). false positive @xmath0, the better. measuring nonrelevant predictions is particularly important given our intended application of building web corpora. this is because the internet mostly consists of non-cs data, so using a classifier with a high fpr on the web will result in a final dataset where most of the content is not relevant @xcite . ## analysis considering the results as a whole, it is clear that none of the models are adequate for the task of detecting the language(s) of cs text. the openlid model is not designed to return multiple labels and so misses many examples of cs sentences, preferring to label them with a single label or not return a label at all. the multilid model has the advantage of being designed to return multiple labels, but the lack of normalisation in the scores means that it is more likely to return spurious labels, as shown in its high fpr and larger number of unique languages in the predictions. franc's algorithm is not suitable for cs text since it assumes a single script and is designed for longer pieces of text. in all cases, the low exact match ratios show that if we were building a corpus from this data, we would miss most of the cs sentences. the performance in general is hampered by one of the inherent problems in cs lid: the boundaries of cs are not defined clearly, even at a linguistic level. in her book on the subject, gardner states that cs "is not an entity which exists out there in the objective world, but a construct which linguists have developed to help them describe their data" @xcite . however, both linguists and language users disagree on what should count as cs, meaning assigning language labels to text can be an ambiguous task in itself. we illustrate our point with two contrasting examples. firstly, this tweet is a fairly straightforward example of a separate english fragment followed by a spanish fragment: @user delete that tweet. . . ya lo hize. this makes it easy (for a human annotator) to assign language labels to it. however, there are many more cases of potential cs which are much more ambiguous and harder to label. the most common of these is a single-word switch in a sentence @xcite , for example: hoy me siento bien senior. . . . these short switches complicate labelling for two main reasons. firstly, there is no clear line between a 'borrowed' word, cs, and a loan word which is now an accepted part of the language (indeed, loan words start out as cs) @xcite . secondly, short fragments of cs can make it difficult to work out which language was intended by the author. this leads to disagreement even amongst expert annotators and consequent 'noisy' labels. we also note that the non-standard orthography of social media and informal text can also hamper n-gram based approaches to lid. ## conclusion we explored the task of scaleable cs lid with the intended use as part of a corpus-building pipeline. we found that three reasonable approaches to the task fell short of the performance required to build useful corpora, demonstrating that the task of realistic cs lid at scale is far from solved. we recommend that future work choose metrics with care to reflect true performance, understand the ambiguity inherent in cs, and fit their definition of cs to the intended task rather than enforce a prescriptive definition of the phenomenon.