M ask LID : Code-Switching Language Identification through Iterative Masking (2024)

Abstract: We present MaskLID, a simple, yet effective, code-switching (CS) language identification (LID) method. MaskLID does not require any training and is designed to complement current high-performance sentence-level LIDs. Sentence-level LIDs are classifiers trained on monolingual texts to provide single labels, typically using a softmax layer to turn scores into probabilities. However, in cases where a sentence is composed in both L1 and L2 languages, the LID classifier often only returns the dominant label L1. To address this limitation, MaskLID employs a strategy to mask text features associated with L1, allowing the LID to classify the text as L2 in the next round. This method uses the LID itself to identify the features that require masking and does not rely on any external resource. In this work, we explore the use of MaskLID for two open-source LIDs (GlotLID and OpenLID), that are both based on the FastText architecture. Code and demo are available at https://github.com/cisnlp/MaskLID.

## introduction code-switching (cs), the juxtaposition of two or more languages within a single discourse @xcite , is prevalent in both written and spoken communication @xcite . while cs has traditionally been explored as a speech phenomenon @xcite , the increasing prevalence of cs in digital communication, such as sms and social media platforms @xcite , requires the development of techniques to also analyze cs in written texts. there is however a lack of cs data for researchers, making it difficult to study cs and to effectively train cs-aware models. this shortage affects many nlp applications dealing with cs scenarios @xcite . a first step towards the collection of high-quality corpora of cs texts is thus to identify samples of cs in running texts. previous works on cs language identification (lid) have mainly focused on building word-level lids for code-switching between specific pairs of languages, and are often limited to recognize only two languages @xcite @xcite . however, such approaches are not realistic on a larger scale, especially considering that texts on the web typically lack prior information about the languages that are actually being used. more recently, @xcite have investigated the use of high-quality lid at the sentencelevel to detect instances of cs. they propose to reformulate cs lid as a sentence-level task and to associate each segment with a set of language labels. their investigation reveals the difficulty of achieving effective cs lid with existing lid models. furthermore, their findings indicate that such lids predominantly predict only one of the languages occurring in cs sentences. in this work, we continue this line of reserach and introduce masklid, a method that also uses high-quality sentence-level lid to identify cs segments. by masking the presence of the text features associated with the dominant language, masklid improves the ability to recognize additional language(s) as well. we explain in detail how masklid works in cooperation with two existing lids that are based on the fasttext @xcite architecture in section 3. as we discuss, our method can identify arbitrary pairs of languages, and is also able to detect mixtures of more than two languages in the same segment. being based on fasttext, it is also extremely fast. this two properties make masklid well suited to mine large web corpora for examples of real-world cs segments, that can then serve as valuable training data for applications designed to handle cs inputs. we evaluate masklid on two test datasets contain-ing both cs and monolingual data, showing the benefits of using masklid (see section 4). ## experiments and results here, we provide an overview of our baselines and test data. we assess the performance of the baselines by testing them both with and without masklid. our setting of hyperparameters is explained in appendix c.2. ## conclusion we present masklid, a simple, yet effective, method for scalable code-switching (cs) language identification (lid). masklid is designed to complement existing high-performance sentence-level lid models and does not require any training. in our experiments, masklid increases cs lid by a factor of 22 in turkish-english, by 22 in nepali-english, by 6 in hindi-english and by 5 in basque-spanish. in future work, we aim to explore the use of subword-level, instead of word-level features, extending the applicability of the method to languages that do not use spaces for word separation. additionally, we plan to generalize this method to other lid models using techniques like lime @xcite to map features to languages. last, we intend to apply masklid on the web data, in the hope that it will help build larger high-quality web corpora for cs.