T ongue S witcher: Fine-Grained Identification of G erman- E nglish Code-Switching (2023)

Abstract: This paper contributes to German-English code-switching research. We provide the largest corpus of naturally occurring German-English code-switching, where English is included in German text, and two methods for code-switching identification. The first method is rule-based, using wordlists and morphological processing. We use this method to compile a corpus of 25.6M tweets employing German-English code-switching. In our second method, we continue pretraining of a neural language model on this corpus and classify tokens based on embeddings from this language model. Our systems establish SoTA on our new corpus and an existing German-English code-switching benchmark. In particular, we systematically study code-switching for language-ambiguous words which can only be resolved in context, and morphologically mixed words consisting of both English and German morphemes. We distribute both corpora and systems to the research community.

## introduction a large proportion of the world's population is multilingual, and that naturally means that a large proportion of the world is code-switching daily, frequently and routinely @xcite @xcite . code-switching occurs when speakers alternate between languages; this can happen at the sentence, word, or even subword level. for many multilingual speakers, code-switching is a natural part of informal language, either as a matter of convenience or possibly because it allows them to express their intended meanings more precisely. several psycholinguistic and sociolinguistic theories of codeswitching exist @xcite @xcite @xcite . the dominant language is called the matrix language, while the subordinate language that is included is tweet: ich glaub ich muss echt m rewatchen e like i feel so empty was soll ich denn jetzt machen translation: i think i really have to rewatch it like i feel so empty what should i do now figure 1 : german-english code-switching called the embedded language @xcite . we refer to any text segment in the embedded language as an island. in the example of code-switching shown in figure 1 , e like i feel so empty is an island. many nlp systems are currently developed to be capable of handling text from informal contexts. code-switching places new pressure on these, particularly for applications that require the recognition and precise extraction of meaning from codeswitched text, or even the generation of such text. available nlp tools lag behind in this respect @xcite ; in particular large language models perform best when fine-tuned on natural code-switching data @xcite . our work is aimed towards nlp tools that can better understand and manipulate code-switched language. we are interested in studying naturally occurring code-switching. social media, where mostly informal conversation take place, is an ideal source of such texts. we study code-switching between english and german, two closely related languages. we encounter many high-frequency words of identical form in both languages, such as "was" in figure 1 , which is a wh-pronoun in german. if the same string "was" appeared in english, it would be the past form of 'to be'. importantly, the two meanings are entirely unrelated. such cases constitute an interesting corner case for code-switched text, and are called interlingual homographs @xcite . a second interesting phenomenon is that, because german is a morphologically rich language, its morphology can act on english mor-phemes, creating intra-word codeswitching, such as in the past participle m rewatchen in figure 1 . a third property of the german-english language pair is the high frequency of english loanwords in german. loanwords are words of foreign origin that have been fully assimilated into the main language. loanwords and code-switching constitute a grey area in language change: whether something is a loanword or part of an island is a hotly debated topic in linguistics @xcite . while we do not directly address the loanword distinction in this work, we believe that our theory-neutral methods can contribute to an empirical way of addressing this in the future. in this paper, we study german-english codeswitching with an analytic interest in fine-grained phenomena (e.g. short islands, mixed morphology and interlingual homographs). this introduces new aspects to the automatic study of code-switching. at the same time, we bring scale to the analysis; the tongueswitcher corpus includes 25.6m german-english code-switching tweets with automatic code-switching identification. we release our corpus and the two code-switching identification methods we developed, one rule-based, one neural foot_0 . ## related work code-switching identification and language identification are closely related tasks, but traditional language identification (li) tools can only determine which languages are present in a given text, not the precise beginning and end of each island. for instance, the li tools provided by @xcite and @xcite rely on character-based n-gram models. fasttext @xcite uses a character-based ngram method to compare statistical properties of the input text with a pre-compiled frequency profile of each language. it distinguishes 176 languages, including english and german, alongside similar languages such as luxembourgish and afrikaans. polyglot is another such tool, which is able to identify more than one single language per document @xcite . it is built from the cld2 tool from @xcite , which uses quadgram ranking. lingua (stahl, 2023) is a blackbox li tool that also offers code-switching identification for many language pairs, including german-english. it combines a language modelling approach with hard-coded rules. its code-switching identification performance has never been experimentally evaluated. @xcite present rulebased code-switching identification systems for vietnamese-english and hindi-english mixed text, which is based on specially-created wordlists for each of these language pairs. all words that appear in both wordlists are manually disambiguated by a human annotator. this is a simple approach to the task that affords the researchers control over their system, as it does not require any training. @xcite detect codeswitching at a finer-grained level. in their denglisch system, tagging proceeds at token-level, and the following labels are used: d and e for german and english tokens respectively; sd, se and so for loanwords imported from german into english, from english imported german, and from other languages. there is also an other category for unclassifiable items, such as punctuation and emojis, and a mixed category for words of mixed morphology. several other code-switching approaches also model mixed morphology. @xcite perform morphological analysis with the morphessor tool to address dutch-limburgish-english code-switching, and @xcite detect intra-word code-switches in german-turkish and spanish-wixarika text using rnns. osmelak and wintner (2023) use crfs, a supervised machine learning framework, in combination with manually curated features, such as orthography, n-gram, morphology, function words, frequency, lexical components and wordlists. the training material consists of 950 reddit comments containing 60k tokens, balanced between english and german. they also use automatically-tagged silverstandard data to the tune of a further 31,500 comments (5 million tokens). in contrast, our solution does not require any human-annotated training material. for neural code-switching identification, the use of word embeddings from a multilingual language model such as mbert @xcite is one possible approach. mbert is an encoder-only transformer-based model which embeds each token into a 768-d vector. @xcite found it is best suited when fine-tuned on naturally occurring code-switching material. @xcite pretrain and fintune a bert-based model for codeswitching identification in hinglish. they use an existing tool to collect and automatically label a large corpus of tweets. when it comes to gold-standard datasets, the majority of code-switching datasets are between (1) languages spoken in india with english @xcite @xcite , (2) mandarin with english @xcite and (3) spanish with english @xcite . for german-english code-switching other than @xcite , @xcite use a mini-corpus consisting of 99 twitter tweets, which is not publically available. our corpus is much larger. ## corpus construction the german tweets we use as input were collected at scale by @xcite . the twitter language identification algorithm assigns a probable language at the time of writing of each tweet; kratzke chose those that were deemed german. this resulted in 149.2m input tweets written between april 2019 to february 2023. we clean the tweets (urls are replaced @xmath0, emojis, emails, phone numbers and mentions are removed) and run fasttext language detection on them, only keeping tweets that are re-assigned the german tag or instead assigned an english language tag. this step eliminates many tweets in luxembourgish and other languages too similar to german for twitter's language identifier to catch. 123.7m tweets remain after this step. in contrast, @xcite filter their input to remove those examples for which polyglot's prediction is not both english and german. to establish a testset, code-switching annotation was performed by the authors of this paper on 1252 tweets. we used the prodigy annotation tool @xcite . to use annotation time efficiently, we wanted to make sure that a good proportion of the cases seen had reasonably high code-switching occurrences. the tweets were therefore processed and pre-filtered by a precursor of tongueswitcher. this system differed from the final version only marginally, e.g. in the ordering of the rules and the quality of the multilingual stemming algorithm. we then random-sampled from two subsets: all input tweets (25%), and those with a high proportion of code-switching (75%). system annotations were not removed before hu-man annotation. there were no explicit guidelines. annotators discarded tweets in german dialects such as swiss german, made sure that german indeed acted as the matrix language, and then marked island start and end points in each surviving tweet. in 63.5% of cases, boundaries were moved; this means that annotators did not simply accept the system's suggestions. regarding the annotation of loanwords, each annotator followed their own intuition about which words were so common as to be used as loanwords, additionally using a contextsensitive definition of loanwords. to establish consistency of the annotation, we randomly sampled 36 tweets consisting of 1172 tokens, which both annotators labelled. inter-annotator agreement was measured at κ=0.68 (@xmath1, @xmath2, @xmath3; @xcite . the annotators agreed fully on all tokens in 15 out of the 36 tweets. the distribution of island sizes in the resulting testset is given in figure 2 . ## tongueswitcher our rule-based method takes as input german tweets and produces labels indicating the language of each word, or sub-word if the word is of mixed morphology. the main algorithm applies several wordlist-based filters to make the decision. additional processing applies if a) the word is genuinely a possible word in both languages or b) it is an intraword code switch. all processing in this method is performed on lowercased words. ## bert-based system we also wanted a neural system that is fine-tuned for german-english identification, so we could investigate to which degree neural word embeddings are suited to the task. to that end, we pretrain a neural language model on the tongueswitcher corpus and fine-tune it for token classification. we then learn the classification layer using the humanlabelled examples from the denglisch corpus. this system is called tsbert. ## experiment systems, competitors, baselines we evaluate our systems, tongueswitcher (ts) and tsbert, against two competitors from the literature, denglisch crf @xcite and lingua @xcite . the denglisch system is not provided as a trained system, so we follow their procedure in training it. to interpret denglisch's output, we match denglisch labels onto our reduced set as follows: english, german, mixed are taken directly. se becomes english, and sd becomes german. denglisch's so labels and punctuation labels are ignored in evaluation. we construct a strong baseline by prompting the gpt-4 llm (openai, 2023) with the prompt given in appendix §a.3. we also train baseline neural classification models by learning the classification layer directly on (english) bert (ebert), german bert (gbert, deepsetai, 2019) and multilingual bert (mbert) models. ## conclusion we have presented two methods for german-english code-switching identification. our rulebased system enabled us to collect the largest corpus of naturally occurring code-switching. our bert-based model, trained on this corpus and finetuned on human-annotated data, established sota on an existing german-english benchmark. we also established sota on our newly formed corpus using token and entity-based metrics. a postanalysis on interlingual homographs revealed that neural language models are the best systems for disambiguating these words. overall, our study combines two aspects we think are important for the future of code-switching: a) the use of largescale empirical methods on naturally occurring data and b) an analytic interest in fine-grained linguistic phenomena.