Code-Switching Metrics Using Intonation Units (2023)

Abstract: Code-switching (CS) metrics in NLP that are based on word-level units are misaligned with true bilingual CS behavior. Crucially, CS is not equally likely between any two words, but follows syntactic and prosodic rules. We adapt two metrics, multilinguality and CS probability, and apply them to transcribed bilingual speech, for the first time putting forward Intonation Units (IUs) – prosodic speech segments – as basic tokens for NLP tasks. In addition, we calculate these two metrics separately for distinct mixing types: alternating-language multi-word strings and single-word incorporations from one language into another. Results indicate that individual differences according to the two CS metrics are independent. However, there is a shared tendency among bilinguals for multi-word CS to occur across, rather than within, IU boundaries. That is, bilinguals tend to prosodically separate their two languages. This constraint is blurred when metric calculations do not distinguish multi-word and single-word items. These results call for a reconsideration of units of analysis in future development of CS datasets for NLP tasks.

## introduction the mismatch between actual bilingual codeswitching behavior and renditions of codeswitching (cs) has been exposed in previous work. in particular, unnatural switching is characterized by the shortness of language spans and the arbitrariness of language boundaries @xcite . yet to date, metrics of cs complexity have been word-level based (e.g., a 4word utterance with 3 switches, w l1 w l2 w l1 w l2 , @xcite . such wordlevel metrics entail that any word is an equally likely site for language switching. in addition, they treat other-language single-word incorporations and alternating-language multi-word strings identically. neither is a foolproof assumption. in this paper, we adapt natural language processing (nlp) word-based metrics to account for prosodicsyntactic constraints on cs by using intonation units as tokens. we calculate two metrics: overall language distribution (multilinguality) and observed cs rate (probability). results indicate that distinguishing single-and multi-word items in intonation unit-based metrics better captures cs constraints than amalgamating the two mixing types. as a working definition, bilingual cs is going back and forth between two languages within a speaker turn, even within a prosodically and grammatically complete sentence, as in the following example, reproduced verbatim from the new mexico spanish-english bilingual (nmseb) corpus @xcite language is not composed of arbitrary strings of words, but is structured syntactically and prosodically. syntactically, sequential word groups make up constructions and constituents (e.g., @xcite . for example, in the active and passive sentence versions, they allow the new school to wear hats in the halls and the new school's allowed to wear hats in the halls, "the new school" (article + adjective + noun) is a unit that as such is the object in one and the subject in the other (cf. @xcite . syntactic word groups are relevant for cs. according to the equivalence constraint, cs is avoided at points of word order incompatibility @xcite (cf. @xcite @xcite . for example, in the spanish-english language pair, articles are placed before nouns in both languages. however, there is a word order conflict for attributive adjectives, which are positioned before the noun overwhelmingly in english but only sometimes so in spanish. as predicted by the equivalence constraint, cs is far more likely after the article than between the adjective and noun (cf. parafita couto and gullberg, 2019:702). instances like "permiten el new school to wear" ('they allow the new school to wear') are quantitatively preferred (at a 4:1 ratio) over instances like "permiten el nuevo school to wear" @xcite . prosodically, spoken language is broken into intonation units (ius), which are speech segments "uttered under a single, coherent intonation contour" @xcite 53-70; see figure 2 below ). in the nmseb corpus, each line of transcription represents an iu, with punctuation marking the transitional continuity between ius, or the terminal pitch contour. ius are bound to the syntactic organization and processing of language. words in the same iu tend to have a closer syntactic relationship than those in neighboring ius @xcite , and iu boundaries are used in planning utterances @xcite and "tracking speech" @xcite . ius are also relevant for cs. corresponding to the equivalence constraint-which requires local equivalence of word order in the two languages around a switch point-is the iu-boundary @xcite . each line represents an intonation unit (iu). for the purposes of readability, we have removed vocal noises, laughter and vowel lengthening, and excerptinitial pauses. a prosodic sentence is an iu or series of ius, containing at least one finite verb, that ends in intonational completion @xcite . responding to these facts, we apply cs metrics using ius rather than individual words as the basis. ## related methods recent work in nlp is concerned with incorporating, understanding, and generating cs text. although there are already large multilingual language models such as m-bert @xcite and xlm-roberta @xcite , these llms were nevertheless trained on parallel monolingual data, which does not accurately capture the nuance of likely switch points in code-switched language inputs. cs datasets, on the other hand, are scarce and noisy, putting cs modelling in the domain of low-resource tasks @xcite . recently, the lince benchmark compiled by @xcite provides different cs datasets for spanish-english, hindi-english, and other language pairs for the tasks of language identification, part of speech tagging, named entity recognition, and sentiment analysis. often, datasets are collected and scraped from social media (i.e., twitter). however, there are two problems with this collection approach. first, cs is known to happen in community settings among interlocutors with similar cs patterns (cf. @xcite . therefore, mass collecting short tweets does not necessarily provide a true representation of intra-community cs. second, these datasets do not hold a distinction between mixing types. specifically, without this distinction, examples that merely use lone items tend to represent a majority of cs examples. in other words, most cs datasets used in nlp inherently rely on asymmetric models such as the matrix language frame (mlf) model, which assumes a base language for the grammatical frame with words or phrases embedded from the other language @xcite . instead, the equivalence constraint and iu-boundary constraint accommodate alternating-language multiword strings. here, we showcase the different perspectives that the treatment of lone items provides on cs. @xcite developed the multilingual-index (m-index) as a measure of the multilinguality of different corpora, or the distribution of languages in a corpus. @xcite also created the integration-index (i-index), which is meant to measure the probability of cs in different multilingual corpora. @xcite use both of these metrics in the creation and validation of their code-switched twitter dataset for hindi-english and spanish-english pairs. however, it is important to note that the i-index was originally developed with the purpose of analyzing continuous text. therefore, disjointed and small input texts such as tweets might not be a good use case. ## methods we use both m-index and i-index in our experiments with continuous and majority monological transcripts of intra-community speech. to our knowledge, this is the first study to use the intonation unit (iu) as the token of measurement for nlp analysis of cs datasets. with this work, we highlight the different perspectives of cs -separating lone items from multi-word strings -and present the iu as a relevant token for future nlp dataset construction. ## results as shown in we also use language distribution graphs to visualize the m-index and i-index within a transcript, in figure 4 . here, in an ordered array of the language tag for each iu token, english ius are colored in purple, whereas spanish is in yellow. these visualizations help elucidate what the quantitative metrics distinguish, by the number and width of the language bands. for example, transcripts 03 and 10 (language distribution graphs in fig. 4c and 4e , respectively), have a similar m-index and across-iu i-index, which is evident from their language distribution graphs. on the other hand, 05 (fig. 4a ) has a patently different m and i-index from these as to how the inclusion of lone items impacts our perspective of cs, it seems to have little impact on the across-iu i-index, as seen in while for the across-iu index the increase is at most 37.5% (for 16, from 0.08 to 0.11), for the within-iu index the increase is as large as 300% (for 03, from 0.01 to 0.04) (disregarding the cases of infinity, for 05 and 27). this makes sense -lone items occur within a single iu (and are usually nouns positioned at the end or interior of the iu with fewer than 15% occurring iu-initially @xcite ). furthermore, merging the single-word items with multi-word cs not only substantially raises the within-iu i-index, but also magnifies differences between speakers with respect to the within-iu i-index. within-iu i-indices range from 0 to 0.01 with 'no ls,' but from 0.01 to 0.04 when lone items are included. this is important because it shows that with the latter perspective (i.e., merging of lone items with multi-word cs), we fail to capture the iu-boundary constraint on cs whereby speakers strongly prefer switching across prosodic units rather than within them. ## conclusion cs metrics are appropriate when they are grounded in spontaneous speech within bilingual communities, the natural habitat of cs. we have illustrated how using the prosodic structure of cs and distinguishing different mixing types achieves this. we put forward the intonation unit as the token level for cs metrics. in addition, we distinguish multi-word cs from single-word (lone) otherlanguage items. application of the iu-based metrics captures the tendency for multi-word cs to be used from one iu to another in across-iu cs. all speakers (regardless of their m-index or across-iu i-index) strongly disfavor within-iu multi-word cs. this is the iu-boundary constraint of bilingual speech. in contrast, lone items, which by definition are used within a single iu, notably raise the i-index for within-iu cs. merging multi-word cs and lone items thus obscures uniform adherence to the iu-boundary constraint, despite individual differences in language distributions (m-index) and cs rate (i-index). with these findings, we highlight the iu as an important token level for future cs datasets, as well as the impact of single-word (lone) items as distinct from multi-word strings on cs metrics. we see relevance for this work in controlled natural language generation (nlg). much of the cs work is slow/iterative because it is hard to find quality cs data @xcite . thus, our work impacts the future construction of transcribed cs datasets, specifically with the injection of linguistically well-founded cs patterns to improve current nlg methods. currently, cs generation has focused on word substitutions @xcite . we have shown here that bilingual behavior is not limited to word substitution. in accounting for multiword language alternations, future nlp metrics can draw on the equivalence constraint (syntax) and the related but independent iu-boundary constraint (prosody). in sum, this paper is the first to use ius for cs metrics. the results provide critical insight for future transcription and synthetic data generation methods that can improve cs datasets, which will ultimately impact all downstream nlp tasks.