Surprisal Predicts Code-Switching in C hinese- E nglish Bilingual Text (2020)

Abstract: Why do bilinguals switch languages within a sentence? The present observational study asks whether word surprisal and word entropy predict code-switching in bilingual written conversation. We describe and model a new dataset of Chinese-English text with 1476 clean code-switched sentences, translated back into Chinese. The model includes known control variables together with word surprisal and word entropy. We found that word surprisal, but not entropy, is a significant predictor that explains code-switching above and beyond other well-known predictors. We also found sentence length to be a significant predictor, which has been related to sentence complexity. We propose high cognitive effort as a reason for code-switching, as it leaves fewer resources for inhibition of the alternative language. We also corroborate previous findings, but this time using a computational model of surprisal, a new language pair, and doing so for written language.

## introduction code-switching (cs) occurs when a speaker alternates from one language to another during linguistic communication (e.g., @xcite . for example, in: "洗衣房在basement 。" ("the laundry room is in the basement."), the speaker alternates from chinese to english by introducing the word "basement", replacing the chinese word "地下室". this behavior is very common among bilinguals. many factors have been shown to affect the propensity of a bilingual to code-switch. among others, there are variables related to the participants in the conversation (e.g., @xcite , the ease of production of the relevant words (e.g., @xcite , the linguistic context (e.g., @xcite , memory limitations of the speaker @xcite , cognitive load and emotional state of the speaker (e.g., @xcite , and the type of information to be conveyed (e.g., @xcite . among the latter, predictability, as measured by word completion, has been correlated with code-switching @xcite . in this paper, we model predictability using word surprisal calculated with a language model. we ask whether word surprisal @xcite and word entropy @xcite affect the probability of cs within a sentence (intra-sentential cs), while controlling for other known psycholinguistic factors. word surprisal measures how unpredictable a word is in its context, typically operationalized as the negative log-probability of a word w i conditioned on a window of t previous words: surp(w i ) = -logp (w i |w i-1 , ..., w i-t ) (1) word entropy before w i measures the uncertainty when w i is still unknown, operationalized as the expectation over the vocabulary of word surprisal: thus, given a context, word surprisal measures how unpredictable a specific word is, while word entropy measures how unpredictable all words are in average. these variables have been related to a very wide range of psycholinguistic phenomena (e.g., @xcite @xcite @xcite @xcite @xcite . we collected a corpus of chinese-english text from online forum conversations where the majority of sentences are in chinese but some sentences contain segments in english. these code-switched sentences were translated into chinese and compared to sentences with similar syntactic structure but without any code-switch, in order to see what factors affected the propensity of cs. with this paper, we make a curated version of this dataset publicly available, together with the code that was used for its extraction and processing. 1 then, we fitted a logistic regression model to predict cs in a sentence, testing whether the addition of surprisal and entropy improves a model that only contains control factors. the results show that word surprisal improves the quality of the model. since surprisal has been related to cognitive effort of language production (e.g., @xcite and comprehension @xcite , we can relate cs to states in which the speaker faces difficulties, and/or, similar to @xcite , as a strategy to signal highly informative content. conversely, we found no evidence of word entropy improving the model. furthermore, word entropy does not reach significance even when used as the only predictor. while further testing is needed, we attribute this result to the fact that during language production, speakers are completely aware of the semantics they try to convey, radically reducing the number of possible word continuations, thus reducing the effort of selecting a word among multiple possibilities. in the case of surprisal, we interpret the effect observed here as the facilitation that the previous words could have on the production of the next word, irrespective of the semantics' effect. the rest of this document is organized as follows: section 2 presents a selection of factors that have been known to affect cs. section 3 explains the method that we used to obtain the chinese-english corpus and analyze it. section 4 shows the results of the analysis. finally, sections 5 and 6 present the discussion and conclusion respectively. ## factors that predict cs we can arrange some of the factors that have been shown to affect cs according to their source: sociocultural: cs can be used to construct identity and modulate social distance and affiliation @xcite . moreover, cs can be affected by the kind of participants in a conversation. for example, @xcite observed that norwegian locals tended to switch from a dialect form to a standard form of norwegian as soon as they felt the presence of non-locals. cs can also be affected by the type of content that is conveyed. for example, speakers can use cs to try to distance themselves while talking about embarrassing @xcite or emotional @xcite topics. linguistic: cs seems to obey certain linguistic rules. e.g., at the morphological level, cs has been proposed to occur only if the switched morpheme is not bound @xcite , and if it does not violate any syntactic rule of the languages involved @xcite . speaker-related: factors related to the difficulties that speakers encounter during language production. indeed, one view of cs is that it occurs to compensate for a lack of language proficiency @xcite . independent of the proficiency level, some words are inherently more difficult to access, in which case a speaker might choose to produce a word in a different language if it is more accessible (e.g., @xcite , in line with the idea of an integrated representation of a bilingual's linguistic knowledge @xcite . for instance, words with higher frequency and shorter length are more accessible (d' @xcite . moreover, words referring to concrete and highly imageable concepts are suggested to be more integrated in the bilingual lexicon than abstract words, predicting a greater probability of cs @xcite . similarly, nouns are suggested to be stored in a common semantic system shared across languages, while other words are stored in language-specific areas, since the latter elicit slower and less consistent associations across languages @xcite . thus, nouns are the class of words that is most frequently code-switched (e.g., myers-scotton, 1993) and borrowed @xcite , followed by verbs and other parts of speech. following dependency locality theory @xcite , @xcite shows that the more intervening words between a potentially codeswitched word and its dependency governor, the more difficult it is for the speaker to track the language of the governor, due to memory limitations, and therefore the more likely to code-switch. comprehender-related: cs has also been proposed as a strategy to facilitate comprehension by marking portions of discourse @xcite @xcite . from this view, the act of cs carries information, similar to how prosody helps comprehenders to recognize the focus of an utterance. thus, cs has been reported to be used to increase the salience of discourse markers @xcite , to signal new discourse topics @xcite , to contrast topic and focus elements @xcite , and to mark important discourse information @xcite . similar to @xcite , @xcite show evidence suggesting that speakers code-switch to mark important information in czech-english bilingual speech, where importance is measured by the amount of semantic information that they convey. then, more informative meanings receive more distinct encodings, reducing the risk of miscommunication. ## methods our initial hypothesis was that bilinguals are more prone to cs when the words to be produced have high surprisal; and at states of high uncertainty, as measured by word entropy. in order to test this, we used binary logistic regression to assess the effect of surprisal and entropy for predicting cs, while controlling for other well-known factors. first, we collected a corpus of bilingual chinese-english text that contains code-switched sentences. then, the code-switched sentences were translated into chinese, obtaining fully chinese sentences. the correctness and fluency of these translations were verified using a survey in amazon mechanical turk. we will refer to each of the translated versions of the code-switched sentences as a cssent; note that these are the sentences that are used for the analysis. afterwards, for each cs-sent, we selected a sentence from those that did not originally contain a code-switch and that had a similar syntactic structure to the corresponding cs-sent, as described in the alignment section below. we use noncs-sent to refer to these sentences. finally we trained a binary logistic regression model to predict whether a sentence was a cs-sent or a noncs-sent. to investigate the effect of surprisal and entropy more directly, the logistic regression model used several control factors reflecting some of the findings in section 2. then, a genetic algorithm was used to select, among all the possible models that can be obtained by combining all control factors and their two-way interactions, the model that would minimize the akaike information criterion (aic) and the bayesian information criterion (bic), which are measures of model quality that are based on the log likelihood of the model, the number of parameters of the model, and the size of the dataset. this selected model is the control model of our experiments. finally, the control model was compared to models that included word surprisal and entropy, respectively, to examine their relative contributions in explaining cs beyond the known correlates. the next subsections explain more in detail the procedure that we followed to collect the corpus, obtain the measurements related to the control factors, and train the final logistic regression model. ## discussion in this study we explore the effect of word surprisal and entropy on cs. the computation of these measures relies on language models trained with sufficient and appropriate data, which is non-trivial in the case of bilingual text. moreover, even assuming a large bilingual corpus, utterances tend to appear in segments of the same language, so any code-switch is likely to cause an increase of surprisal on the word where the switch occurs. for example, in: "洗衣房在basement 。", we expect "basement" to have high surprisal at least partly because an english word does not tend to follow a sequence of chinese words. this makes it difficult to assess whether the increase of surprisal causes the code-switch or vice versa. conversely, by using a translated version, if there is an increase of surprisal, it would not be because of the code-switch, but possibly because the concept or chinese word is infrequent. considering these aspects, we used a monolin-gual chinese language model to see whether word surprisal at the cs-point can predict cs assuming that the switch never occurred, as previously described. while we performed several steps to verify the quality of the translations, it is possible that the translation process could introduce uncommon constructions, increasing the surprisal values. however, in more than half of the codeswitched sentences, the switch corresponded to a single word, likely resulting in few translation alternatives. moreover, the verification survey showed that the translations are fluent and semantically correct. consequently, we expect the sentences that we used to be appropriate for our study. using this new chinese-english cs dataset, we tested several factors that have been shown to affect cs. we found that long and infrequent words are more likely to be code-switched, which is compatible with previous findings suggesting that words with these characteristics are less accessible and more likely to be code-switched. another important predictor was sentence length, where longer sentences are more likely to be codeswitched. this could reflect the effort related to produce longer sentences, as they require more structures to be retrieved and handled. critically, word surprisal was also a relevant factor for predicting cs. since surprisal has been related to cognitive effort in language comprehension @xcite and to some extent in language production (e.g., @xcite , we may interpret word surprisal as the degree to which the words that were previously produced facilitate production. from this view, word surprisal would index context-dependent accessibility, in contrast to word frequency, which would index context-free word accessibiity. then, words with high surprisal would be less accessible, similar to infrequent words. unexpectedly, word entropy did not seem to predict cs. if word entropy indexes the effort of choosing among multiple possible word continuations, then our calculation did not reflect the true probability distribution, as the number of candidate words would be drastically reduced by the semantics the speaker tries to convey. something similar would happen with word surprisal, in that case, we interpret the results observed here as the facilitation that the previous words have on the production of the next word, beyond the effect that the semantics could have. we expect that entropy calculations conditioned on semantics could give different re-sults. the current model of bilingual representations suggest that bilinguals actively inhibit the alternative language when speaking in a second language @xcite . nonetheless, bilinguals often code-switch, suggesting that inhibition might depend on the context, available resources, and even audience design. for example, when a bilingual is with other bilinguals, he/she might feel more free to code-switch using the most accessible words, knowing that the audience would understand both languages (blanco-elorrieta and pylkkänen, 2018); while in monolingual situations, the bilingual would use the appropriate language, inhibiting the alternative one (e.g., @xcite . alternatively, if the bilingual is under high cognitive load, the available resources for inhibition would be less, reducing inhibition and increasing the probability of cs (e.g., @xcite . most predictors in our model can be related to production effort: long and infrequent words are less accessible and therefore harder to produce; longer sentences require more structures to be handled; assuming word surprisal reflects context-dependent accessibility, high-surprisal words would also be harder to produce. in this context, we propose cs as a result of high cognitive effort, leaving less resources for inhibition, and thus increasing the probability of cs. another explanation for cs, within audience design, is one of strategy, where cs can highlight segments with high information density in order to emphasize their content. this might increase the probability of successful comprehension. to study that possibility, @xcite used predictability of meaning (pom). this notion relates to word surprisal perhaps in a similar way to how word surprisal relates to word frequency: they both encode how unlikely a given word is, correlating to some degree, however they are not completely redundant. pom is calculated by asking comprehenders to guess possible word continuations -in either language -given a context and within a limited number of guesses. pom corresponds to the accuracy of that guess. this is where our study contributes important data using the automatic, neutral, and well-studied surprisal metric. word surprisal is calculated using a language model, permitting more nuanced online estimations. in terms of their interpretation, pom is, as its name suggests, more related to semantics, as it is language-independent and elicited offline (without time pressure). by contrast, word surprisal is language-specific and encodes both how likely a meaning and a specific word are. thus, an example in which these metrics would differ is when a meaning is predictable, but the related word in a specific language is not, giving a high pom but also a high word surprisal. further modeling is needed to disentangle these explanations (cognitive effort, audience design). it is possible that if a speaker has difficulties during production, cs would be more likely to occur; and at the same time, if the speaker believes cs would help to communicate his/her message (as proposed by @xcite , then he/she might choose to code-switch. as previous findings show, cs is a phenomenon that can be affected simultaneously by a wide variety of factors. ## conclusion we investigated the effect of word surprisal and word entropy on the probability of code-switching (cs) in chinese-english written communication. a corpus of text containing chinese-english conversations was collected and its code-switched phrases were translated back into their context language. the translations of the code-switched sentences were compared to sentences with similar syntactic structure but without code-switches, in order to see what factors affected the propensity to cs. surprisal predicts cs. since surprisal has been associated with cognitive effort during language production (e.g., @xcite and comprehension @xcite , we can relate cs to situations in which the speaker faces difficulties; and/or similar to @xcite , situations where the speaker uses cs as a strategy to emphasize highly informative content to the comprehender in order to facilitate communication. we found no evidence showing that entropy, as opposed to surprisal, predicts cs. this may be due to the formulation of entropy that we used, which does not consider the semantics the speaker tries to convey. this paper makes two specific contributions. the first one is the finding that cs in written language is reliably affected by sentence length, word length, arguably word frequency, and, most importantly word surprisal. the second contribution is a new chinese-english cs dataset, which includes translations to the dominant language, which we hope will be used in further models of cs.