Measuring Entrainment in Spontaneous Code-switched Speech (2024)

Abstract: It is well-known that speakers who entrain to one another have more successful conversations than those who do not. Previous research has shown that interlocutors entrain on linguistic features in both written and spoken \emph{monolingual} domains. More recent work on \emph{code-switched} communication has also shown preliminary evidence of entrainment on certain aspects of code-switching (CSW). However, such studies of entrainment in code-switched domains have been extremely few and restricted to human-machine textual interactions. Our work studies code-switched spontaneous speech between humans, finding that (1) patterns of written and spoken entrainment in monolingual settings largely generalize to code-switched settings, and (2) some patterns of entrainment on code-switching in dialogue agent-generated text generalize to spontaneous code-switched speech. Our findings give rise to important implications for the potentially “universal” nature of entrainment as a communication phenomenon, and potential applications in inclusive and interactive speech technology.

## introduction when people speak with one another, they often subconsciously adapt aspects of their communication style to that of their conversational partner. interlocutors who do so perform what is known as entrainment -also called accommodation, alignment, or coordination. this has been shown to produce more successful conversations than conversations without entrainment @xcite . in written and spoken domains of communication, people entrain on multiple dimensions of language production: diction and syntax @xcite , speaking rate, voice quality, and pause frequency @xcite @xcite , jokes and laughter @xcite , and facial expression and gesture figure 1 : the novelty of our work comes from incorporating acoustic-prosodic features in the study of entrainment in code-switched speech. here we highlight the value of identifying multiple dimensions and feature sets of entrainment in csw. while entrainment is evident by inspection of the lexical and csw strategy features of the s a1 , s b1 interaction (top) (@xmath0; @xmath1, the acoustic-prosodic features are essential for correctly identifying entrainment in the s a2 , s b2 interaction (bottom) (@xmath2; @xmath3; purple, @xmath4. @xcite @xcite , among others. however, most of these findings come from studies on monolingual communication, which is much simpler to analyze than multilingual alternatives since there is no need to identify which language is being spoken for each word or phrase -a challenging task applicable to most multilingual corpora. this evidence of monolingual entrainment in prior research suggests that, when multilingual speakers converse, they may also entrain to one another in aspects of multilingual language production such as code-switching (csw), which occurs when a speaker alternates between one language and another in written or spoken communication @xcite . there has been relatively little work on investigating entrainment in code-switched domains, particularly in speech production, leaving open the question of how entrainment occurs in such multilingual settings. answers to this ques-tion would be useful for understanding the conversations of the majority of speakers in the world who speak more than one language @xcite and could inform future innovation in inclusive and interactive speech technology. we aim to fill this gap by studying how entrainment manifests itself in naturalistic code-switched speech between humans. we first focus on spanish-english csw, which is common in hispanic communities in the united states, by examining the lexical, acoustic-prosodic, and multilingual characteristics of the bangor miami code-switched conversations @xcite (figure 1 ). our main contributions from doing so are (1) adapting metrics of entrainment previously used in monolingual settings to multilingual ones. we study lexical and acoustic-prosodic entrainment in a code-switched setting and find evidence that entrainment occurs in multilingual domains just as it does in monolingual ones. (2) we identify csw patterns of entrainment in spontaneous human-human speech to inform the design of interactive speech technology capable of both understanding and appropriately responding to code-switched speech in a human-like way. entrainment for csw. recent work on entrainment has begun to explore code-switched domains, but most studies have been restricted to the textual modality. @xcite found that dutch-english speakers entrain on csw more when their interlocutor has just code-switched, and that they use shared word order in order to code-switch then, focusing entirely on lexical entrainment. @xcite examined only convergence of the amount of csw that occurs between two spanish-english speakers over the course of a conversation, analyzing speech transcripts. similarly, @xcite and @xcite studied entrainment of csw strategies in human-computer communication in spanish-english and hindi-english respectively, distinguishing between relatively simple insertional (inserting a single code-switched word or short phrase into an otherwise monolingual utterance, e.g. todos estamos con un calor y work-ing.) and more complex alternational (csw occurring at grammatical clause boundaries, e.g. no tienes que pagar mucho porque you don't have the check.) code-switches. given the nature of human-machine dialogue systems, this work was also restricted to analysis of written interactions. the findings of these studies beg the question: how does entrainment manifest in naturalistic code-switched speech between humans, leading to our second research question: rq2: do pat-terns of entrainment in code-switched text (some of which is produced by virtual dialogue agents) generalize to code-switched spontaneous speech? do code-switched language behaviors depend on the modality of language production and/or the nature of the interlocutor, and are there similarly generalizable characteristics of csw that are independent of these conversational factors? ## corpus we examine the bangor miami corpus of informal conversations @xcite . 2 this corpus is composed of a mix of monolingual and codeswitched english and spanish spontaneous speech, along with transcripts of these, as produced by 84 unique bilingual spanish-english speakers living in miami, florida. the corpus contains 35 hours of recorded conversation and 242,475 words of transcribed text. the transcribed text comes manually annotated with word-level language identification labels for each utterance in each conversation. the complete corpus consists of 56 conversations, with the number of speakers per conversation ranging between one and four. of these, 39 are dyadic conversations (i.e. conversations that involve exactly two speakers) that contain code-switched utterances. we restrict our investigation to the 20 hours of these conversations only. ## method preprocessing: data annotation. we begin by annotating the bangor miami corpus to identify the different csw strategies used by speakers in the filtered corpus of 39 dyadic conversations. we accomplish this by inspection of the conversation transcripts. we first automatically label each utterance of each conversation's transcript for whether it is code-switched or monolingual, based on its word-level language tags. two annotators fluent in spanish and english then perform additional manual annotations on the code-switched utterances to distinguish between insertional (i), alternational (a), and "other" (o) forms of csw. 3 the annotators define "other" csw as the strategy used in any utterance where a code-switched filler word such as okay or pues appears at the outset or end of a sentence whose remainder is monolingual in the op-posite language. we find that 95% of utterances in the filtered corpus are monolingual. among the 5% of utterances in the corpus that are code-switched, 72% of these use insertional csw, 13% use alternational csw, and 18% use "other" csw (some code-switched utterances use more than one csw strategy). preprocessing: denoising audio data. next, we handle the babble noise (i.e. the kind of background noise often heard in restaurants, bars, or airports) present in most of the corpus' audio files, to ensure reliable downstream feature extraction and analysis. to achieve this, we create a denoising pipeline. this is a two-step process that consists of first applying to each audio file a conformerbased metric generative adversarial network @xcite , which enhances the magnitude and spectrogram features of a speech signal, followed by using audacity's publicly-available noise reduction tools. this approach effectively reduces background noise while preserving the integrity of the speech signal, foot_3 as evidenced by our calculated speech-to-noise (snr) ratio in each denoised audio file exceeding 30 db, the threshold for clean speech signals (how is the snr of a speech example defined?). the mean, mode, and median snr in the corpus post-denoising are 54.3db, 74.7db, and 56.3db respectively. these are significant improvements over the pre-denoising mean, mode, and median snr of 29.3db, 35.6db, and 29.3db respectively. we share this newly cleaned version of the corpus with the spoken language processing community. measuring entrainment across feature sets. following the preprocessing of the corpus, we calculate several measures of entrainment across lexical, acoustic-prosodic, and csw feature sets. the lexical feature set includes the following word classes, as in @xcite : most frequent words within the corpus (top-100 and top-25), most frequent words within each conversation (top-25), affirmative cue words (e.g. alright, gotcha, okay, uh-huh, yeah, and their respective spanish equivalents), and filled pauses (e.g. uh, um, mm, and their respective spanish equivalents). see appendix a for the full list of affirmative cue words and filled pauses considered. note that both spanish and english words are members of each of the above word classes. our experiments on these lexical features of the corpus use equation 1 to calculate entrainment in each of the word classes w under investigation. this equation was first proposed in @xcite . it defines entrainment between two speakers s a and s b on a particular word class w as the negated absolute value of the difference between the fraction of times a particular word @xmath5, summed over all the words in w . we also compute lexical entrainment on overall language use in the corpus, adopting the method used by @xcite . we use the speech transcripts of each speaker as training data for estimating a kneser-essen-ney smoothed trigram model, using the kenlm toolkit for this task, and resulting in 78 such trigram models -two for each conversation in the corpus. for each conversation, we evaluate the trained model for s a by computing its perplexity on the speech transcripts of s b . the negated perplexity value produces an entrainment score reflecting how much s a entrains to s b . we do the same perplexity calculations in the opposite direction to obtain an entrainment score for how much s b entrains to s a . the lower the perplexity, the greater the entrainment. the members of the acoustic-prosodic feature set for each utterance are: minimum, mean, maximum, and standard deviation in pitch; minimum, mean, maximum, and standard deviation in intensity; jitter; shimmer; harmonics-tonoise ratio (hnr); and speaking rate, measured in syllables per second. we extract these acousticprosodic features automatically using the parselmouth python library @xcite for the praat software @xcite , with all parameters set to their default values. we define the set of csw features such that, for each utterance, we examine the binary presence of csw (coded as 0 for monolingual utterances or 1 for code-switched ones), the amount of csw (normalized by the word length of the utterance; this produces a csw ratio), and the strategy of csw used (i, a, or o for code-switched utterances; -1 for monolingual utterances). for our experiments on the acoustic-prosodic and csw feature sets, we calculate entrainment in terms of proximity (the absolute similarity of a feature over an entire turn/conversation), convergence (the degree to which a feature becomes more similar over the course of a conversation), and synchrony (turn-byturn relative coordination between interlocutors, as when speakers have different pitch ranges but raise and lower their pitch similarly to their partner) at the turn-and conversation-level, following the methods used in @xcite . for turn-level proximity, we perform calculations in the following way. for each target speaker turn, we compute a partner difference (equation 2 ) and other difference (equation 3 ), such that turn p is adjacent to the target turn and uttered by the target turn speaker's conversational partner, and turn i is uttered by the target turn speaker's conversational partner but is not adjacent to the target turn, for ten random turns. we compare partner differences and other differences with a paired t-test. we infer proximity when partner differences from the prior speaker turn are smaller than differences from other speaker turns. for turn-level convergence, we compute the pearson's correlation coefficient between the absolute value of the difference between adjacent turns and the turn number, which effectively measures time elapsed in the conversation. we infer strong, moderate, or weak convergence from pearson's coefficients @xmath6.7, 0.5 ≤ @xmath7.7, and 0 < @xmath8.5, respectively. we infer strong, moderate, and weak divergence from pearson's coefficients @xmath9.7, -0.7 < @xmath10.5, and -0.5 < @xmath11, respectively. for turn-level synchrony, we compute the pearson's correlation coefficient between adjacent turns from different speakers, testing for significance with a two-sided t-test. we infer strong, moderate, and weak synchrony from pearson's coefficients @xmath12.7, 0.5 ≤ @xmath13.7, and 0 < @xmath14.5, respectively. we infer strong, moderate, and weak asynchrony from pearson's coefficients @xmath15.7, -0.7 < @xmath16.5, and -0.5 < @xmath17, respectively. for conversation-level proximity, we use paired t-tests on two sets of differences. for each speaker, we calculate a partner difference (the difference between the speaker's value for a particular feature and that of their partner) and an other difference (the mean of the differences between the speaker's value and the values of each speaker in the corpus who was not their interlocutor). we infer proximity when partner differences are smaller than other differences. for conversation-level convergence, we split each conversation into two halves. in each half, we calculate each speaker's mean value for a particular feature. we then compare differences in speaker mean values in the first half to their differences in mean values in the second half using a paired t-test. we infer convergence when the differences in the second half are significantly smaller. we perform z-score normalization by speaker on all acoustic-prosodic features prior to any entrainment calculation to account for each individual speaker's natural pitch range and voice qualities. ## discussion from our experiments on the lexical features of code-switched conversations, we find statistically significant evidence of lexical entrainment across all of the word classes considered, as well as on overall language use within the corpus. we now briefly discuss why some speakers do not entrain to their interlocutors' overall language use even though their interlocutors entrain to theirs. prior research suggests that power differentials between speakers and other sociolinguistic factors tend to have an influence on such asymmetric entraining behavior, e.g. @xcite . we suspect that l2 speaker proficiency could additionally play a role in this, although this has not yet been explored, to our knowledge, because existing data sets do not include this information. this is an interesting direction for further work, and we plan to collect the relevant demographic data for inclusion in our future studies of entrainment in code-switching between different male speakers were about equally likely to initiate csw, with female interlocutors doing so in 8 of 15 such conversations. feature % w (ff+mm) % w fm csw pres. our experiments on acoustic-prosodic features provide evidence of entrainment in terms of turnlevel proximity and convergence. as with the lexical features we investigated, this finding aligns with prior work on monolingual domains, e.g. levitan and hirschberg (2011), though we acknowledge that these authors additionally found varying degrees of significant entrainment in terms of conversation-level proximity and convergence and turn-level synchrony. we suspect that this difference might be due to the undirected nature of the data we use, unlike the task-oriented conversations levitan and hirschberg studied. nevertheless, we still see indications of patterns of local entrainment in monolingual conversations generalizing to codeswitched conversations. we generally find consistent evidence of proximity at the turn-and conversation-level on the csw features of the bangor miami conversations. with respect to the presence of csw, we find turn-and conversation-level proximity and turn-level synchrony. though we find limited evidence of convergence at the turn-or conversation-level on csw presence, our results suggest that speakers are entraining to one another in a way that surfaces as one speaker's choice to code-switch in speech encouraging the other to choose the same and vary in turn. these results are mirrored in our experiments on the amount of csw, where we continue to observe that speakers seem to match one another's csw behavior, specifically in terms of the quantity of code-switched utterances used within a turn, and the variation of the same across turns. finally, from our experiments on csw strategies, we find significant synchrony at the turn-level on insertional csw, and significant proximity at the conversation-level on alternational csw. the first of these findings mirrors the trend we see in entrainment with respect to the presence of csw. since insertional csw is a strategy lacking strict syntactic structure, when speakers mimic their interlocutors' variation in csw in conversation, using insertional csw is a simple way to achieve this kind of entrainment. this finding also aligns with that of @xcite , who similarly found this csw strategy to show the most entrainment in human-machine written interaction. the second of these findings is more interesting, however, because alternational csw requires knowledge of both languages' syntax and is, by definition, the most structured strategy for speakers to employ. this probably makes it easier for interlocutors to notice and mimic in comparison to insertional or other csw. we particularly highlight this finding because significant evidence of entrainment on alternational csw has not been found in previous work on written csw, which suggests that there may be a key difference in how csw and subsequent entrainment behaviors are produced in the spoken versus written modality. finally, our preliminary qualitative gender analysis shows a slight skew toward entrainment taking place in same-gender conversations over mixed-gender ones. compared to prior work that has considered the role of speaker gender in entrainment behavior, both @xcite and @xcite found more entrainment in mixed-gender pairs than same-gender pairs, but these were on task-oriented monolingual conversations. our opposing results may suggest that the linguistic differences between task-oriented and undirected spontaneous speech (e.g. as detailed in @xcite ) carry through to differences in entrainment behaviors in these differing conversational settings, but further work is required to confirm or refute this. ## conclusion in this work, we have found that patterns of entrainment originally seen in monolingual domains are also present in a code-switched setting (rq1). this is shown in particular by our experiments on lexical and acoustic-prosodic features of a corpus of code-switched conversations.we also find evidence of entrainment on measures of csw -our work is the first to study entrainment in the context of spontaneous code-switched speech. while some patterns of entrainment previously found in code-switched written interactions are also present in spontaneous speech (rq2), we have found additional ones that suggest a fundamental difference in csw and corresponding entraining behaviors between modalities. we particularly find differences in the dimensions over which entrainment occurs in code-switched settings depending on the aspect of language production under consideration -i.e. turn-level proximity and convergence on acousticprosodic features, versus turn-and conversationlevel proximity on csw features -which points to the importance of examining multiple dimensions of entrainment when conducting analyses across feature sets. our preliminary gender analysis similarly shows possible differences in entrainment between undirected and task-oriented speech. while we have uncovered these distinctions, we plan to study them further in order to identify the reasons behind them in future work. overall, these results provide promising implications for the potential "universal" nature of entrainment as a paralinguistic communication phenomenon that may be independent of monolingual versus multilingual language production. 7 we hope this work will serve as a stepping stone for informing the design of spoken language technology applications, particularly in the generation of more naturalistic and context-appropriate code-switched utterances by voice assistants.