Speaker Information Can Guide Models to Better Inductive Biases: A Case Study On Predicting Code-Switching (2022)

Abstract: Natural language processing (NLP) models trained on people-generated data can be unreliable because, without any constraints, they can learn from spurious correlations that are not relevant to the task. We hypothesize that enriching models with speaker information in a controlled, educated way can guide them to pick up on relevant inductive biases. For the speaker-driven task of predicting code-switching points in English–Spanish bilingual dialogues, we show that adding sociolinguistically-grounded speaker features as prepended prompts significantly improves accuracy. We find that by adding influential phrases to the input, speaker-informed models learn useful and explainable linguistic information. To our knowledge, we are the first to incorporate speaker characteristics in a neural model for code-switching, and more generally, take a step towards developing transparent, personalized models that use speaker information in a controlled way.

## introduction imbalanced datasets, flawed annotation schemes, and even model architectures themselves can all cause neural models to encode and propagate biases by incorrectly correlating social information with labels for a task @xcite . as a result, models may be brittle and offensive in the presence of racial or gender attributes @xcite , unsuitable for processing mixed-language text or dialect variations @xcite @xcite , or ones that can miscommunicate intents in translation setups. contextualizing models in social factors is important for preventing these issues and building more socially intelligent and culturally sensitive nlp technologies @xcite . we hypothesize that grounding models in speaker information can help them learn more useful inductive biases, thereby improving performance on person-oriented classification tasks. we test this hypothesis on the task of predicting code-switching (language mixing) in a multilingual dialogue, which is inherently linguistically and socially driven @xcite . prior approaches for predicting code-switching consider only shallow linguistic context @xcite . as we show in our experiments foot_0 , using a standard transformer-based classifier @xcite trained with only linguistic context results in suboptimal and unstable models. moreover, we believe code-switch prediction is a suitable first task for learning speaker-driven inductive biases; we can test whether models learn useful relationships between social attributes while minimizing the risk of building a model that perpetuates social prejudices. we ground the models in relevant social factors, such as age, native language, and languagemixing preference of the interlocutors, via textbased speaker descriptions or prompts (cf. @xcite . we find that prepending speaker prompts to dialogue contexts improves performance significantly, and leads to more stable generalizations. our prompts are different from the embedding-based personas of @xcite and the synthesized descriptions from persona-chat @xcite , capturing theoretically grounded social and linguistic properties of speakers, as opposed to hobbies or occupations. to analyze the inductive biases that the models learn, we use selfexplain @xcite )an interpretable text classification model highlighting key phrases in the input text. we propose a new method for aggregating the interpretations produced by selfexplain to explain model predictions and align them with sociolinguistic literature. we motivate our study of predicting codeswitching in §2, and describe the task and inter-pretable neural text classification models in §3. after outlining important ethical considerations in §4, we detail our experiments ( §5) and results ( §6), and provide an analysis of speaker-aware model generalizations that are grounded in prior psycholinguistic research on code-switching ( §7). ## motivation our overarching goal is to develop a general and theoretically-informed methodology to ground neural models in a social context, because a wide array of person-centric classification tasks, such as sentiment prediction or hate speech detection, can fail without proper social contextualization @xcite @xcite . we choose a speaker-driven task that is ethically safer to experiment with (see a detailed discussion in §4): predicting code-switching in human-human dialogues. code-switching is the alternation between languages within and between utterances (see appendix a.1 for a detailed example of code-switched dialogue.) it is a language-and speaker-driven phenomenon, reflecting speaker identities and relationships between them, in addition to their linguistic backgrounds, preferences and topical constraints @xcite . prior sociolinguistic work established the importance of speaker context for code-switching, and existing multilingual models-trained with only monolingual linguistic context-are not speaker-grounded nor well-suited for dealing with mixed-language data, leaving gaps which we begin to address. figure 1 provides a key motivating example of how global speaker features of two bilingual conversational participants influence their local speech production. blue, whose native language is spanish, begins speaking in spanish, while green responds in english. following green's clarification question about the actor the rock, green begins in english, but will accommodate blue @xcite to reply with el actor (spanish), motivating the need for social context when processing mixed-language data. ## methodology in this section we introduce the task of predicting code-switching points and describe the base model for it, with a self-explainable architecture as its backbone. we then describe how we incorporate speaker-grounding prompts into the model. but they [actors] are very serious for those movies z 🇨🇺 🤖 multilingual transformer if you're a good actor you can do it! look at the rock. 👧 🇬🇧 👵 is ___ spanish? (code-switched) (2) ours: speaker-aware yes, because blue prefers spanish; intj. ("uh") no, because previous two utterances are in english ✅ ❌ pero son muy serios para esas películas the rock? uh, the (1) baseline: speaker-agnostic ## evaluation speaker prompts significantly improve codeswitch prediction. we see similar trends, regarding accuracy, f1, and standard deviation, in using irrelevant speaker descriptions worsens model performance. as a control, we generated synthetic descriptions for each speaker, including features such as favorite foods and weather, owned pets, and height. none of these attributes are discussed in the conversations and would not explicitly influence code-switch production. after generating descriptions in the sentence and partner format, we prepend them to dialogues using a context window of 5. according to the results in trained with a context window of 5. the results indicate that domain knowledge is useful to understand which speaker features to add to a model to improve performance, and they give more support to the claim that relevant speaker information helps guide models to useful inductive biases. ## explaining performance gaps compared to baseline models, speaker models not only attain higher accuracy and f1 scores, but they also have a much smaller standard deviation in scores. for these experiments, we seek to explain our findings using the important phrases identified by lil. within a speaker prompt p, each speaker characteristic maps to its own phrase (i.e., from an english-speaking country); in the dialogue, we extract 5-gram phrases using a sliding window. we detail our approach to scoring phrase influence and analyze key dialogue and speaker features. ## related work our use of prompts foot_1 is similar to @xcite and @xcite , who rely on prompts to put models in different states for different tasks. speaker personas open-domain dialogue agents which act according to a persona are more natural and engaging than the non-personalized baselines @xcite ; these personas can be short, superficial descriptions generated through crowdsourcing @xcite , gathered from reddit @xcite , or self-learned (inferred) from dialogue context @xcite . these works, however, primarily evaluate dialogue content and only in one language (english) instead of analyzing how speaker properties influence the downstream dialogue structure. addressing model bias prior works for mitigating social biases feature adversarial learning @xcite , counterfactual data augmentation @xcite or dataset balancing @xcite , and more recently, using an interpretability-driven approach to uncover and controllably demote hidden biases @xcite . techniques for adapting to linguistic variants and mixed-language data include adversarial learning to pick up on key linguistic cues @xcite , augmenting datasets with synthetic text @xcite or examples of variants that models underperform on @xcite , discriminative learning (gonen and goldberg, 2019), and transfer learning with morphological cues @xcite . ## conclusion to the best of our knowledge, this is the first work incorporating sociolinguistically-grounded social factors in an interpretable neural model for codeswitch point prediction. our speaker-aware models can better leverage mixed-language linguistic cues, compared to a text-only baseline: specifically, we showed performance gains of up to 7% in accuracy and .05 points in f1 scores on an imbalanced code-switching dataset. our work is limited to one language pair and uses a small dataset. thus, additional studies are necessary to assess the generalizability of our findings to other languages. moreover, speaker identities can change dynamically in different settings. linguistic preferences may also change over time. we could move beyond static personas, refining them using local dialogue context. in addition, speaker-grounded models must be carefully engineered to protect user privacy, using proxies for personal information and keeping private information away from shared resources. in the future, we would like to explore whether such speaker prompting can improve models in other person-centered tasks, e.g., coreference resolution (especially for datasets explicitly testing gender biases) or sentiment analysis. using techniques such as data augmentation, we can explicitly guide models away from biases learned during training. with ethical considerations in mind, our work advances the state-of-the-art in building more adaptable and person-aware nlp technologies. ric yeah? she knows about it? seb she's good with it. ric i have seen the… this es un… ese un era un documental*. seb de lo que paso en realidad con, this guy, uh, el que adivina el futuro* seb seb i prefer english. i can also hold a conversation in spanish. ric i prefer spanish, but i am a skilled english speaker. -0.2 and case (2) with 0.2 to distinguish that in case (1), the ablated phrase changes the final prediction.