We’re Calling an Intervention: Exploring Fundamental Hurdles in Adapting Language Models to Nonstandard Text (2025)

Abstract: We present a suite of experiments that allow us to understand the underlying challenges of language model adaptation to nonstandard text. We do so by designing interventions that approximate core features of user-generated text and their interactions with existing biases of language models. Applying our interventions during language model adaptation to nonstandard text variations, we gain important insights into when such adaptation is successful, as well as the aspects of text variation and noise that are particularly difficult for language models to handle. For instance, on text with character-level variation, out-of-the-box performance improves even with a few additional training examples but approaches a plateau, suggesting that more data is not the solution. In contrast, on text with variation involving new words or meanings, far more data is needed, but it leads to a massive breakthrough in performance. Our findings reveal that existing models lack the necessary infrastructure to handle diverse forms of nonstandard text, guiding the development of more resilient language modeling techniques. We make the code for our interventions, which can be applied to any English text data, publicly available.

## introduction nonstandard text is all around us. whether a user adopts a regional dialect, follows different spelling conventions, or uses culturally-specific vocabulary, encountering text variation in most day-to-day nlp use cases is inevitable @xcite . yet, recent work continues to find large gaps in performance between standard and nonstandard text and speech @xcite , and efforts to reduce this gap are typically case-, variety-, or task-dependent @xcite . the distinction between standard and nonstandard text is often contextual. in written english, going to is standard, while gonna or gunna is nonstandard. likewise, i am not (standard) contrasts with i ain't @xcite . similarly, color (american english) vs. colour (british english) reflects orthographic variation. whether deploying a large-scale system for diverse users or working towards language model personalization, it is important to understand the challenges of adapting a language model to different varieties. specifically, how can we improve a model's performance in a new domain (in this case, a new variety) by leveraging knowledge from an existing one (i.e., the standard one)? this is tricky; it is difficult to tease apart the interactions between the complex and intertwined features that comprise linguistic variation, and the black-box nature of language models makes it even more difficult to do so. but we can reach some firmer conclusions if, on the one hand, we can devise ways of controlling different levels of text variation, and, on the other hand, we study the structure of the model (as opposed to its parameter values). we find that the model structure itself induces biases towards the standard variety of a language, and it does so in different ways at different levels of linguistic structure. current language models like bert @xcite and gpt @xcite are actually a hybrid of two models: a frequency-based subword tokenizer and a transformer-based encoder or decoder. the tokenizer determines subword units, and the transformer embeds the subwords into a vector space where it operates on the vectors. frequent words are often kept together as a single token, while infrequent words are often broken up into several shorter-length tokens. when both the tokenizer and the transformer are biased towards standard text, a rigid relationship develops between tokens and their vector representations. small changes to the token sequence (e.g., resulting from spelling variation) can break the model's ability to understand it properly @xcite , 2021; @xcite @xcite . see the example below, where tok ; denotes tokenization: coffee tok ; coffee cofee tok ; co, fe, e while the model would associate the token coffee with the appropriate word, one misspelling in the input means the model must figure out that the token sequence co, fe, e refers to the same thing as coffee, which is certainly not trivial. moreover, since the subword tokenizer of a pre-trained model cannot be modified without large-scale retraining, such issues can typically only be addressed through model adaptation methods (e.g., fine-tuning). however, because this approach does not resolve the underlying problem, it is crucial to explore the challenges and success cases in adapting language models to different forms of nonstandard text. to this end, we develop synthetic manipulations that exploit our knowledge of what happens when a transformer-based language model interacts with nonstandard text. our experiments isolate datarelated factors that can play a role in language model adaptation (e.g., type, amount, and composition of training data), and we assemble a suite of nine interventions to synthetically induce tokenization and embedding disruption, grounded in traits of user-generated text and at different levels of linguistic structure (e.g., orthographic, morphological, lexical), in controlled settings. we make the code for these interventions publicly available.1 our experiments (outlined in figure 1 ) evaluate and stress-test bert's ability to adapt to various types of nonstandard text under different conditions. our findings inform important questions about the fundamental hurdles of adapting language models 1https ://github.com/aarsri/ interventions-linguistic-variation to nonstandard text: ## related work as larger language models with new capabilities emerge, engaging with nonstandard text (e.g., dialects, language varieties, noisy text) -a hallmark of content generated by today's diverse user base -remains a challenge. language models are not robust to noisy text, plummeting in performance when faced with seemingly simple issues like misspellings, typos, and grammatical errors, even though such issues naturally arise in almost any use case @xcite @xcite . these limitations have been documented even in large models. for instance, @xcite find that llama 3 and 3.1 perform poorly in robustness tests involving character-level noise, and @xcite highlight the sensitivity of large language models to character-level variations. improving model performance on nonstandard text is not trivial. for instance, @xcite document a persistent performance gap between standard and dialectal text, even after in-variety fine-tuning. approaches to increasing robustness to linguistic variation, though often successful, tend to be highly dependent on the language variety and task, typically requiring data for each dialect in question @xcite and exposing differences in performance across tasks and varieties @xcite . given the vast, unpredictable nature of user-generated text, which often involves multiple types of linguistic variation at once, these challenges become even more pronounced. thus, it is essential to take a step back and develop a deeper understanding of why such adaptation remains difficult and what is needed to facilitate effective learning. past work has explored challenges of crosslingual transfer @xcite . for instance, @xcite investigate three key factors -embedding space disruption, tokenization changes, and word order changes -by transforming glue datasets to induce each type of shift. they find that while language models can adapt to tokenization and word order changes, they struggle to recover from embedding space disruption (e.g., learning new alignments). similar conclusions are drawn by @xcite and @xcite ; @xcite highlight the importance of subword overlap and token embedding alignment for successful knowledge transfer, while @xcite suggest leveraging word alignment and dictionary matching techniques. building on these findings, our experiments examine core features of nonstandard text and explore their interactions with these modeling challenges. ## levels of text variation variation is a natural response to the productive nature of language. it can be observed at all levels of linguistic reasoning @xcite : character-level change (e.g., spelling, abbreviation, orthography), morphological and syntactic variation (i.e., word and sentence structure), lexical and semantic variation (new word senses and meanings), and variation in style and tone, often related to language-external sociopolitical factors. style and tone variation operate at a higher level of reasoning, integrating features from the first three categories. viewing user-generated text in terms of these core features elucidates the challenges language models face in adapting to its variability. when dealing with nonstandard text, we encounter infrequent strings far more often, which inherently challenges existing biases of language models and impacts downstream performance. the tokenizer produces longer sequences of tokens comprised of shorter subwords, a phenomenon called oversegmentation @xcite . shorter subwords can appear in many more contexts and take on different meanings each time, a phenomenon called subword replacement @xcite . even in the case of new word senses/meanings, for which the token sequence may not change as much, the model would lack the relevant knowledge. drawing on documented features of usergenerated text, we have devised a suite of 9 interventions that operate at different levels of linguistic structure. we apply these interventions to all text used in our experiments. the interventions fall into four categories: character-level change, subword boundary manipulation, morphological variation, and lexical variation. all the interventions require the model to learn to map between standard (seen) and nonstandard (rare or unseen) text for effective knowledge transfer. however, differences in how the interventions affect tokenization can influence the difficulty of learning these mappings during fine-tuning. for this reason, the interventions are designed to cause a fair amount of disruption, with some acting as stress tests. all of our data and experiments are in english, but the algorithmic nature of many of our interventions can be extended to other languages. ## data preparation to compare the model's ability to adapt to each synthetic variety, we fine-tune and test it with the mask-filling objective on text with the appropriate intervention applied. we provide a sketch of our train/test pipeline in figure 1 . all of our data is sourced from wikicorpus @xcite to reduce external effects of choice of data. we reserve half the wikicorpus articles for fine-tuning and half for testing. these are separated into sentences using the sentence tokenizer from nltk @xcite . in our experiments, a word is any string that satisfies python's isalpha function and is an element of nltk word tokenizer's output. sentences with 0 or 1 words are eliminated. when fine-tuning, we vary the amount of data used. each split is a fixed set of sentences sampled from the fine-tuning articles. the number of sentences in each split covers three orders of magnitude. we report statistics in fine-tuning typically focuses on adapting a model to a specific task, making it challenging to simultaneously adapt the model to a new language variety. task-specific factors can also complicate this process, especially in experiments involving disrupted text. for example, some tasks, like intent classification, rely on identifying key words in the input, while others, like sentiment analysis, require an overall understanding of the sentence, and more linguistically complex tasks, such as linguistic acceptability, demand deeper grammatical reasoning. these variations create a complex interaction between handling perturbed text and meeting the requirements of each task, potentially introducing bias into the experiment. to avoid these complications, we select mask-filling as our fine-tuning task. mask-filling aligns with the models' pre-training objective, allowing fine-tuning resources to focus exclusively on adapting to the new language variety without interference from task-specific factors. our masking policy during fine-tuning is as follows. one word per sentence is randomly selected to be masked. we practice whole-word masking, meaning the model could be asked to fill one or more consecutive mask tokens, given the number of subword tokens that comprise the masked word. depending on the word to be masked, it is possible the intervention does not actually change that word. while this is natural for fine-tuning, to make sure the comparison during testing is fair, we mandate that the word masked at test time is actually modified by all nine interventions. this filtering yields a test set of 931 sentences sampled from the testing articles. for each sentence, the same word is masked in all interventions/tests for consistency. ## experiments there are four axes of variation in our experiments: amount of fine-tuning data (small, medium, large), composition of fine-tuning data (50% (mixed) or 100% (full) of sentences are noised), intervention to be applied (9 total), and multilinguality (monolingual or multilingual pre-trained model). we use bert-base-cased2 (bert) as the monolingual english model, and bert-base-multilingual-cased3 (mbert) as the multilingual model. we do not include character-level models like characterbert @xcite and canine @xcite in our experiments; unlike bert and mbert, they cannot be used out-of-the-box for mask-filling and would not yield comparable results. for larger models, it is often unclear whether success stems from pre-training exposure, parameterbased knowledge retention, or true adaptation (our focus). using bert, we minimize such confounds. for each possible combination of the four axes of variation, a pre-trained model is fine-tuned on the corresponding data with the masked language modeling objective. as described in section 4, one word per sentence is masked (whole word masking), and the fine-tuning objective is to fill the masked 2https ://huggingface.co/ bert-base-cased 3https ://huggingface.co/ bert-base-multilingual-cased token(s) with the tokens of the original word. we use low-rank adaptation (lora, hu et al. ( 2021 )) for parameter-efficient fine-tuning, which adapts the attention weights of each transformer encoder layer and freezes all other parameters. we follow hu et al.'s guidelines for hyperparameter choice, using the adamw optimizer @xcite with a linear scheduler, lora rank of 8, and lora scaling factor 𝛼 of 8. we use learning rate 7 • 10 -4 for the small data amount and 5 • 10 -4 for the medium and large data amounts. on an nvidia a10 tensor core gpu, fine-tuning takes 12 seconds/epoch for the small data amount, 2 minutes/epoch for the medium data amount, and 18 minutes/epoch for the large data amount. lora was chosen over standard fine-tuning for two key reasons: (1) standard fine-tuning is highly susceptible to distribution shift issues, and (2) lora provides greater control over where learning occurs (in encoder parameters associated with attention). we measure performance with three metrics: from most to least strict, exact match, 1-best, and 5-best accuracy. exact match accuracy measures for how many masked words each token of the word is predicted correctly, divided by the test set size. 1best accuracy measures the total number of masked tokens filled correctly (by the top probability prediction) divided by the total number of masked tokens in the test set. similarly, 5-best accuracy measures the total number of masked tokens whose top five predictions include the correct answer, divided by the total number of masked tokens in the test set. the 1-best accuracy metric provides the best summary of the results ## results the main 1-best results for our experiments are found in ## conclusion we introduce a suite of interventions that synthetically modify english text to analyze interactions between features of nonstandard and user-generated text with underlying biases of language models. our experiments isolate data-related factors that can contribute to language model adaptation, revealing critical insights into the limitations of adapting language models to nonstandard text. we explore several cases, ranging from character-level changes that over-segment the token sequence to lexical variations that alter contextual representations. some interventions require learning one-to-many mappings within subwords, while others demand associations across multiple subwords. our findings highlight key adaptation challenges. bert-like models struggle to adapt to characterlevel changes, even with additional data, but can successfully handle lexical shifts and new word senses with enough exposure to relevant fine-tuning data. notably, interventions involving affix changes and antonym substitutions achieve performance com-parable to or exceeding mbert baselines. this suggests that while models can effectively learn word-to-word mappings, structural constraints hinder their ability to process finer-grained variations within subwords or at the character level. these limitations arise from how tokens are segmented and represented within the model, restricting its ability to capture within-subword (sub-atomic) variations. ultimately, while current models adjust well to meaning-related variations given enough data, they struggle with fine-grained structural disruptions, where constituents are smaller than a subword (e.g., character-level). despite the prevalence of such variation (e.g., social media), current language models lack the capability to facilitate the required flexibility between tokens and embeddings for straightforward adaptation methods like finetuning, unless the relevant knowledge is already incorporated during pre-training. this underscores the need for more flexible tokenization and modeling approaches, especially for handling the complexities of user-generated text and enabling the model to effectively capture within-subword (e.g., character-level) information.