Mixed-effects transformers for hierarchical adaptation (2022)

Abstract: Language differs dramatically from context to context. To some degree, large language models like GPT-3 account for such variation by conditioning on strings of initial input text, or prompts. However, prompting can be ineffective when contexts are sparse, out-of-sample, or extra-textual. In this paper, we introduce the mixed-effects transformer (MET), a novel approach for learning hierarchically-structured prefixes— lightweight modules prepended to an input sequence— to account for structured variation in language use. Specifically, we show how the popular class of mixed-effects regression models may be extended to transformer-based architectures using a regularized prefix-tuning procedure with dropout. We evaluate this approach on several domain-adaptation benchmarks, finding that it learns contextual variation from minimal data while generalizing well to unseen contexts.

## introduction while certain aspects of language use are nearly universal -such as basic grammatical acceptability @xcite or simple lexical judgements @xcite -these often seem to be the exception that proves the rule. contextual variation is ubiquitous in language, where predictions may differ as a function of speaker identity @xcite @xcite , location @xcite , time @xcite @xcite , or usage domain @xcite @xcite . although such variation has long been recognized in psycholinguistics @xcite and sociolinguistics @xcite , the dominant approach in modern nlp has been to train monolithic models @xcite and fine-tune for individual domains if necessary (e.g. @xcite . recent large language models (llms) like @xcite have begun to provide a more systematic approach for handling context-specific variance. by adding relevant contextual information to the text input (i.e. prompting), these models have been able to account for known demographic information such as the speaker's age, gender, or country of origin @xcite . however, it is less clear how to use prompting when context is extra-textual, contains multiple features, or lies outside the training distribution. for example, llms trained prior to the covid-19 pandemic failed catastrophically on the torrent of new tweets and medical papers @xcite @xcite . in these cases, some degree of online adaptation is required. one particularly promising adaptation technique is prefix-tuning, where a lightweight module is prepended to the input and fine-tuned to modulate a downstream network that has been frozen @xcite . to date, however, this technique has only been used to fine-tune prefixes for distinct downstream tasks (see also @xcite @xcite . in this paper, we suggest that the prefix-tuning approach is particularly wellsuited for hierarchical adaptation in language modeling. specifically, we show how a form of dropout may be used to implement random effects, yielding a mixed-effects transformer (met; figure 1 ). this approach allows the model to learn strong domain-specific predictions for frequently occurring prefixes while abstracting away generalizable inductive biases for sparser or unseen contexts. our code is available at https://github.com/ juliaiwhite/mixed-effects-transformers . ## approach we begin by reviewing mixed-effects models in a classic hierarchical regression setting before extending it to explicitly model contextual variation with modern language models. ## datasets we examine language use across contexts in three distinct domains: product reviews, online posts, and movie dialogue. 100,000 sentences were sampled for training from 10 distinct product categories within the amazon customer reviews dataset foot_1 , a.k.a product reviews; 100,000 sentences were sampled from 10 subreddits (subsidiary forums representing distinct topical communities) within the reddit corpus @xcite ; and, 10,000 sentences were sampled from 10 genres within the cornell movie-dialogs corpus (danescu-niculescu-mizil and lee, 2011), a.k.a movie corpus. further information about ## results we evaluate the ability of the met to capture language use within known and novel contexts. further, we assess the data efficiency of our method and its ability to represent complex contexts with multiple relevant features. we compare the performance of our approach against several baselines. in the complete-pooling and no-pooling variants of prefix-tuning we ablate different components, only learning a single prefix shared across all features, or only learning independent prefixes, respectively. we also compare a traditional domain adaptation approach, where we omit prefixes and fine-tune the transformer end-to-end either on the entire dataset (complete pooling) or for each feature separately (no pooling). finally, we compare our method against conditional fine-tuning, where a string representing the prefix text (e.g. [corpus] movie_dialogue [genre] horror ) is prepended to the input and the model is fine-tuned end-to-end. see appendix b for additional details. ## conclusion human language is flexible, and people are able to adapt their expectations to many aspects of context, from speaker identity to the conversational setting. in this paper, we introduce mixed-effects transformers (mets) as an effective method of adapting to hierarchically structured domains of language use across labeled contextual features. beyond language modeling, this approach may be useful for controlled generation and more qualitative analyses of what makes certain features distinctive (see appendix d for preliminary analysis).