CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals (2022)

Abstract: We propose a framework to modularize the training of neural language models that use diverse forms of context by eliminating the need to jointly train context and within-sentence encoders. Our approach, contextual universal embeddings (CUE), trains LMs on one type of contextual data and adapts to novel context types. The model consists of a pretrained neural sentence LM, a BERT-based contextual encoder, and a masked transfomer decoder that estimates LM probabilities using sentence-internal and contextual evidence. When contextually annotated data is unavailable, our model learns to combine contextual and sentence-internal information using noisy oracle unigram embeddings as a proxy. Real context data can be introduced later and used to adapt a small number of parameters that map contextual data into the decoder’s embedding space. We validate the CUE framework on a NYTimes text corpus with multiple metadata types, for which the LM perplexity can be lowered from 36.6 to 27.4 by conditioning on context. Bootstrapping a contextual LM with only a subset of the metadata during training retains 85% of the achievable gain. Training the model initially with proxy context retains 67% of the perplexity gain after adapting to real context. Furthermore, we can swap one type of pretrained sentence LM for another without retraining the context encoders, by only adapting the decoder model. Overall, we obtain a modular framework that allows incremental, scalable training of context-enhanced LMs.

## introduction language models (lms) estimate the prior probabilities of token sequences and are key probabilistic modeling components in a variety of applications, such as speech recognition, machine translation, or software keyboards. when modeling linguistic token sequences, typical lms model one sentence or utterance at a time, reflecting the fact that the strongest predictors of words are syntactic constraints and semantic associations within the sentence. however, it has long been recognized that context beyond the sentence has substantial influence on the word probabilities within a sentence. context literally means the surrounding text (or preceding text, when predicting words in temporal order), but can also refer to any extra-linguistic information, such as metadata (e.g., authorship, time, location) or associated other modalities (e.g., visual cues associated with a spoken utterance). there is a large literature on leveraging such contextual information for language modeling, some of which we review below (section 2). however, including context in language modeling presents major challenges for operational settings, especially when lms need to be trained and deployed at scale: a newspaper corpus has metadata that is very different from spoken language data. all these difficulties lead to context being used sparingly in most practical settings, and only when it yields substantial benefits (such as in using a user's personal contact list in voice dialing). in this paper, we propose a modular modeling framework for contextual language models, called contextual universal embeddings @xcite . the fundamental idea is to separate the modeling of (1) sentence-internal lm, (2) context embedding and (3) combination of sentence-internal and contextual information each into their own modules. first of all, we show that this architecture is an effective way to bring context to bear on the lm task, achieving 25% relative perplexity reduction over a sentence-internal model, on a corpus of newspaper articles with rich metadata. more importantly, each module can be trained separately, as opposed to jointly with the other modules. through experimentation we show that, for the practically important use-cases, training modules separately or incrementally preserves most of the achievable gain from contextual information. specifically, we can replace one type of context with another, while only adapting the context encoders to the new context, and retain 85% of the best-case perplexity gain. maybe more surprisingly, we can train the decoder that combines context and sentence-internal information without any actual context, instead using noisy oracle unigram embeddings as a proxy. this recovers 67% of the best-case gain after adapting to real context. (adapting the context encoders affects much fewer parameters, and takes much less data, than the model overall.) finally, we show that context encoders can be frozen and a whole different sentence-lm architecture swapped into the model ensemble. after adapting only the combiner-decoder we obtain perplexity gains close to the optimum that would have been achieved by joint training of combiner and context embedding. ## prior work longer text history is the most commonly used context in language models (lms) @xcite @xcite . a naive way to bias a lm over text history is to ignore the sentence boundaries and train the contextual lm as the standard neural lm @xcite . however, recurrent neural networks suffer training difficulties on longer sequences @xcite while transformerstyle models are effective at incorporating this extra information @xcite . another approach is to summarize context into a single context embedding using a separate model. for example, @xcite and @xcite use topic information extracted from the context. @xcite use a pretrained latent dirichlet allocation (lda) model while @xcite use paragraph embeddings learned during lm training. @xcite on the other hand, use a bag-of-words of whole text or individual sentences in the context to build the context vector. @xcite and @xcite further extend sentence-based contextual models by using hierarchical embedding techniques. this approach learns a representation of the context that is directly used as input to a neural lm. other sequence tasks in natural language processing (nlp) also leverage contextual information. neural machine translation (nmt) capitalizes on the availability of previous sentences on the source and target sides when translating documents @xcite @xcite . the only difference in the approaches is how the context is encoded into a representation optimized for nmt. automatic speech recognition (asr) and conversational dialog systems also use contextual information, such as recent advances in shallow or deep fusion of end-to-end neural architectures @xcite @xcite @xcite . recent papers have also considered biasing lms with context beyond the previous sentence and incorporate additional signals such as date-time, geolocation or gender @xcite or application metadata like dialog act or intent @xcite @xcite . other sources of context used to bias lms are personalized content @xcite ; conversational turn-taking @xcite ; multi-modal sources @xcite ; or even user demographics to suggest fashion suggestions (denk and peleteiro ramallo, 2020). ## architecture our task is to estimate an auto-regressive language model conditioned not only on the previous words in the sentence, @xmath0, w 2 , . . . , w n , but also on several contextual signals, where @xmath1, ..., c k ] represents a set of k contextual inputs that may vary with each sentence, and where each c k is a sequence of tokens from the same vocabulary as the target sentence. adapting recent work in hierarchical contextual embeddings @xcite , our cue architecture has three components (see figure 1 ): our goal is to train a separate context encoder that can be updated without modifying the sentence encoder or decoder and run inference independently of the other modules. this is important for operational practicalities, to precompute context embeddings and update them incrementally without requiring downstream components to change. ## adapting to evolving context we now no longer assume that the set of context types is static between training and test. for example, an api providing context may be retired; or business rules improving customer privacy may remove geographic information. the set of contexts may evolve over the life-cycle of our cue encoder and we now introduce an adaptation step. 1 ideally, we would jointly fine-tune the entire model architecture (context encoder, sentence encoder and decoder) on annotated data that contains the new context types. however, this creates an operational burden since different downstream decoders that use context embeddings would each need retraining. our goal is to adapt the cue context encoder while leaving the decoder parameters frozen. this will minimize the number of parameters to be retrained and simplify model deployment by factoring the context encoder from the decoder. we break the training process into two phases: training constructs the initial set of model parameters and is not constrained by operational needs. adaptation happens at some later point in time after the set of context types changes. section 4.1 considers the scenario where new context types are added to or replace the initial training types. section 4.2 assumes no context is available during model training, only at adaptation time. ## corpus we used the new york times annotated corpus @xcite released through linguistic data consortium (catalog number ldc2008t19) containing over 1.8m english articles spanning 1987 to 2007. this corpus includes a rich collection of contextual annotations for each article, ideal for evaluating our cue framework. each article contains up to 47 different metadata types that were labeled by humans (author, title, desk) or algorithmically (locations, topic). we down-selected from 47 to 11 distinct metadata signals after removing redundant or uninformative context. all context types were character sequences and include previous sentence, title, author, entities present in the article, section descriptors, date, and topic descriptors (see appendix a for details). articles averaged 32 sentences in length and average sentence length (after tokenization) was 26. we trained the sentence encoder on a large subset of articles; used separate training and adaptation corpora and separate validation and test sets ( ## discussion we analyzed whether a sentence's context behaves like a cache model, since it contains textual data from the previous sentence, title, and other contexts. to better understand this effect, we divided test data tokens into two bins: those that appeared in the text of the sentence's context and those that did not. we measured the relative gain in log likelihood when conditioning the lm on context versus not. 30% of the tokens appeared in the context (cache) and the relative gain was 74%-there clearly is a strong benefit for recurring tokens and the cue encoders capture this effect. the 70% of tokens that do not occur in the cache improved their log likelihood by 26% relative. so the cache effect does not explain the entire benefit of cue vectors and correlations among different token types are captured as well. the top context types that had tokens in the sentence were previous sentence (23%), title (9%) and person (6%). to verify that the empirical improvements from the previous sections are semantically plausible, we analyzed the context embeddings of the first sentence of 5000 heldout articles. these embeddings do not contain information from the previous sentence and thus represent the entire article's metadata. figure 6 projects these embeddings down to two dimensions with t-sne. we then clustered the vectors with k-means and aggregated word counts for all articles within a cluster. finally, we display the five most salient words (computed with tf-idf) from the context and, separately, from the article text. even though the articles were clustered based only on context, the groupings of article text are semantically meaningful, with clear clusters related to newspaper sections such as corrections, marriage announcements, sports and other news related topics. our context embedding is preserving semantic information. one limitation of the proxy embedding approach is that they may not extend to other nlp tasks, like named entity tagging. since they are derived from the unigram embedding, they directly encode the targets of the language model task. this may not prove useful for higher-order annotations and further work should look into a multi-task proxy embedding that directly optimizes an "interface" embedding space instead of a unigram distribution. ## conclusions we introduced the cue framework to factor context encoding and next word prediction of contextaware neural language models. unlike previous work, we do not assume that the set of context signals is constant between training and test. we optimize the model architecture to reduce the operational burden of managing and retraining of large neural lms over their life cycle. our approach is robust to changing context types; by adapting only 5% of the parameters, we recover 85% of the possible gain from jointly training all components. furthermore, we introduce proxy embeddings to pretrain a decoder to be attuned to external context embeddings even when those are not known at training time. this approach is 67% as good as jointly training with all context. in future work, we would like to handle missing context at inference time through data imputation or dropout approaches. furthermore, we plan to extend the proxy embedding approach such that the context encoders can be trained fully independent of the decoder.