Nonparametric Masked Language Modeling (2023)

Abstract: Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NPM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NPM fills in the [MASK] solely from retrieving a token from a text corpus. We show that NPM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval. Zero-shot evaluation on 16 tasks including classification, fact probing and question answering demonstrates that NPM outperforms significantly larger parametric models, either with or without a retrieve-and-generate approach. It is particularly better at dealing with rare patterns (word senses or facts) and predicting rare or nearly unseen words (e.g., non-Latin script). We release the model and code at github.com/facebookresearch/NPM.

## introduction current large language models, despite their wide use and impressive performance, are expensive to scale, difficult to update, and struggle with long-tail knowledge and patterns @xcite . recent work follows a retrieve-and-generate approach to partially address these issues @xcite ; however, their final predictions are still made by a parametric model. in particular, they still include a softmax over a finite vocabulary, which limits expressivity @xcite and can make them reluctant to predict rare or unseen tokens (e.g., thessaloniki in figure 1 ). in this paper, we introduce npm, the first nonparametric masked language model that predicts tokens solely based on a nonparametric distribution over phrases in a text corpus (figure 1 ). npm consists of an encoder that maps the text item delivered broken. very cheaply made and does not even function. ## method we introduce npm, the first nonparametric masked language model. npm consists of an encoder and a reference corpus, and models a nonparametric distribution over a reference corpus (figure 1 ). the key idea is to map all the phrases in the corpus into a dense vector space using the encoder and, when given a query with a [mask] at inference, use the encoder to locate the nearest phrase from the corpus and fill in the @xcite . encoder-only models are competitive representation models @xcite , outperforming the other two classes of models in classification tasks (section 5.4). however, existing encoderonly models are unable to make a prediction whose number of tokens is unknown, making their use cases limited without fine-tuning. npm addresses this issue, since it can fill in the [mask] with an arbitrary number of tokens by retrieving a phrase. we first describe inference of npm assuming a learned encoder (section 3.1), and then describe how we train the encoder to map the text into a good vector space (section 3.2). ## experiments: open-set tasks we include zero-shot evaluation on open-set tasks whose answer can be any arbitrary-length string. ## conclusion we introduced npm, a nonparametric masked language model that replaces a softmax over the output vocabulary with a nonparametric distribution over a reference corpus. npm can be efficiently trained using a contrastive objective and an in-batch approximation to a full corpus. zero-shot evaluation on 16 tasks shows that npm outperforms significantly larger parametric models. npm is particularly good at rare patterns (word senses or facts), scaling and updating at test time, and predicting extremely rare if not unseen characters.