Language Models as Context-sensitive Word Search Engines (2022)

Abstract: Context-sensitive word search engines are writing assistants that support word choice, phrasing, and idiomatic language use by indexing large-scale n-gram collections and implementing a wildcard search. However, search results become unreliable with increasing context size (e.g., n>=5), when observations become sparse. This paper proposes two strategies for word search with larger n, based on masked and conditional language modeling. We build such search engines using BERT and BART and compare their capabilities in answering English context queries with those of the n-gram-based word search engine Netspeak. Our proposed strategies score within 5 percentage points MRR of n-gram collections while answering up to 5 times as many queries.

## introduction a wide range of computer tools has been developed to support the writing process, including both active and passive ones. active tools automatically paraphrase a text as it is written, if the text is highly likely to be incorrect or stylistically inappropriate. passive tools suggest either spelling, grammar, and style corrections or how to continue a sentence. passive tools that are less integrated into word processors are context-free and context-sensitive word search engines. context-free search engines include searchable dictionaries, thesauri, and collections of idioms in which queries are made about a known word or phrase for which alternatives are sought. in the absence of context, their search results are usually sorted alphabetically. contextsensitive word search engines allow their users to formulate cloze-style queries to search for an unknown word or phrase, ranking the search results according to their frequency of use. a conventional context-sensitive word search engine, as shown in figure 1 , answers a cloze 1 our code is available at github and our data is available at zenodo. query @xmath0. formally, the index µ : @xmath1, which is implemented as wildcard retrieval, and the results µ(q) = d q are ordered by their occurrence frequency in a large text corpus, which approximates the frequency of use. assuming a sufficiently large text corpus is available such that each n-gram matching a given cloze query q has been observed sufficiently often, ranking these n-grams by their frequency satisfies the probability ranking principle @xcite . in other words, if one asks a sufficiently large number of people to answer a cloze query, the frequency distribution of the answers would correlate with that of the n-grams found. the main limitations of this approach are, (1) that the number of context words in each cloze query is limited by n, with more context reducing the size of the cloze accordingly, and, (2) that the size of the text corpus required to observe q sufficiently often increases exponentially with n, so that in practice @xmath2. in this work, these two limitations are addressed by using transformer-based language models to predict phrases corresponding to a query, rather than retrieving them from an n-gram index. in particular, we propose a masked language model and an autoregressive model for conditional generation to answer cloze queries (section 3). these models are compared to netspeak, a state-of-the-art netspeak dbert dbert f t bart bart f t (1) this @xmath3: selected context queries with @xmath4. the original token in the results is underlined, the overlap with netspeak's results is boldface. ## related work in general, context-sensitive word search engines are supportive writing assistants targeting the editing phase of the writing process @xcite . supportive writing assistants take the form of online dictionaries, thesauri, concordancers (like writebetter (bellino and bascuñán, 2020)), or other resources offering definitions, synonyms, and translations. more advanced assistants provide a tailored query language that allows for searching words matching a pattern (onelook.com), words that rhyme (rhymezone.com), or words that fit a given context (e.g., netspeak @xcite , google n-gram viewer @xcite , linggle @xcite , and phrasefinder. io). context-sensitive word search is related to several foundational nlp tasks like lexical substitution (mccarthy and navigli, 2007; lee et al., @xmath5.3 0.14 0.07 ... ... red silver old ... ... the red fox the old fox the silver fox ... d q bidirectional encoder (a) mlm for word search. @xmath6.3 0.14 0.07 ... ... red silver old ... ... <@xmath7. decoder d q (b) denoising for word search. @xmath8. @xmath9. figure 2: context-sensitive word search can be learned using masked (mlm) or conditional language modeling (cdlm) with denoising or infilling. the result set d q for mlm and denoising is the output at the mask's position sorted by likelihood. for infilling, d q is the generation target. our proposed mlm is trained and finetuned as usual; our cdlm is trained by denoising and finetuned by infilling, but predicts via denoising. ## language modeling for word search in this work, we formulate context-sensitive word search with language models as learning a distribution p(w q | q), where @xmath10. either q l or q r can be empty. the result set d q consist of all n-grams q l w q,i q l for all w q,@xmath11, in masked language modeling masked language modeling (mlm) is equivalent to context-sensitive word search with only a single token as the answer. since large language models based on transformerencoders solve mlm by learning p(w q | q) and scoring all options in the vocabulary, the scored vocabulary can be used to extract d q . as shown in figure 2a , we use a bidirectional transformerencoder (bert) model, pre-trained with mlm, to estimate p(w q | q). we extract the 30 tokens with the highest score from the output logits of the language modeling head as d q . we fine-tune the model with a specialized masked language modeling task, using individual n-grams as input. although any bert variant can be used, we choose distilbert for its size and speed, since contextsensitive word search is a real-time search task. conditional language modeling conditional language modeling (cdlm) is causal (or generative) language modeling given a condition. contextsensitive word search can be formulated as cdlm with two strategies: denoising (see figure 2b ) and infilling (see figure 2c ). denoising takes the query as the condition and generates the original sequence, where d q can be extracted from the output logits at the mask's position, as with an mlm. infilling takes the query as condition and generates d q . we use a sequence-to-sequence model for conditional generation (bart) and predict d q with denoising, extracting the 30 tokens with the highest score. we fine-tune bart using infilling, but use denoising to predict d q after the fine-tuning. ## experimental setup we implemented both strategies of learning contextsensitive word search using the huggingface @xcite implementation of distilbert for mlm and bart for cdlm. we evaluate the pretrained and the fine-tuned models against the two datasets with word search queries shown in ## evaluation we quantitatively evaluate our proposed methods using the mean reciprocal rank (mrr) and the normalized discounted cumulative gain (ndcg) @xcite . system performance we evaluate the system performance using the mrr of the author's chosen word, shown in the mrr results allow three conclusions. first, our proposed fine-tuning strategy improves the pre-trained baseline's performance consistently for bart and on queries from wikitext for dbert. second, on queries from ra, the best models already perform close to netspeak. third, both fine-tuned models can answer 4-5 times as many queries than netspeak, which can be observed from the ratio between ra and ot datasets. since the ot dataset contains up to 82% uncommon queries, which have no support in the google ngrams indexed by netspeak, the language models score up to 9 percentage points lower than on ra. the mrr increases with increasing context size since additional context can only reduce the set of potentially matching answers. ranking we evaluate the ranking of the results using the ndcg as shown in figure 3 . consistent with the mrr results, the fine-tuned models outperform their pre-trained counterpart, dbert profits more from fine-tuning and performs best. most of the relevant results are in the top ranks since the ndcg scores only marginally improve past rank 10. position and word class we evaluate further query attributes besides size and genre, wildcard position, and wildcard word class, using the mrr as shown in figure 4 . these results show that a large part of the performance gain when fine-tuning can be attributed to gains in the closed-class words. the mrr is lower for open-class words since there are more plausible options for each query and the original token is on a lower rank more often. finetuning has only a marginal impact on open-class words. dbert scores the lowest when the wildcard is either at the beginning or at the end of the query, while bart scores the lowest for wildcards at the beginning. fine-tuning significantly improves the performance in these cases, with only marginal improving queries with wildcards in the center positions. the performance difference between closed and open-class words also partially explains the substantially lower mrr and ndcg scores over cloth queries for all models: the answers to cloth-queries more often belong to lower scoring open classes, the answers to wikitext-queries more frequently belong to the high scoring closed classes. runtime we compare the runtime performance by measuring the average time to answer a query (see ## conclusion this paper investigates whether state-of-the-art language models can mitigate the shortcomings of n-gram indices in context-sensitive word search engines. we present strategies to fine-tune masked and conditional language models so that they can answer word search queries. our evaluation shows that our proposed methods can answer short queries (3 tokens) nearly as well as by observing actual n-gram frequencies in a large text corpus. further-more, our fine-tuned models perform well when supporting observations are scarce so that n-gram indices provide no results. since this already is the dominant case for @xmath12, we can conclude that language models, fine-tuned for word search queries, are a suitable extension to context-sensitive word search engines.