Lexical Substitution as Causal Language Modeling (2024)

Abstract: Causal language models such as the GPT series have achieved significant success across various domains. However, their application to the lexical substitution task (LST) remains largely unexplored due to inherent limitations in autoregressive decoding. Our work is motivated by our observation that existing LST approaches tend to suffer from a misalignment between the pre-training objectives of the language models that they employ, and their subsequent fine-tuning and application for substitute generation. We introduce PromptSub, the first system to use causal language modeling (CLM) for LST. Through prompt-aware fine-tuning, PromptSub not only enriches the given context with additional knowledge, but also leverages the unidirectional nature of autoregressive decoding. PromptSub consistently outperforms GeneSis, the best previously published supervised LST method. Further analysis demonstrates the potential of PromptSub to further benefit from increased model capacity, expanded data resources, and retrieval of external knowledge. By framing LST within the paradigm of CLM, our approach indicates the versatility of general CLM-based systems, such as ChatGPT, in catering to specialized tasks, including LST.

## introduction lexical substitution task (lst) is to identify appropriate replacements for a designated target word in context while maintaining the contextual meaning and coherence of the text @xcite . for example, given the sentence "let me begin again", an lst system would be expected to provide words such as start or commence as substitutes for begin. lst is an important task due to its numerous applications, including word sense disambiguation @xcite , word sense induction @xcite , 1 our code and data are publicly available on github: https://github.com/shininglab/promptsub encoder-decoder the "begin" in the sentence "let me begin again." can be substituted with " start, commence, open, ... ## related work conventional lst techniques predominantly capitalize on external knowledge bases @xcite @xcite and learned word embeddings to identify and rank potential substitution candidates based on predefined metrics @xcite . these methods often depend heavily on external resources like wordnet @xcite , with additional processes such as the manual ranking and rule construction often required to optimize outcomes. recognizing these limitations, recent initiatives have emerged to har-ness the advantages of plms. prior work indicates that contextualized representations obtained from plms can be applied to lst by incorporating context-based scores @xcite and decontextualized embeddings @xcite . in an effort to augment plms with knowledge derived from lexical resources, @xcite proposed involving gloss matching in pre-training. @xcite advocate for the incorporation of structured knowledge from lexical databases. on the one hand, certain of these approaches utilize plms primarily as feature extractors. thus, the complete potential of plms remains untapped due to the disconnect between their pre-training objectives and subsequent applications. on the other hand, to align with pre-training, others @xcite estimate the probability distribution of potential replacements through masked language modeling @xcite . this inclination towards mlms, as opposed to clms, has led to the over-representation of encoder-only plms, leaving the application of decoder-only architectures largely unexplored. similarly, while prior work has explored the ideas of enriching lst inputs with target words @xcite and semantic knowledge @xcite , how to inject such knowledge into plms remains an open question. this issue is particularly true within the prevailing trend of unsupervised methods that exclude the fine-tuning stage. supervised approaches to lst, such as those by @xcite , were initially limited by data scarcity until the advent of genesis @xcite and parals @xcite . genesis adopts a sequence-to-sequence model, generating substitutes given the context and marked target word. by concatenating multiple datasets, fine-tuning a plm specifically for lst was made viable, achieving strong results despite the scarcity of annotated data in the domain. par-als produces substitutes through a paraphraser, utilizing a heuristics-based decoding strategy. this facilitates fine-tuning plms on paraphrase data, which is available in relatively large quantities. however, in both genesis and parals, a discernible gap persists between the pre-training of plms and their subsequent fine-tuning. furthermore, they are both rooted in an encoder-decoder framework @xcite , depending on external resources, and require post-processing steps at position {position} in the sentence, "{context}", the {pos} "{context[position]}", derived from the lemma "{target}", can be substituted with "{substitute}".' that involve adjustable heuristics and thresholds. this raises a pivotal question: does lst have to be approached in a two-step manner, where substitutions are first generated and then reranked using manually designed scores? or, is it possible to create a single-step, end-to-end, generative solution, that also sidesteps the need for external resources and manually-crafted heuristics? in presenting promptsub, we argue for the latter: a firstof-its-kind single-step approach to generating substitutions via a decoder-only language model. ## methodology in this section, we formally define lst and clm, outline our sampling strategy, and detail our prompt engineering techniques. ## experiments in this section, we describe our empirical comparison of promptsub to the top-performing previously published lst methods. after brief descriptions of the benchmark datasets (section 4.1) and our experimental setup (section 4.2), we proceed with a comparative analysis of promptsub and gene-sis, two supervised generative approaches (section 4.3). we then extend this experiment to include more methods and test of the full suite of datasets (section 4.4). unless stated otherwise, we apply promptsub with freqsub sampling and augp for corpus generation, as these settings gave the best performance in our development experiments. further sensitivity analysis will be presented in section 5. ## analysis we now present a sensitivity analysis of our method. we measure the impact of various aspects of our experimental setup, including training size, model capacity, sampling strategy, prompt engineering, and external knowledge. we maintain the same experimental setup, modifying one aspect of our methods to observe the resulting performance change on ls21. the random seed is held constant at 0. training size regarding training data size, we have introduced promptsub+, a variant of prompt-sub, that includes the validation set in its training data. across all experiments, promptsub+ consistently outperforms promptsub on the test data, demonstrating its ability to benefit from additional data. this finding underscores the challenge posed by limited data resources in most existing lst benchmarks, which affects the broader application of promptsub and other supervised methods @xcite . ## conclusion we have presented promptsub, a framework for recasting lst as clm, which overcomes the limitations of earlier methods by bridging the gap between pre-training and fine-tuning. promptsub is flexible and extensible: it allows for variations in the prompt template, facilitating the inclusion of additional knowledge; further analysis reveals the potential for further improvement through scaling up model capacity and data size, applying prompt engineering, and retrieving external knowledge via rag. our extensive experiments found that promptsub consistently outperforms the previous generative approach, genesis, on ls07, and establishes a new overall state of the art. as the first attempt to fine-tune decoder-only plms for lst, our work highlights the broader applicability of plms to semantic tasks.