One-Shot and Few-Shot Exemplification Modeling (2023)

Abstract: Exemplification modeling is a task where the goal is to produce a viable example sentence that uses a target word with a target definition. The task is non-trivial for polysemous words, and previous works have only explored settings where ample labeled training data is available. In this paper, we demonstrate that exemplification modeling can be performed without a large labeled training corpus by either changing the format of the task (one-shot) or prompting large language models (few-shot), and ablate key components of our proposed one-shot and few-shot systems. We provide extensive automatic and human evaluations of model performance and find that our proposed one-shot and few-shot approaches perform similarly to a fully supervised baseline. We compare and contrast each method in terms of labeled training dataset size, performance, and model size, and find that each technique has at least one tradeoff that another approach does not.

## introduction many words can have several different meanings depending on the context in which they are used. given this ambiguity, it is often necessary to resolve the meaning of words in context for the purpose of understanding the semantics of a sentence or learning how to use new words properly. this task is called word sense disambiguation (wsd) and has been widely studied @xcite @xcite . recently, two related generative tasks have spawned from wsd, namely definition modeling (dm) @xcite and exemplification modeling (em) @xcite . dm is similar to wsd, except that instead of choosing a word sense from a predefined inventory (classification), a definition is generated for a given word in context. em can be seen as the inverse task to dm, where an example sentence is generated given a target word and definition. we provide an example input/output pair below: input output cool: composure under strain she kept her cool during the interview. in this paper, we examine the ability of different systems to perform em in one-shot or few-shot settings. one-shot em. to perform em in a one-shot fashion (oneem), we replace the definition with an example sentence using the target word with the intended sense. in this setting, the semantics of the target word are inferred from context in a sentence. we will show that the oneem format of the task has new applications not possible for em (see section 1.1) and can be trained in a self-supervised fashion, requiring only raw text with no word sense labels. an example input/output pair for the oneem task is provided below: input output cool: drinking a cool beverage is refreshing. he felt the cool breeze in his hair. few-shot em. we explore the ability of large language models (llm) to perform em in a few-shot setting (fewem). the prompt is created from examples formatted as in @xcite , where the fewem task is performed by completing the last example (see figure 3 ). ## one-shot exemplification modeling one-shot exemplification modeling (oneem) relies on an example sentence for the semantic signal of the target word sense instead of the gloss. as we will show in the following subsections, oneem can be trained in a self-supervised fashion, requiring only raw sentences as training data. selfsupervised training is possible due to the empirical observation that bidirectional language models (lm) like bert @xcite and roberta @xcite create high-quality representations of words that indicate meaning in context from masked language modeling (mlm) pretraining alone. the one-shot training approach discussed here was proposed previously by @xcite but is significantly improved in this paper via better disentanglement of information. one-shot training. to train a oneem system in a self-supervised fashion, we use a neural autoencoder and call our approach sense2sentence (s2s). given a sentence s, we want to reconstruct it by conditioning on a target word w and a latent vector representation of the intended target word meaning. we call this vector the latent sense representation (lsr) and denote it as l. we create one-shot training data by selecting w uniformly from s and then extracting l from the lsr module, i.e. @xmath0, w). we then maximize the following probability mass function: we model the distribution using a pretrained bart model @xcite and train using the crossentropy loss with teacher forcing. see figure 1 . latent sense representation. previous work has shown that word embeddings at the output of a bidirectional transformer-based lm contain information related to the meaning of a given word in context @xcite , so we use these contextual word embeddings (cwe) as the lsr. concretely, given a target word w, we take the average of all subword embeddings making up w from sentence s. ## disentanglement of word meaning while cwes contain information relevant to a given word's meaning in context, cwes also contain information about the sentence in which the word is placed. when using cwes extracted from pretrained models like bert @xcite as the lsr during training of our proposed oneem system, we find empirically that there is enough information in the cwes to make reconstruction of the input sentence easy. 1 this results in the model learning to copy the input sentence and makes it incapable of generating new and diverse examples. therefore, disentanglement of the target word meaning from other information in the sentence is critical for the proper functioning of our proposed autoencoding system. the two disentanglement methods we explore in this paper are: 1) finetuning of bert on wsd using the bi-encoder model (bem) framework from blevins and zettlemoyer (2020) or 2) vector quantization via a synset codebook. ## few-shot exemplification modeling given the recent success of llms across a variety of tasks @xcite @xcite @xcite , we explore the ability of llms to perform em in a few-shot setting (fewem). we provide several formatted examples as a prompt to bloom or bloomz 2 (scao et al., 2022) and perform few-shot inference by appending an incomplete example and continuing generation using nucleus sampling (see figure 3 ). ## baseline both previous works on exemplification modeling make use of the exmaker model @xcite . we reimplement ex-maker from scratch, where we provide the target word (lemma) and definition as input and train the model to maximize the probability of a given labeled example sentence. for direct comparison with our proposed oneem system, we use the same bart model for the exmaker baseline. additionally, we train a sense-agnostic version of the baseline where we exclude the definition and only provide the target word as input. we call this the vanilla version (exmaker v ) and use it to provide a lower bound on em performance for polysemous words since it cannot take the target definition into account. ## data we focus our evaluation on polysemous words, since em is a trivial task for monosemous words. we discuss the training, validation and test data for em, oneem, and fewem methods below. training. for our proposed oneem system, we do not require labels and thus train using the 74m raw sentences from bookcorpus @xcite . for exmaker, we use oxford dictionary foot_1 , which contains 1.4m labeled examples. validation. due to the relatively small size of oxford dictionary, we must validate the exmaker baseline to avoid overfitting. we validate using cross-entropy loss on 142k held-out examples. test set. we create a test set 4 of 167 word senses by hand using polysemous words with two or more distinct meanings (homographs). the goal of manual construction of the dataset is to insure word meanings for the test senses are clearly separate 5 and that gold example sentences are high-quality and easy to read. for each example, we provide the word, lemma, word sense (wordnet 3.0), part-ofspeech, definition, and an example sentence. we use the example sentence as input for the oneem models and as gold data in our evaluations. ## experimental setup lemmatization of target word. for em and fewem, we lemmatize the target word to allow word form flexibility in a given generated sentence. for oneem, we do not lemmatize due to the main target application for the task, which is a language learning tool that further clarifies the meaning of a word by generating more examples (see section 1.1). by not lemmatizing the target word for oneem, we force the generated sentence to produce the target word in the same form as is present in the one-shot example. the main motivation for this is that some words are only homographs in certain forms. for example, the word "saw" is a homograph, because it can refer to the past tense of the verb "to see" and the noun meaning "instrument used to cut wood." if we were to lemmatize ("saw"→"see"), we could produce an incorrect example depending on which meaning was used in the one-shot input example. training and validation hyperparameters. we train for 500k steps and set batch size to 64 for exmaker and sense2sentence (s2s) models. we validate exmaker every 25k steps and use the 500k checkpoint for s2s. generation. we use @xmath1.5 for nucleus sampling for all methods (exmaker, s2s, llm). for definition modeling (dm) verification, we use three beams during beam search due to gpu memory constraints (see appendix c). ## evaluations we want to evaluate three aspects of generated example sentences: 1) how well the target word takes on its intended meaning (semantic match), 2) fluency of text, 3) diversity of generated examples from one another. we measure diversity automatically and use both automatic and human evaluations to measure semantic match and fluency. for all oneem approaches, we compute pearson's correlation coefficient between input and output sentence lengths to provide insight into how well each lsr disentanglement method can remove information unrelated to target word meaning. ## results numerical results for automatic and human evaluations are given in tables 1 and 2 , respectively. we discuss several key takeaways below. self-supervised performance (oneem). compared to gold examples, those generated using the self-supervised approach are of high quality based on both automatic and human evaluations. the fully-supervised em baseline, exmaker, only slightly outperforms the best self-supervised method (row 1 in tables 1 and 2 ) for semantic match and fluency, and achieves almost identical diversity values. we find that codebook size only has a small effect on performance, where we notice a slight word 1: fan definition: an ardent follower and admirer pos: noun one-shot example: i've been a fan of hemingway's writing for years. s2s 150k (mwic) i wasn't a big fan of the slow motion movies, but i had to admit, i liked the latter. s2s (bem) but she is a fan of the book of juliet, and she likes to read. exmaker he is a fan of the arts, and a huge supporter of the arts in general. bloom 7b the beatles were a huge fan of elvis presley. word 2: fan definition: a device for creating a current of air by movement of a surface or surfaces pos: noun one-shot example: she used a large fan during the summer to keep her shop cool. s2s 150k (mwic) the voice in the room seemed to be some kind of a gas-powered fan. s2s (bem) the house was fitted with a fan and made the people to love the air. exmaker a pilot fan is used to provide air for the exhaust system. bloom 7b a fan is a device that creates a current of air by moving a surface or surfaces. word 3: lie definition: assume a reclining position pos: verb one-shot example: i'm tired and need to go lie down. s2s 150k (mwic) no, you don't have to lie down, said joshua. s2s (bem) i just need to go and stretch out to lie down. exmaker we were both watching the kids lie down on the couch. bloom 7b the boy lay down on the bed and fell asleep. word 4: lie definition: tell an untruth; pretend with intent to deceive pos: verb one-shot example: i need to know what's really going on, so don't lie to me. s2s 150k (mwic) she had a feeling that he wouldn't lie to her. s2s (bem) i need to be able to hear you, so do it, then try to lie to me. exmaker he was willing to lie and deceive, but was not prepared to give his life for the sins of the devil. bloom 7b the boy lies that he has a friend in the next room. ## generation examples we provide some generated examples drawn randomly from our human evaluations in ## discussion the approaches to em, oneem, and fewem discussed in this paper each have benefits and drawbacks, which are summarized in figure 2 . overall, we find there is a tradeoff between amount of labeled training data and model size needed for good performance. based on human evaluations, we see the best performance from the fewem setting, which uses a large language model (llm). similar performance can be achieved using a much smaller model (exmaker), but requires a large amount of labeled training data. finally, we can achieve similar performance to exmaker when using a small model without a large corpus of labeled training data (self-supervised training), but require at least one example at inference time in order to generate more examples (oneem). ## conclusions in this paper, we proposed two variations of the exemplification modeling (em) task, namely oneshot em (oneem) and few-shot em (fewem). we discussed novel applications and described a self-supervised solution for the oneem task, and ablated several configurations (codebook size, cwe model) to better understand performance. we also performed extensive experiments using llms to solve the fewem task, and found that a pretrained llm can perform fewem extremely well when prompted with only a few examples. we additionally studied the use of few-shot definition modeling (dm) to verify fewem examples and found that such an approach leads to improved performance. in addition to being useful for downstream applications like data augmentation, icall vocabulary learning systems, and dialog systems, the family of em tasks discussed in this paper provide evidence that pretrained language models have a strong understanding of differences between various word senses. our experiments demonstrate that target word meaning can be inferred from either a definition (em, fewem) or an in-context example (oneem), providing flexibility for example generation depending on downstream task constraints.