Exploiting Language Model Prompts Using Similarity Measures: A Case Study on the Word-in-Context Task (2022)

Abstract: As a recent development in few-shot learning, prompt-based techniques have demonstrated promising potential in a variety of natural language processing tasks. However, despite proving competitive on most tasks in the GLUE and SuperGLUE benchmarks, existing prompt-based techniques fail on the semantic distinction task of the Word-in-Context (WiC) dataset. Specifically, none of the existing few-shot approaches (including the in-context learning of GPT-3) can attain a performance that is meaningfully different from the random baseline. Trying to fill this gap, we propose a new prompting technique, based on similarity metrics, which boosts few-shot performance to the level of fully supervised methods. Our simple adaptation shows that the failure of existing prompt-based techniques in semantic distinction is due to their improper configuration, rather than lack of relevant knowledge in the representations. We also show that this approach can be effectively extended to other downstream tasks for which a single prompt is sufficient.

## introduction recently, there has been a resurgence of interest in few-shot learning, especially after the introduction of @xcite . the current dominant few-shot approach is the so-called promptbased learning which involves a simple reformulation of the target task as a cloze-style @xcite fill-in-the-blank objective. the core idea is to extract knowledge by asking the right question from the pre-trained language model (plm) using a task-specific prompting template which directs the plm to generate a textual output corresponding to a target class. this paradigm has proven its effectiveness in the few-shot setting, even for relatively smaller models, such as bert @xcite and roberta @xcite , when combined with ensembling and fine-tuning @xcite . from the practical point of view, prompt-based learning is particularly well-suited for massive models, such as gpt-3, since it does not involve parameter tuning. prompt-based techniques have shown impressive performance in the few-shot setting, especially when compared to standard fine-tuning on datasets of hundreds of data points @xcite . however, surprisingly, the word-in-context task (pilehvar and camacho-collados, 2019) -one of the tasks in the superglue benchmark @xcite -is one exception on which these methods fail to stay on par with their fine-tuned counterparts. ‡ while a simple fine-tuned bert-base model achieves around 69% accuracy on this task @xcite , gpt-3, with more than 100 times the number of parameters, performs no better than a random baseline by employing a promptbased approach @xcite . the same pattern of failure is also observed in the more recent prompt based attempts @xcite . the natural question that arises here is if the failure of few-shot techniques on wic is due to lack of relevant encoded knowledge in plms or the inefficiency of the employed prompt-based methods. two issues could be responsible for the latter case: (1) improper prompt, or (2) inefficient utilization of plm's response. to address the first issue, there have been proposals to automatically find a suitable prompt template using a search in the discrete token space @xcite or in the continuous embedding space @xcite . however, none of these have shown success on the wic task. in this work we investigate the latter issue by introducing a new configuration for prompting. given the comparison-based nature of wic, we hypothesize that conventional prompting methods fall short since they only utilize a single prompt response. hence, instead of relying on a single response, we make use of the similarity of plm's response to the combination of a pair of prompts. the experimental results on the wic dataset shows that, with only 16 instances per class, our proposed prompt-based technique can achieve comparable results to the fine-tuned models (with access to full training data of 2700+ instances per class). moreover, we show that with few adjustments, this simple approach can be effectively used for other downstream tasks. ## methodology fine-tuning on a specific task can potentially update plms on what the task is and how to solve it. assuming that plms know how to solve some tasks (to some extent), prompt-based learning focuses on the former, i.e., teaching the model what the task is, without needing to resort to large amounts of data or additional parameters. the common approach in prompt-based learning is to reformulate the task as a cloze-style question. for instance, to ask about the sentiment of a movie review, one can augment the review with a cloze question like "this movie was --.". existing methods often pick a set of one or few word predictions as a representative for each class, utilizing the language model's response in a sub-optimal manner. we propose a similarity-based method that not only better exploits the response, but also allows using multiple prompts which paves the way for comparisonbased tasks, such as wic. in what follows in this section, we describe our similarity-based prompting approach which we will refer to as sp (similarity prompting). as shown in figure 1 , sp consists of three main steps: (1) prompt generation, (2) feature extraction, and (3) prediction. given a task-specific input consisting of one or more text sequences, we first use a template function to generate a prompt-a sequence of tokens containing one [mask] token-per input sequence. for instance, in sentiment analysis, for the movie review "just give it a chance.", a valid template function would generate as output prompt: "just give it a chance. this movie was --.". the next step is feature extraction from a plm. this is done by giving the generated prompts to the plm as input and obtaining its contextualized embedding at the mask index. the third step is where sp differs from existing prompt-based approaches. here, we first obtain class-specific centroids by taking the average of the mask embeddings of our few training examples. to classify a new sample at inference time, a simple approach would be to employ a nearest centroid classifier. however, this assumes the variance of different classes to be equal in the embedding space. to alleviate the problem, we perform a class centroid-based dimension reduction (i.e. by taking the similarity to each centroid as a feature), and train a simple linear classifier. this linear model is then used at inference time to evaluate sp on test set. ## conclusion we proposed an adaptation of prompt-based learning which addresses the common failure of existing techniques on the wic dataset. in this work we showed that similarity based approach to promptbased learning is capable of achieving comparable results to purely fine-tuning based methods on word-in-context task, in which previous few-shot attempts have failed. we also showed that spearman's ranking correlation is a more robust choice of similarity measure compared to cosine similarity in this setting. we hope that our positive results inspire other prompting strategies to better exploit the encoded knowledge in plms. as future work, one interesting direction could be to perform further analysis on the behaviour of spearman's correlation compared to cosine similarity anywhere it is applicable as a similarity measure. m. marelli, s. menini, marco baroni, l. bentivogli, r. bernardi, and roberto zamparelli. 2014. a sick cure for the evaluation of compositional distributional semantic models. in lrec. mohammad taher pilehvar and jose camacho-collados. 2019. wic: the word-in-context dataset for evaluating context-sensitive meaning representations. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 1267-1273, minneapolis, minnesota. association for computational linguistics. timo schick and hinrich schütze. 2021a. exploiting cloze-questions for few-shot text classification and natural language inference. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 255-269, online. association for computational linguistics. timo schick and hinrich schütze. 2021b. it's not just size that matters: small language models are also few-shot learners. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 2339-2352, online. association for computational linguistics. taylor shin, yasaman razeghi, robert l. logan iv, eric wallace, and sameer singh. 2020. autoprompt: eliciting knowledge from language models with automatically generated prompts. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 4222-4235, online. association for computational linguistics. richard socher, alex perelygin, jean wu, jason chuang, christopher d. manning, andrew ng, and christopher potts. 2013. recursive deep models for semantic compositionality over a sentiment treebank. in proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, seattle, washington, usa. association for computational linguistics. wilson l. taylor. 1953. "cloze procedure": a new tool for measuring readability. journalism quarterly, 30(4):415-433. alex wang, yada pruksachatkun, nikita nangia, amanpreet singh, julian michael, felix hill, omer levy, and samuel r bowman. 2019. superglue: a stickier benchmark for general-purpose language understanding systems. advances in neural information processing systems, 32.