SPE : Symmetrical Prompt Enhancement for Fact Probing (2022)

Abstract: Pretrained language models (PLMs) have been shown to accumulate factual knowledge during pretraining (Petroni et al. 2019). Recent works probe PLMs for the extent of this knowledge through prompts either in discrete or continuous forms. However, these methods do not consider symmetry of the task: object prediction and subject prediction. In this work, we propose Symmetrical Prompt Enhancement (SPE), a continuous prompt-based method for factual probing in PLMs that leverages the symmetry of the task by constructing symmetrical prompts for subject and object prediction. Our results on a popular factual probing dataset, LAMA, show significant improvement of SPE over previous probing methods.

## introduction prompt-based learning proposes to formulate different nlp tasks into language modeling problems @xcite . it is a novel paradigm that effectively uses pretrained language models (plms) @xcite , and achieves comparable or better performance than fine-tuning @xcite . prompt-based learning has also been used for the task of factual knowledge probing in plms. in this task, the goal is to predict the (masked) object of factual tuples of type (subject, relation, object) using plms. prompting methods assume that plms gather and store factual knowledge during their pre-training, and cloze-style prompts can be used to probe plms to gauge how much knowledge they contain @xcite . the prompts are either handcrafted @xcite or automatically generated @xcite . for example, to probe plms about their knowledge of geographic location of luxembourg, a prompt can be formed by filling luxembourg in the first blank of the following template: "____ is located in ____.". an figure 1 : example of factual probing: given a subject and relation, predict the object. spe uses a fixed template to generate a prompt for predicting object given subject (green box) as well as several symmetrical prompts for predicting the subject given object candidates (yellow boxes). the final prediction is obtained using the likelihoods of the object candidates and of the given subject as obtained using the symmetrical prompts. bars represent probabilities from bert. spe is a continuous prompt method but we use natural language prompts and template here for illustration. effective prompt will probe the plm to output europe as the most likely prediction for the second blank. such methods are promising but brittle. minor changes in the template can lead to significant difference in the performance @xcite . recent works have shown that continuous prompts obtained via gradient-based learning, are more effective and robust than discrete prompts since there are less restrictions on the search space @xcite @xcite @xcite . existing methods for learning prompts do not leverage the symmetry inherent in the task's definition. for example, while luxembourg is located in europe, europe contains luxembourg. similar ideas have been used for learning prompts for relation classification @xcite and other nlp tasks @xcite @xcite . in this work, we propose symmetrical prompt enhancement (spe)-a continuous prompting method that learns prompt that incorporates the above mentioned symmetry. specifically, in addition to generating a prompt to predict the object given the subject, spe also generates an additional symmetrical prompt to predict the subject given the object. using the first prompt (see green box in fig. 1 ), spe obtains a few high-probability candidate objects like germany, france, and europe. thereafter, for each object candidate, it generates a symmetrical prompt (see yellow boxes), and obtains the likelihood of the subject, luxembourg. at the heart of spe is a prompt generation model that is trained by maximizing the joint likelihood of both the candidates as well as the subject (given the candidates). our experiments on the factual probing dataset lama @xcite show that spe achieves significant improvement over previous approaches and our analysis points to sources of this performance gain. these experiments demonstrate that like spe, probing methods should learn prompts that leverage the symmetry of the task because that can help plms in producing better answers when they are being probed for stored factual knowledge. ## symmetrical prompt enhancement the goal of factual probing via prompt generation is to output object o for given subject i and relation r by constructing a prompt p. most methods operate by assuming a template t , and generating the prompt p from t , i and r. fig. 1 shows an example of subject (luxembourg), relation (location), object (europe), template (____ is located in ____.), and the corresponding prompt (luxembourg is located in ____.). the figure shows a natural language template and prompts for readability. however, for continuous prompt methods like ours, the template is a sequences of vectors like orig ) but here we refer to them as p orig for simplicity. the model's prediction, ô, is the most likely object candidate for the b o as determined by the plm using p orig . our proposed approach, symmetrical prompt enhancement (spe), leverages the inherent symmetry of the task. specifically, in addition to learning the original prompt p orig for predicting the object given the subject, spe also generates several symmetrical prompts, p sym , for predicting the subject given the object. like p orig , p sym is also generated from t except that this time b o is filled by the (representation of) o. the prompt is used for probing the plm which outputs prediction for b i . here p(v|p) is the probability distribution of word or phrases v in plm given prompt p as input.the model is trained by optimizing a linear combination of the cross-entropy objectives of predicting the object o and the subject i: for inference, spe selects top k predictions c k : and uses each prediction c @xmath0. finally, the model's prediction ô is: in practice, l and p plm are normalized by input length to account for inputs with multiple tokens. ## implementation details we conduct experiments on the fact retrieval part of lama dataset @xcite , which consists of fact triples with single-token objects from 41 relations in wikidata @xcite . we use the training set extended by @xcite . we choose masked language models bert @xcite and roberta @xcite as plms, which are fixed during training to serve as static knowledge bases. for implementation, we use plms in huggingface library of transformers @xcite . we follow @xcite for designing templates and the prompt generation component of our model. in particular, we use bilstm @xcite and use the following generic and relation-agnostic format for template, t : the model and the template are randomly initialized. for i with multiple tokens, we mask them one token at a time to generate p sym , and use the average of pseudo likelihoods from all p sym s to represent log p(v ′ = i|p sym ). in practice, we find that masking one token at a time is better than masking the entire phrase at once, and averaging the pseudolikelihood has better performance. the training batch size is 8. we set k to be 15 during inference, and λ to be 0.8 based on our experiments on the development set. the results are evaluated by accuracy at top 1 (p@1) and top 10 (p@10) predictions, and mean reciprocal rank (mrr) as in @xcite . appendix a includes more setup details and discussion on choice of λ. ## results we compare our results with both discrete and continuous prompt methods. discrete prompt methods include prompts from manually designed templates @xcite ; lpaqa @xcite , which uses text mining based prompts; and au-toprompt @xcite , which uses discrete lexicalized trigger tokens for prompt generation. continuous prompt methods include p-tuning @xcite , which uses a neural network to generate prompts; optiprompt @xcite , which uses manually initialized prompts; and soft-prompt @xcite , which ensembles multiple prompts initialized with mined templates. quantitative results: effect of candidates pool size: ## limitations we note that spe may not help if the correct objects are broad concepts (e.g. "mathematics" vs "algebra", "river" vs "tributary", "fifa" vs "uefa"). typical relations with such objects include p279 subclass of, p361 part of and p463 member of. the top 5 predictions by spe (and also p-tuning) for the subclass of relation are shown in also, in general, spe can get affected by error propagation because of its two-step inference process that first predicts object candidates and then ranks them. though the proposed symmetrical prompt method improves knowledge probing, the utility of the technique in other nlp tasks is not yet investigated. besides, the experiments are only conducted for masked language models but there has been recent progress in other types of language models which are not explored in the paper. lastly, the proposed method requires additional computational cost compared the baselines. ## conclusion this work introduces symmetrical prompt enhancement (spe) -a continuous prompt-learning method for factual probing of plms by learning prompts that utilize the inherent symmetry of the task. our experiments show that spe outperforms existing sota methods thereby helping us know more about how much knowledge is stored in a plm. future work could explore this idea of using task symmetry for other nlp tasks. 1 1 code is available here.