R ead P rompt: A Readable Prompting Method for Reliable Knowledge Probing (2023)

Abstract: Knowledge probing is a task to assess the knowledge encoded within pre-trained language models (PLMs) by having the PLM complete prompts such as “Italy is located in __,”. The model’s prediction precision serves as a lower bound for the amount of knowledge it contains. Subsequent works explore training a series of vectors as prompts to guide PLMs towards more accurate predictions. However, these methods compromise the readability of the prompts. We cannot directly understand these prompts from their literal meaning, making it difficult to verify whether they are correct. Consequently, the credibility of probing results derived from these prompts is diminished. To address the issue, we propose a novel method called ReadPrompt, which aims to identify meaningful sentences to serve as prompts. Experiments show that ReadPrompt achieves state-of-the-art performance on the current knowledge probing benchmark. Moreover, since the prompt is readable, we discovered a misalignment between constructed prompts and knowledge, which is also present in current prompting methods verified by an attack experiment. We claim that the probing outcomes of the current prompting methods are unreliable that overestimate the knowledge contained within PLMs.

## introduction pre-trained language models (plms), such as bert @xcite , contain factual knowledge from their training data. recent studies explore the possibility of replacing traditional knowledge bases (kbs) with plms @xcite . however, it is unclear if a plm pre-trained on a corpus contains sufficient knowledge compared to a kb. therefore, understanding and probing the knowledge stored in plms is crucial. one approach is to have the plm complete prompts about factual knowledge @xcite @xcite . for instance, given the fact "(italy, located-in, europe)," a prompt can be constructed by masking the object, like "italy is located in @xcite ." if the plm fills in "europe" for the mask, it is considered to know the fact. the precision of plms on such factual questions serves as a lower bound for the amount of knowledge they encode. as the plm is sensitive to the context, recent studies investigate ways to construct better prompts that can trigger the plm to answer more factual questions. @xcite proposed a searchbased method that automatically selects discrete words from the vocabulary to formulate prompts. some works further believed that prompts are not necessarily composed of actual words and proposed a tuning-based method to optimize a sequence of vectors as prompts, i.e., continuous prompts @xcite @xcite . although these prompts can encourage plms to answer more factual questions, compared to manual prompts, they sacrifice the readability of the prompts themselves, which reduces the reliability of their probing results. due to the entire prompt being replaced with disjoint tokens or a series of vectors, it is hard to discern the literal meaning of these prompts. for instance, in figure 1 , given "italy national art petroleum outside" as a prompt, the plm outputs the matching object "europe." nevertheless, it is difficult to claim that the plm knows the fact (italy, located-in, europe) based on this prompt. the same issue arises with the softprompt @xcite illustrated in figure 1 , which uses vectors as prompts. consequently, it is unreliable to judge whether a plm knows a particular fact solely based on whether its output matches the object. we believe that ensuring the consistency between the prompt and the fact is a prerequisite for knowledge probing, and readability is fundamental for examining the prompt's meaning. to this end, we aim to construct prompts with good readability for knowledge probing that allows humans to determine whether the combination of the plm's output and the prompt constitutes a fact. referring to molnar (2020), we define readability as the degree to which it is understood by humans. however, assessing readability is challenging due to its subjective nature. to maintain fairness, we use the perplexity score @xcite of the plm itself as a metric. intuitively, lower scores indicate that the prompt is closer to natural language and easier to understand @xcite . to enhance the readability of prompts, we propose readprompt, a method for searching sentences to construct prompts. following the idea of @xcite , we use first-order estimation to predict the change in classification loss for each word in the vocabulary when selected into the prompt to generate a golden answer. we then select the word that most reduces the loss into the prompt. moreover, we introduce the perplexity of the plm as a constraint, necessitating that the selected word reduces the overall perplexity of the prompt, ensuring its consistency with the current context and maintaining the readability of the entire prompt. we iteratively update each word in the prompt, repeating the process until all selected words remain unchanged, ultimately obtaining the final prompt. the experimental results demonstrate that readprompt achieves state-of-the-art performance on the current knowledge probing benchmark. furthermore, readprompt significantly improves the readability of prompts compared to other automated prompting methods. upon examining the constructed prompts, we found that prompts contradicting facts surprisingly led the plm to answer more questions correctly. this indicates that plms have confusion in understanding certain knowledge, and the knowledge detected by these incorrect prompts should not be considered as knowledge mastered by plms. we further verified through attack experiments that the probing results of current prompting methods also contain knowledge that plms are confused about, leading to an overestimation of the plms' knowledge volume. prompting methods can be categorized into two types based on their composition: discrete prompts and continuous prompts. discrete prompts are formulated by real words. for example, lpaqa @xcite mined prompts from wikipedia. several works proposed a gradient-based method to search disjoint words as prompts @xcite . while continuous prompt is composed of trainable vectors sampled from word embedding space @xcite @xcite . in that way, the tokens in the prompt are not necessarily related to actual words. on the one hand, it brings higher degrees of freedom for prompts, which leads to better results. on the other hand, those prompts become completely unknown to humans @xcite . ## discussion: stochastic parrot or intelligence? the emergence of large language models has attracted widespread attention. many studies question whether large language models are truly intelligent or merely a "stochastic parrot" (chen et al., 2022) that assemble sequences of linguistic forms based on probabilistic information, without any reference to meaning @xcite . our experiments with readprompt reveal confusion in some knowledge triplets within the plm. interestingly, using prompts with opposite meanings can help plms achieve better results on current knowledge probing benchmarks. attack experiments confirm that some plms or llms only capture high-frequency co-occurrence relationships between parts of the subject and object, resembling a stochastic parrot. in contrast, gpt-3.5 and gpt-4 successfully resist attack samples, demonstrating sufficient intelligence in the attack experiment. although some studies @xcite @xcite claim that moderately-sized (e.g., 7b) large language model can achieve results comparable to gpt-4, our knowledge probing experiments reveal that these models fall short in terms of the amount of knowledge they contain (lower bound), reliability, and their adherence to human prompts. ## conclusion in this work, we introduce readprompt, a novel method to construct prompts for the knowledge probing task. on one hand, readprompt can guide plms to answer more factual questions correctly. on the other hand, readprompt is readable, providing a way for people to verify its correctness. our case studies reveal that plms occasionally confuse facts, which should not be regarded as their inherent knowledge. furthermore, we demonstrate through attack experiments that existing prompting methods intensify plms' confusion of facts, leading to higher but unreliable results. ultimately, we argue that for probing tasks, a reliable outcome is of greater importance than a merely "better" one.