Dissecting Paraphrases: The Impact of Prompt Syntax and supplementary Information on Knowledge Retrieval from Pretrained Language Models (2024)

Abstract: Pre-trained Language Models (PLMs) are known to contain various kinds of knowledge.One method to infer relational knowledge is through the use of cloze-style prompts, where a model is tasked to predict missing subjects orobjects. Typically, designing these prompts is a tedious task because small differences in syntax or semantics can have a substantial impact on knowledge retrieval performance. Simultaneously, evaluating the impact of either prompt syntax or information is challenging due to their interdependence. We designed CONPARE-LAMA – a dedicated probe, consisting of 34 million distinct prompts that facilitate comparison across minimal paraphrases. These paraphrases follow a unified meta-template enabling the controlled variation of syntax and semantics across arbitrary relations.CONPARE-LAMA enables insights into the independent impact of either syntactical form or semantic information of paraphrases on the knowledge retrieval performance of PLMs. Extensive knowledge retrieval experiments using our probe reveal that prompts following clausal syntax have several desirable properties in comparison to appositive syntax: i) they are more useful when querying PLMs with a combination of supplementary information, ii) knowledge is more consistently recalled across different combinations of supplementary information, and iii) they decrease response uncertainty when retrieving known facts. In addition, range information can boost knowledge retrieval performance more than domain information, even though domain information is more reliably helpful across syntactic forms.

## introduction symbolic knowledge bases provide relational knowledge and are widely used for tasks like question-answering. however, they rely on costly * corresponding author manual or automated, often supervised, information extraction pipelines to retrieve and represent relational knowledge. relational knowledge refers to knowledge about relations between entities, e.g. 'paris', 'capitalof', 'france', where 'capitalof' is the relation, 'paris' is the subject, and 'france' is the object. previous research on relational knowledge retrieval (rkr) from pre-trained language models (plms) @xcite has demonstrated that relational knowledge can be retrieved directly from the parameters of a plm. this finding has led to a plethora of research concerned with knowledge retrieval and reasoning capacities of plms @xcite @xcite @xcite , where rkr performance is seen as an indicator of plm's capacities to understand and reason. several benchmarks have been proposed that aim at measuring rkr performance as the ability of a plm to predict masked objects as part of cloze-style prompts @xcite @xcite . it was found that some types of supplementary information (sinf) are helpful to plms @xcite @xcite while other types deteriorate knowledge retrieval performance @xcite , and that plms primarily rely on memorization, hence, low-frequency examples are less well remembered @xcite . additionally, prior works have shown that rkr through prompts is inconsistent across different paraphrases @xcite . paraphrasing a prompt may introduce a variety of changes, including semantic ones that change the information content of the prompt, i.e., domain information ('paris is a city and is the capital of [mask]') or range information ('paris is the capital of @xcite , which is a country.'), as well as syntactic ones that merely change the form in which the same content is expressed i.e., clausal ('paris is a city and is the capital of [mask]') or an appositive syntax ('the city paris is the capital of [mask]'). previous works have not evaluated the combined impact of syntax and semantics or struggled to control all involved variables @xcite @xcite . thus, dedicated probes are required that can control the effects of different syntactic and semantic realisations on rkr. our main contributions include: controlled paraphrasing. we apply a metatemplate that streamlines prompt engineering across arbitrary relations while enabling control over syntactic form and semantic content ( §3). in terms of syntactic form, in english single-sentence prompts, information must be added either in the form of a noun phrase appositive or as an additional clause. the meta-template covers prompts with clausal (as compound, complex, or compoundcomplex) and appositive syntax. additionally, it includes placeholders for sinf as domain or range information or both. in our case, automated prompt construction is enabled by fetching domain/range information (as sinf) for given relations from established knowledge bases such as wikidata. given this method, we can compare paraphrases focused on particular combinations of semantics and syntax while controlling for variables not under investigation. probe and benchmark (conpare-lama). we introduce conpare-lama 1 , a novel controlled paraphrasing probe for lama ( §3.4), that is to the best of our knowledge the first rkr probe that facilitates extensive experiments by controlling for both syntax and semantics of prompts and is the largest rkr probe so far publicly released. we investigate a set of 60 relations, derived from the established lama-probe, ensuring wide comparability with other knowledge retrieval research. more specifically, we utilise the trex, googlere and conceptnet corpora from lama as described in @xcite . for each relation, using our meta-template, we generate prompts where sinf is added to the subject, the object, both, or neither. the sinf is realized with different syntax, resulting in a total of seven prompts per relation. varying the sinf obtained from wikidata for each such prompts results in roughly 34 million prompts (trex: 7 mio, conceptnet: 26 1 https://github.com/stephan-linzbach/ conpare-lama mio, googlere: 1 mio) unique prompts contained in conpare-lama. experiments and findings. we conduct experiments on the base versions of three well-established plms, i.e., bert @xcite , roberta @xcite and luke @xcite to advance the understanding of rkr from plms. in particular, we investigate the following research questions. [rq 1:] what is the impact of prompt syntax and information on rkr performance? [rq 2:] what is the impact of syntax on the plms' ability to efficiently combine sinf in prompts?; [rq 3:] how consistent are the answers of plms when comparing the set of correctly retrieved samples for different prompt syntax and information content?; [rq 4:] how does prompt syntax impact the response uncertainty of plms? we find that all models perform better on prompts using sinf through clausal syntax, on all investigated corpora, as compared to information added via appositives. bert achieves the best performance in this case. in addition, knowledge retrieved through prompts that rely on clausal syntax when adding sinf is more consistent given assumptions about the a priori knowledge available in the prompt. underlining these findings is our observation that the uncertainty of models for responses to known facts decreases when adding sinf through clausal syntax, which is not the case for appositive syntax. ## related work in this section, we provide the necessary background and discuss prior work. ## controlled paraphrasing we propose conpare-lama (controlled pararphrasing probe for lama) to investigate how syntax and semantics of paraphrases impact knowledge retrieval performance of plms. we hypothesize that certain syntactic forms facilitate correct interpretation, while certain semantic additions are more useful than others. as we are working with relations, we found that a natural addition of information would be domain and range type constraints. using this as sinf, we hardly change the semantics of the original task of rkr. furthermore, we restricted our work to singlesentence english prompts. we can classify all such sentences as either clausal (cf. fig. 1 , upper row) or appositive syntax (cf. fig. 1 , lower row). additionally, we are interested in the sentence's overall shape, as reflected by the classification of sentences by traditional grammars as simple (orange), compound (blue), complex (red), or compound-complex (purple) (cf. @xcite . we introduce our probe in three steps: first we control prompt syntax and semantic effects ( §3.1) by selecting prompt sentence types and sinf, then we describe the meta-template definition ( §3.2), and lastly we show the automatic template instantiation ( §3.3). relation lama ours occupation [s] is a [mask] by profession. [s] has occupation [mask]. native language the native language of [s] is [mask]. [s] natively speaks [mask]. hasproperty the [world] today is getting more and more [mask]. [s] can be described as [mask]. ## experiments we use base models of three different plms for evaluation: bert @xcite as it is a well established model and it has already been assessed using the lama probe, roberta (liu corpus grouping #relations #facts dom rng trex 1:1 2 651 6.5 5.5 n:1 23 18682 9.5 6.4 n:m 16 10190 15.6 10.1 total 41 29523 11.6 7.7 googlere death place 1 649 10 10 birth place 1 2404 7 8 birth date 1 1565 16 1 total 3 4618 11 6.3 conceptnet total 16 22739 13.2 12.6 this is done to ensure best possible performance per prompt and triple. we use the p@1 metric to measure rkr performance following @xcite . we only conduct our experiment on base models as consistency concerning paraphrases only marginally increases from base to large configurations of the plms @xcite . ## conclusion & future work in this paper, we introduced a controlled paraphrasing method and contributed the conpare-lama to advance the investigation of knowledge retrieval from plms. using conpare-lama, we examined the impact of paraphrasing on the knowledge retrieval performance of plms, studying both clausal and appositive forms in conjunction with relation-specific sinf from wikidata. our experimental findings reveal substantial variations in how plms process information based on prompt syntax. particularly, we demonstrated that knowledge consistency is enhanced when prompts utilize clausal syntax. at the same time, we observe the vulnerability of language representations in plms, especially for appositive phrases. this susceptibility may be attributed to factors such as the prevalence of clausal syntax in the training data, the semantic function, and the syntactic interaction of words with the textual encoding of relations. this interpretation aligns with the conclusion by @xcite that plms carry more knowl-edge than previously assumed and are highly sensitive to prompt paraphrasing. although earlier research suggested limited benefits from adding shallow syntactical features, our study found that intentionally applied (clausal) syntax provides increased regularity that can be exploited by contextualized word representations. therefore, we assume that harnessing these regularities through dedicated syntax-aware pre-training potentially facilitates a more robust knowledge representation. to gain more conclusive insights into the knowledge retrieval capacities of plms, experiments on models trained on non-fictional and factually correct pre-training corpora are crucial. this approach can help distinguish between false pre-training knowledge and wrong retrieval. exclusively pretrained plms, on non-fictional, peer-reviewed corpora like wikipedia or scholarly publications, or synthetic corpora @xcite , can provide promising insights. for future work, we plan to expand the template to include more diversity in syntax and knowledge. this involves applying our meta-template to additional relations derived from knowledge bases such as wikidata and incorporating more complex syntactical structures.