Assessing Factual Reliability of Large Language Model Knowledge (2024)

Abstract: The factual knowledge of LLMs is typically evaluated using accuracy, yet this metric does not capture the vulnerability of LLMs to hallucination-inducing factors like prompt and context variability. How do we evaluate the capabilities of LLMs to consistently produce factually correct answers? In this paper, we propose MOdel kNowledge relIabiliTy scORe (MONITOR), a novel metric designed to directly measure LLMs’ factual reliability. MONITOR is designed to compute the distance between the probability distributions of a valid output and its counterparts produced by the same LLM probing the same fact using different styles of prompts and contexts. Experiments on a comprehensive range of 12 LLMs demonstrate the effectiveness of MONITOR in evaluating the factual reliability of LLMs while maintaining a low computational overhead. In addition, we release the FKTC (Factual Knowledge Test Corpus) to foster research along this line https://github.com/Vicky-Wil/MONITOR.

## introduction recently, large pre-trained language models (llms) have been used as de facto storage for factual knowledge @xcite . however, applying llms to real-world scenarios inevitably leads to language generation deviating from known facts (aka "factual hallucination" @xcite ) due to multiple causes. for example, @xcite argued that the performance of an llm is over-estimated due to biased prompts overfitting datasets (also referred to as the framing effect in @xcite ) and in-context information leakage. given the variability of llms' performance under different prompts and contexts, it becomes evident that relying solely on accuracy as an evaluation metric is insufficient. we also need to gauge 1 https://github.com/vicky-wil/monitor how robust llms are to variations in prompting. in figure 1 we show examples of factual probes where either the framing of the prompt, or the context to the prompt, is varied, leading to the issue of "accuracy instability". prompt framing effect: an llm generates different predictions depending on how prompts are framed. predictions are associated with prompts instead of factual knowledge learned in llms. as shown in figure 1 (a), for a fact represented in a triplet ⟨ cunter, is located in, switzerland ⟩ , the generated predictions for re-framed prompts "which country is cunter situated?" and "cunter is located in switzerland. true or false?" are non-factual. effect of in-context interference: an llm leverages in-context information during its decoding stage, but this information may negatively affect an llm's prediction during knowledge probing. as shown in figure 1 (b), for the same fact, when presented with a context "england." concatenated with the prompting question "which country is the location of cunter?", an llm generates a non-factual prediction "england". how do we assess the reliability of factual knowledge of llms under the effects of these hallucination-inducing factors? investigations into the behaviors of language models during knowl-edge probing @xcite @xcite have mainly used metrics like precision and accuracy to quantify errors under a specified factor like prompt framing @xcite or mis-primed information @xcite . despite the insights gained by showing the instability of llms during knowledge probing, these studies are subject to two limitations: no exploration of uncertainty. metrics like top-one accuracy may capture the ordering of predictions in the output space, but they lack the resolution to reflect on the degree of factual knowledge being learned by llms. figure 2 depicts an example where two llms (models a and b) may produce the same result even though their output probabilities vary. by equating the performance of model a with that of model b, one introduces a level of approximation in representation, which can be regarded as a source of uncertainty. in this paper, we directly use the output probabilities and construct a high-resolution metric to perform knowledge assessment. limited scope. previous works focus on understanding the effect of variability of a specific type. we design experiments to investigate the combined effects of multiple causes of accuracy instability: prompt framing and in-context interference during knowledge assessment. in addition, few studies have experimented on llms with billions of parameters. in contrast, we investigate the knowledge reliability of 12 freely downloadable llms with a range of parameter sizes and origins (with and without instruction fine-tuning). 2 in this paper, we propose a novel distancebased approach model knowledge reliability score (monitor) which captures the deviation of output probability distributions under contexts of prompting variance, interference from mispriming @xcite and positively-primed prompts. we perform experiments on a comprehensive set of knowledge probing tasks and investigate the effectiveness of monitor in assessing llms' factual reliability. through experiments with a large variety of different facts, we show that a lower-monitor llm is less likely to suffer from "accuracy instability" issue. computing mon-itor takes only one-third gpu hours of those consumed by a comprehensive accuracy reliability study, making monitor a low-cost metric for assessing factual knowledge reliability of llms. ## llms in hallucination in this section, we investigate llms' accuracy under the influence of various hallucination-inducing causes mentioned above. we design five formats of prompts to demonstrate two categories of hallucination-inducing causes during knowledge probing ( ## methodology in this section, we introduce monitor, a distance-based score, to assess how the factual knowledge of llms is affected by the previously mentioned prompt framing and in-context interference. firstly, we introduce a new variable (i) to represent hallucination-inducing in-context information into the initial knowledge representation triplet ⟨ subject, relation, object ⟩. the newly formed knowledge representation quadruple can be expressed as ⟨ s, r, o, i ⟩. the information i can be further categorized into two variables: we use a factual object entity to implement a positive information i + ; and the negative information i -represents interference when predicting o. for example, "england" is considered as an i -when acting as a noisy condition to negatively affect an llm in predicting a desirable outcome ⟨ switzerland ⟩ for a fact ⟨ cunter, is located in, switzerland ⟩. corresponding to an object, p (o|s, r, i) is the probability of the model generating the object o with the conditions of subject s, prompt framing expression r, and the in-context information i. to quantify the effect of i on llms, we establish "anchor" as a reference point, which is the gold answer with its probability in the output space. a "primary anchor" (shown as the red font "switzerland 0.9117" in figure 3 ) is defined as an enforcedaccurate answer with its probability produced by an llm in response to a knowledge probe. a primary anchor is produced by prompting an llm with a qa template prefixed with positive information i + (i.e. template (4) in monitor consists of two distance-based measurement components: prompt-framing degree (pfd) and interference-relevance degree (ird). ## conclusion in this paper, we show that large language models are subject to the influence of various hallucinationinducing causes. we propose a novel distancebased metric, directly computing the output probabilities and their changes to address "accuracy instability" caused by the prompt framing effect and in-context interference. a comprehensive set of experiments demonstrates that the proposed mon-itor is a high-resolution economic method suitable for evaluating the reliability of large language model knowledge. monitor can be used in conjunction with an end-to-end metric (i.e., accuracy) as part of a multi-dimensional approach to llm knowledge evaluation. the constructed fktc, consisting of 210,171 question answering prompts on 20 fact datasets, will be made available to the public to foster research along this line.