Knowledge-based Consistency Testing of Large Language Models (2024)

Abstract: In this work, we systematically expose and measure the inconsistency and knowledge gaps of Large Language Models (LLMs). Specifically, we propose an automated testing framework (called KONTEST) which leverages a knowledge graph to construct test cases. KONTEST probes and measures the inconsistencies in the LLM’s knowledge of the world via a combination of semantically-equivalent queries and test oracles (metamorphic or ontological oracle). KONTEST further mitigates knowledge gaps via a weighted LLM model ensemble. Using four state-of-the-art LLMs (Falcon, Gemini, GPT3.5, and Llama2), we show that KONTEST generates 19.2% error inducing inputs (1917 errors from 9979 test inputs). It also reveals a 16.5% knowledge gap across all tested LLMs. A mitigation method informed by KONTEST’s test suite reduces LLM knowledge gap by 32.48%. Our ablation study further shows that GPT3.5 is not suitable for knowledge-based consistency testing because it is only 60%-68% effective in knowledge construction.

## introduction large language models (llms) are being increasingly utilized in real-world applications. llms are powerful in solving many tasks, but their reliability remains a concern @xcite . this is alarming because inconsistent behaviors adversely affect critical downstream tasks and influence adoption. in this paper, we study the problem of assessing inconsistency in llm behaviors. previous works have demonstrated the prevalence and severity of inconsistent responses in llms @xcite @xcite . to address this challenge, we conceptualize and design kontest 1 -a novel test generation methodol-ogy to systematically discover consistency errors in llms and highlight their knowledge gaps. figure 1 shows examples of consistency errors and knowledge gap discovered by kon-test in gpt3.5. these are errors in gpt3.5 about the place (spatial) domain. these errors may have adverse effects in critical areas (e.g., automobiles, aeronautics) where spatial llms are deployed, e.g., navigation systems -map-box's mapgpt (mapbox), llm-geo @xcite and mapgpt @xcite , l3mvn @xcite ). kontest allows to automatically discover and expose such errors to end-users/developers. this is the first step to enable their mitigation and repair for llm improvement. kontest leverages a knowledge graph for consistency testing (see figure 2 ). it first automatically extracts a set of entities and entity relationships from the knowledge graph. this is then used to systematically generate (semantically-equivalent) yes/no queries and create test cases for the subject llm under test (sut) across various settings (e.g., atomic query vs. sequential query). the test case generation involves minimal, one-time effort for each type of entity relation considered (e.g., only two templates for 6730 test cases) and such templates can be reused for testing arbitrary llms. kontest's test suite can additionally be used to mitigate knowledge gaps by leveraging the likelihood of llm inconsistencies to construct a weighted model ensemble. to the best of our knowledge, kontest is the first systematic approach for automatically generating consistency tests for assessing llms. knowledge-based test generation provides a unique and systematic method for exploring the sut's knowledge and compute a test adequacy metric for the sut in terms of the covered entities and relations. more importantly, the responses from the sut allow kontest to construct the sut's knowledge base as a subset of the extracted 1 ) and (b) in conversation asking semantically equivalent questions in a sequence (sequential error in this paper provides an overview of kontest (section 2), and makes the following contributions: after the related works (section 6), we conclude in section 7 and discuss limitations (section 8). ## overview in this section, we outline the motivation behind our approach and present an illustrative example to demonstrate the overall process of our approach. this enables practitioners to selectively focus on the knowledge gaps highlighted by kontest for improving the llm (e.g., by fine-tuning). running example: figure 2 outlines our approach. kontest broadly comprises of three key components: 1 ⃝ knowledge graph construction ("knowledge graph" in figure 2 ), 2 ⃝ test generation ("test generation" in figure 2 ), and 3 ⃝ test oracles ("test oracles" in figure 2 ). given a set of entities (e.g., countries) and a maximum graph depth, kontest locates the given entities in the knowledge graph (e.g., wikidata knowledge graph (vrandečić and krötzsch, 2014)), and constructs the associated knowledge graph before extracting the relevant entities and relationships. given the relationships present in figure 1(d ), kontest constructs two semantically equivalent queries for each relation with the aid of a template (see figure 3 ) relating the two involved entities. the resultant queries are then fed to the sut generating responses to both atomic and sequential queries. figure 1(a ) shows two such semantically equivalent queries, each of which was part of a new conversation with gpt3.5 (aka "atomic"), while figure 1 (b) illustrates one such consistency error with inconsistent responses for semantically-equivalent queries that were within the same conversation (aka "sequential-inter"). finally, figure 1 (c) illustrates a series of queries, whose responses from gpt3.5 collectively show inconsistent behavior (aka "ontological"). specifically, the positive responses to the first two queries imply that ireland does have a kinawley. however, due to the negative response to the third query, our ontological oracle reveals a consistency error. we further note that such errors cannot be uncovered by counter-questioning llms. the relevant sub-graph of the knowledge graph for gpt3.5 is shown in figure 1(e ). this clearly illustrates the knowledge gap (i.e., kinawley↛ireland) for gpt3.5. ## evaluation setup to evaluate our approach (kontest), we pose the following research questions (rqs): knowledge graph: we rely on wikidata's knowledge graph @xcite ) and wikidata's sparql query service to extract information pertinent to the two tested domains: places and music. we choose these domains as inconsistent knowledge about places may have adversarial consequences in navigation use cases of llms (mapbox). concurrently, inconsistencies in the music domain may inaccurately capture the ownership of copyrighted materials (e.g., music albums). for the places domain, we take the list of countries ranked by (nominal) gdp per capita and select the top ten countries as our initial entity list (imf). we then iteratively build our knowledge graph by finding "administrative divisions" that are "located in the administrative territorial entity" of the parent entity. for the music domain, we take the top 200 musical acts since 2000 as our initial entity list (chart2000). we then find the set of "albums" that have the parent entity as a "performer". we also find the set of songs and singles that are "part of" the album being considered. once the graph is fully constructed, we randomly select 50 leaf nodes for each domain and extract the full path associated with the selected leaf nodes. subject programs and query construction: ## conclusion in this paper, we propose kontest where the key intuition is to distill (a subset of) facts from a knowledge graph, which are subsequently used to generate queries and formulate test cases for detecting a variety of consistency errors. our evaluation reveals realistic consistency errors across state-of-the-art llms. moreover, kontest opens opportunities to investigate llms through the lens of their knowledge gaps, which, in turn, is indicated as part of our framework. this helps designers and end users to understand and mitigate the effect of such knowledge gaps e.g., via prompt engineering or fine tuning. in future, we aim to investigate automated mitigation of consistency errors by leveraging the kontest framework. we provide our code and experimental data in the following: https://github.com/sparkssss/kontest 8 limitations and threats to validity construct validity: this relates to the metrics and measures employed in our experimental analysis. to mitigate this threat, we have employed standard testing metrics such as the number/rate of generated inputs, error-inducing inputs, (knowledge) coverage and testing time. for automatic analysis of hundreds of responses, our analysis does not handle expressive, non-binary llm responses. however, we mitigate this by employing system prompts to ensure the model provides binary responses and we discard non-binary responses (as invalid). internal validity: this refers to the threat that our implementation of kontest performs its intended knowledge graph-based test generation. we conduct several manual and automated tests, as well as inspection of sampled outcomes of kontest to ensure the correctness of our approach. our ablation study (rq4) further allows us to probe the correctness of the sub-step of kontest versus using gpt3.5. we experimentally verify that over 90% of the entities we evaluate against existed in wikidata prior to 2020. we also find that under 15% of the errors found by kontest are linked to these entities. in addition, we also note that the sut ought to answer the question in a consistent manner regardless of whether the entity existed prior to the corresponding knowledge cutoff date. external validity: the main threat to external validity of this work is the generalizability of kon-test and findings to llms, knowledge graphs, templates and entities beyond the ones used in this work. we mitigate the llm generalizability threat by employing state-of-the-art off-the-shelf, mature, open model weights llms (llama2 and falcon), as well as commercial llms (such as gpt3.5 and gemini). similarly, our entity selection and template construction may be limited to our experimental setting. however, we demonstrate the applicability of kontest by using two different domains with multiple relations and entity types. however, we note that kontest might not be easily adapted to information that cannot be encapsulated within a knowledge graph. in addition, we employ few-shot prompting to conduct our ablation study, and our findings might not generalize to other prompting techniques. finally, kontest employs wikidata, a well-maintained knowledge graph that is popularly used in both academia and industry (peters et al., 2019). llm stability and correctness: researchers have identified several stability concerns about llms @xcite , such as non-determinism, randomness, sensitivity to prompting methods, and api/model updates. to mitigate these threats we performed several actions: first, we set the temperature of all models to zero (0), when possible @xcite . secondly, we reduce the risk of model updates by limiting our testing time (to about a day each per model) and checking for news of model updates before and after testing. thirdly, we also use models with frozen weights (falcon and llama2) to reduce the non-determinism. to automatically validate model outcomes, we employ few-shot prompting, which has been shown to be effective for querying llms @xcite . we examined whether llms are comparable to kontest (rq4) using gpt3.5 since it produces the most valid responses (99.9%) (see