C omparison QA : Evaluating Factuality Robustness of LLM s Through Knowledge Frequency Control and Uncertainty (2025)

Abstract: The rapid development of LLMs has sparked extensive research into their factual knowledge. Current works find that LLMs fall short on questions around low-frequency entities. However, such proofs are unreliable since the questions can differ not only in entity frequency but also in difficulty themselves. So we introduce **ComparisonQA** benchmark, containing **283K** abstract questions, each instantiated by a pair of high-frequency and low-frequency entities. It ensures a controllable comparison to study the role of knowledge frequency in the performance of LLMs. Because the difference between such a pair is only the entity with different frequencies. In addition, we use both correctness and uncertainty to develop a two-round method to evaluate LLMs’ knowledge robustness. It aims to avoid possible semantic shortcuts which is a serious problem of current QA study. Experiments reveal that LLMs, including GPT-4o, exhibit particularly low robustness regarding low-frequency knowledge. Besides, we find that uncertainty can be used to effectively identify high-quality and shortcut-free questions while maintaining the data size. Based on this, we propose an automatic method to select such questions to form a subset called **ComparisonQA-Hard**, containing only hard low-frequency questions.

## introduction the rapid advancement of large language models (llms) has promoted a lot of study on their factual knowledge and reasoning ability @xcite . @xcite and @xcite compare llms' performance on questions around entities with different frequencies. llms are found struggle to handle tail knowledge. however, the questions they study are different and can vary in figure 1 : an example from comparisonqa difficulty levels, not only in entity frequencies. as @xcite emphasized, we need a more "controlled, synthetic experiment that confirms the weakness of llms" nowadays. therefore, existing comparisons are not enough since they can not guarantee that the low frequency is the only cause of llms' poor performance. to tackle the issue, we introduce our compar-isonqa benchmark. pairs of high-frequency and low-frequency entities share the same question, as shown in figure 1 . the shared abstract question, with a hypernym to represent the two specific entities, guarantees that the difference between such a pair is only the entity. this allows for a controllable comparison between high-frequency and lowfrequency entities. comparisonqa is a large scale dataset containing 283k such question pairs. it is generated through an automatic pipeline base on raw knowledge base, ensuring both diversity and scalability. through this benchmark, we can completely compare llm's performance on different knowledge frequencies. for a more robust and accurate evaluation, we use the multiple-choice format @xcite in our benchmark. but semantic shortcuts @xcite between questions and options may help llms to guess the answer, which is also a common but severe problem recently. thus, we further design a two-round method using both correctness and uncertainty to evaluate llms' knowledge. during experiments, we found that llms have very poor robustness, especially on low-frequency knowledge, where even the powerful gpt-4o also performs badly. recent benchmarks, like simpleqa @xcite , ensure their difficulty by collecting questions adversarially against llms' responses. but relying only on accuracy, they ignore the quality of their questions, and the difficulty is closely related to the models they use. fortunately, our experiments find that uncertainty is also an effective tool in selecting both high-quality and shortcut-free questions while maintaining the benchmark size. combining accuracy and uncertainty, we propose a new flexible method to select our subset called comparisonqa-hard for future study. it contains 81k difficult low-frequency questions with high-quality and no semantic shortcuts. in summary, we have three main contributions: (1) @xcite we introduce comparisonqa benchmark, where a pair of entities share the same abstract question. it enables a more controllable and reasonable proof that llms perform worse when the required knowledge is less frequent. ( §3) (2) @xcite we design a two-round method using correctness and uncertainty to evaluate llms' robust knowledge. @xcite llms can not stand such a test, especially on low-frequency knowledge, where even gpt-4o performs badly. ( §4) (3) @xcite uncertainty is more helpful to find questions with high quality. @xcite through this, we select comparisonqa-hard benchmark containing only hard and low-frequency questions of high quality and no shortcuts. ( §5) 2 related works ## comparisonqa to ensure more fair and controllable comparisons between llms' performance on high-frequency and low-frequency factual knowledge, we propose a new benchmark called comparisonqa. hypernym footballer season racer game brand jamie stauffer relationship count: 66 entity ......alexis vuillermoz is a french road bicycle racer, who rides for uci proteam team totalenergies...... …… jamie stauffer is a professional motorcycle racer who competes in the australian superbike championship...... alexis vuillermoz relationship count: 238 hypernym: racer step 1: generate a shared abstract question step 2: generate answers for both entities separately step 3: generate a distractor for both entities step 4: form the multiple-choice question using the four options step 5: reflect and check -> succeed / fail high frequency entity: alexis vuillermoz low frequency entity: jamie stauffer 1) entity pairs extraction from dbpedia comparisonqa 3) hard high-quality question selecting comparisonqa -hard comparisonqa × n correctness answer : …… uncertainty × n 2) abstract question generation and through the third part, (3) hard high-quality question selecting, we can get a harder subset, containing only difficult low-frequency questions with high quality and no semantic shortcut. ## robust knowledge measurement with the help of comparisonqa, we can conduct a more detailed and controllable study of the factual knowledge of llms. when considering how humans tackle multiplechoice questions, it's often the case that we do not really know the correct answers. instead, we rely on semantic shortcuts within the questions to make educated guesses. although we intentionally exclude these shortcuts when constructing the benchmark, they are difficult to eliminate entirely from multiple-choice questions. such a situation often occurs even in human exam questions. therefore, we need to devise an effective method to evaluate the robustness of factual knowledge of llms in the form of multiple-choice questions. ## comparisonqa-hard directly collected questions, especially the highfrequency part, are a bit simple for today's llms. more importantly, they may have low quality and semantic shortcuts. so we introduce a new method to select a subset called comparisonqa-hard, containing only difficult, low-frequency questions that have high quality and no semantic shortcuts. ## conclusions in this paper, we first introduce comparisonqa benchmark to evaluate llms' factual knowledge, with a fully automatic pipeline. this benchmark allows for more controllable and detailed comparisons between high-frequency and low-frequency knowledge of llms. then, we propose a tworound method utilizing correctness and uncertainty to measure llms' knowledge robustness. and we are surprised to find that even powerful llms like gpt-4o can not stand such a test, especially on the low-frequency knowledge. at last, we discover that uncertainty is more effective in filtering out questions with low quality and shortcuts compared with correctness, which is often used by recent works. based on this method, we provide a subset called comparisonqa-hard, which contains only difficult low-frequency questions of high quality and no shortcuts for future study.