Investigating Factuality in Long-Form Text Generation: The Roles of Self-Known and Self-Unknownan (2025)

Abstract: Large language models (LLMs) have demonstrated strong capabilities in text understanding and generation. However, they often lack factuality, producing a mixture of true and false in- formation, especially in long-form generation. In this work, we investigates the factuality of long-form text generation across various large language models (LLMs), including GPT-4, Gemini-1.5-Pro, Claude-3-Opus, Llama-3-70B, and Mistral. Our analysis reveals that factuality tend to decline in later sentences of the generated text, accompanied by a rise in the number of unsupported claims. Furthermore, we explore the effectiveness of different evaluation settings to assess whether LLMs can accurately judge the correctness of their own outputs: Self- Known (the percentage of supported atomic claims, decomposed from LLM outputs, that the corresponding LLMs judge as correct) and Self-Unknown (the percentage of unsupported atomic claims that the corresponding LLMs judge as incorrect). The results indicate that even advanced models fail to achieve perfect Self-Known scores, while their Self-Unknown scores remain notably above zero, reflecting ongoing uncertainty in their self-assessments. Moreover, we find a correlation between higher Self-Known scores and improved factuality, while higher Self-Unknown scores are associated with lower factuality. Even without sig nificant changes in the models’ self-judgment (Self-Known and Self-Unknown), the number of unsupported claims can increases, likely as an artifact of long-form generation. Additional Retrieval-Augmented Generation (RAG) experiments also show the limitations of current LLMs in long-form generation, and provide the more research is needed to improve factuality in long-form text generation.

## introduction the long-context capabilities of large language models (llms) (openai, 2023b; ai@meta, 2024; * now at google. @xcite @xcite have seen significant advancements in recent years. lots of work @xcite @xcite @xcite have explored the ability of llms to handle long contexts, however, relatively few have examined their ability for long-form text generation. despite llms have the impressive generative abilities, these models are prone to producing hallucinations @xcite where the generated content often blends factual and fabricated information. this tendency not only undermines performance but also poses substantial risks in practical applications. to assess the factuality of responses from llms, recent research @xcite @xcite has introduced a method that breaks down generations into atomic claims -short statements each containing a single piece of information. these atomic claims are then individually evaluated to determine whether they are supported by evidence or unsupported. to ensure the reliable use of llms, it is also crucial that they possess the ability to recognize not only "what they know" but also "what they don't know." recent studies, such as those by @xcite ; @xcite ; @xcite , have shown that language models can assess the validity of their own claims. however, srivastava et (2023); @xcite have pointed out the limitations of llms in recognizing their own knowledge gaps. in this work, we investigate the factuality of long-form text generation across various llms. we first check the factuality of long-form generation at different relative positions using two annotated datasets and two models: chatgpt and perplexityai (which integrates a search engine). our findings verify that sentences generated earlier in the sequence generally demonstrate higher fac-tuality. however, these later-generated sentences contain more unsupported claims and fewer supported claims. to explain this phenomenon, we try to examine whether llms exhibit reduced knowledge in later generations with wo metrics: the self-known score (the percentage of supported atomic claims judged as correct by the llms) and the self-unknown score (the percentage of unsupported atomic claims judged as incorrect by the llms). these two metrics are used to quantify the corresponding models' ability to judge the correctness of atomic claims. in order to accurately compute the two metrics, we have tried three different approaches, one of which is a novel approach that adds an answer option: 'none of the above'. this modification appears to provide a more accurate measure of the llms' abilities, as evidenced by a higher flip rate for supported claims and an increasing flip rate at later relative positions. this suggests that the model reassesses its confidence when faced with an option signaling uncertainty. in contrast, the low flip rate for unsupported claims indicates a consistent judgment of their incorrectness. these results suggest a nuanced understanding by llms of supported versus unsupported claims and underscore the importance of specific evaluation settings to accurately gauge model performance. our findings align with human annotations for two llms, although some discrepancies, particularly with the perplexityai model, suggest gaps in estimation. later, we apply this modified approach to compute the self-known and self-unknown scores across various llms, including gpt-4, gemini-1.5-pro, claude-3-opus, llama-3-70b, and mistral. we also develop a mathematical framework that links these scores to factuality. overall, both empirical and theoretical results demonstrate a strong relationship between factuality and the self-known and self-unknown scores. we observe that these scores can vary significantly across different models. however, even when the self-known and self-unknown scores remain relatively stable, the number of unsupported claims tends to increase in later parts of the generated text. this suggests that lower factuality in later sentences is not solely due to score changes, but also influenced by error propagation and diminished model knowledge over time. the main contributions of our work are as follows: ## long-form text generation to evaluate the factuality of llm responses, recent work @xcite @xcite breaks a generation into a series of atomic claims-short statements that each contain one piece of information. each atomic claim is then individually evaluated to determine whether it is supported or unsupported. in this section, we first explore the factuality patterns of these atomic claims in long-form text generation. ## self-known and self-unknown to investigate these questions, we examine whether the corresponding llms recognize their atomic claims by computing two metrics: self-known (the percentage of supported atomic claims that the corresponding llms judge as correct) and self-unknown (the percentage of unsupported atomic claims that the corresponding llms judge as incorrect). while there is related work, such as rajpurkar et al. (2018); xiong et al. (2024) , our approach differs in two key ways: (1) evaluation is conducted on atomic claims, which are derived from sentences in long-form generation, rather than assigning a score to the entire model output; (2) our focus is on factuality (whether an atomic claim is true or false), rather than on uncertainty scores (i.e., "how likely is the above answer to be correct?"). we explore the computation of self-known and self-unknown using the following three approaches ( with the corresponding prompt templates provided in appendix section b): we compute the self-known score and the self-unknown score using these prompt templates. the human annotated data on chatgpt foot_0 are used in this experiments. figure 2 presents the results on chatgpt. comparison on the above three evaluation settings with the first two settings, the results of self-known score and self-unknown score are similar. however, the results of the third setting differ from the other two. we hypothesize that the reason is that the added choice, "none of the above" which allows the llm to determine whether it knows the answer to the question. to examine the effect of this setting, we plot the flip rate (claims judged as correct by the llm in setting (b) but judged as incorrect in setting (c)) for supported and unsupported claims. as shown in figure 2d , there is a high flip rate for supported claims, and this rate increases with higher relative positions. in contrast, there is almost no flipping for unsupported claims. therefore, setting (c) is more suitable for checking whether the llm knows a atomic claim. the high flip rate observed for supported claims suggests that the model is reconsidering its initial judgments when presented with the 0 25 50 75 100 [, 20) [20, 40) [40, 60) [60, 80) [80, ) s n s (s-known) n (s-unknown) (a) direct-asking 0 25 50 75 100 [, 20) [20, 40) [40, 60) [60, 80) [80, ) s n s (s-known) n (s-unknown) (b) question-answering 0 25 50 75 100 [, 20) [20, 40) [40, 60) [60, 80) [80, ) s n s (s-known) n (s-unknown) (c) question-answering w/ noa 0 10 20 30 40 [, 20) [20, 40) [40, 60) [60, 80) [80, ) supported unsupported flip rate (d) flip rate (%) from setting (b) to setting (c) across different relative positions (%) for both supported claims and unsupported claims. the high flip rate of supported claims indicates that prompting with setting (c) better recognizes whether llms accurately assess their knowledge of atomic claims. option "none of the above". this indicates that the model may not be entirely confident in its original answers and is more likely to recognize uncertainty. the increasing flip rate for higher relative positions further supports this, implying that the model's confidence decreases as the position of the claim within the context changes. in summary, we observed similar results between setting (a) (direct-asking) and setting (b) (question-answering), and a significant difference between setting (b) (question-answering) and setting c (question-answering w/ noa). the deeper analysis between setting (b) and setting (c) revealed that setting (c) recognizes atomic claims more confidently and treats atomic claims that flip as unknown. this is why we chose to use setting (c) in the subsequent experiments. ## analysis we denote the prompt input of llms as x and long output of llms as y. the binary auxiliary label @xmath0. we assume that p(@xmath1, x) is equal to factuality score foot_1 σ of llm output y. given x, the joint distribution of between the auxiliary label and model output (d, y) is where σ is the factuality score. ## automatic results on additional llms in the previous section, our experiments were conducted using human-annotated factuality data. in this section, we first introduce an automated tool for factuality evaluation. then, using the proposed approach from section 3 to compute self-known and self-unknown scores, we analyze the trends in factuality, self-known, and self-unknown scores across other advanced llms ## related work factuality evaluation recent advancements have seen significant efforts in quantifying the factuality of llm generations. for short answers, factuality often correlates with fact verification, which directly assesses whether the generation aligns with extensive knowledge sources and references @xcite or utilizes language models @xcite . however, evaluating factuality in long-form content poses greater challenges due to the complexity of the generation process. recent studies @xcite @xcite have approached this challenge by breaking down long generations into atomic claims. while these approaches predominantly focus on factual precision, some studies @xcite also consider evaluating factual recall. in our work, we concentrate on factual precision akin to @xcite . moving forward, the development of more robust automatic tools will be crucial for advancing factuality exploration in long-form generation tasks. self-know and self-unknown recent studies have extensively explored the concepts of self-known and self-unknown in language models. for instance, @xcite ; @xcite demonstrated that language models are capable of assessing the validity of their own claims and predicting their ability with answering true/false questions accurately. meanwhile, srivastava et (2023); @xcite highlighted the limitations of llms in acknowledging their unknowns, focusing on their ability to recognize unknown knowledge. in our work, we specifically investigate whether llms can identify and reconsider unsupported claims generated from their own outputs. our results indicate that llms struggle to accurately judge unsupported atomic claims from their own generations. we also find that a lower self-unknown score or a higher self-known score corresponds to higher factuality. evaluation of self-know and self-unknown in this work, we design three different methods for estimating self-known and self-unknown scores on llms' own generation. we find that the third setting (c), which includes the option "none of the above," is effective in determining whether llms can accurately judge the correctness of claims generated from their own outputs. although our results show that these scores are well aligned with the estimation of factuality scores using equation 3, exploring better methods for evaluating the correctness of claims with llms would still be beneficial for future study. factuality evaluation in this work, we limit the domain of long-form generation to ensure accurate factuality evaluation. the concern is that broadening the topic range might compromise the accuracy of our factuality assessments, rendering our study less effective. recently, evaluation tools @xcite @xcite have been explored. with stronger tools, it is possible to explore a wider range of domains beyond wikipedia. moreover, in this work, we primarily focus on factuality precision. however, considering factuality recall is also important, as it ensures that the omission of significant pieces of information is penalized during evaluation. by incorporating both precision and recall, we can achieve a more comprehensive and accurate assessment of factuality in long-form generation.