Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations (2024)

Abstract: Long-form generations from large language models (LLMs) contain a mix of factual and non-factual claims, making evaluating factuality difficult.Prior works evaluate the factuality of a long paragraph by decomposing it into multiple facts, verifying those facts independently, and aggregating the results.Such methods assume that combining factual claims forms a factual paragraph.The above assumption can be violated: we show that strong open-source models like Llama-chat can generate paragraphs that contain verifiable facts, but the facts are combined into a non-factual paragraph due to entity ambiguity.We further reveal that existing factuality metrics, including FActScore and citation recall, cannot properly evaluate these non-factual paragraphs and overestimate their factuality.To address this, we introduce an enhanced metric, **D-FActScore**, specifically designed for content with ambiguous entities.We evaluate the D-FActScores of people biographies generated by retrieval-augmented LLMs.We show that D-FActScore can better assess the factuality of paragraphs with entity ambiguity than FActScore.We also find that four widely used open-source LLMs tend to mix information of distinct entities to form non-factual paragraphs, making their D-FActScore much lower than FActScore by over 10%.

## introduction llms can generate high-quality texts, making llms prevalent in everyday usage @xcite . however, llm's generation may not always have a high factual precision @xcite . factual precision measures whether the information conveyed in the text is factually accurate. 1 as long-form generations can contain a mix of factual and non-factual claims, recent 1 we only focus on factual precision and do not consider factual recall, i.e., how well the generation covers the information. this paper will use factuality to refer to factual precision. works propose to evaluate the factuality of longform generation in a more fine-grained way that considers the factuality of each claim in the generation. precisely, these methods decompose the generation into several claims (called atomic facts in @xcite ) and then verify each claim independently. the factuality of the whole generation is the percentage of verifiable claims. widely used evaluation metrics like factscore @xcite and citation recall @xcite fall into this type. the correctness of the previous factuality metrics relies on the assumption that "as long as each claim is factual, the combination of those claims is factual." this paper shows that this assumption can be violated due to entity ambiguity in the generation. figure 1 shows such a case, where llama-2-13b-chat says that dick hanley was born in 1936 and died in 1970, where the birth date is true for dick hanley (swimmer) and the time of death is true for dick hanley (american football), but readers will think all the information refers to the same individual. since the paragraph is not written in a way that allows readers to understand the content factually, the paragraph is non-factual. however, since each claim in the generation is supported by wikipedia, the factscore of the above misleading paragraph is 100% by definition. we emphasize the significance of this evaluation problem by showing that llms can easily generate this kind of non-factual paragraph. we collect 500 ambiguous names from wikipedia to form a dataset, ambigbio, where each name corresponds to multiple entities in wikipedia. we prompt llms to generate a biography for a name in ambigbio using retrieval-augmented generation @xcite , where the target name is shared by multiple entities, and the retrieved documents may also include the wikipedia pages of multiple entities. we find that llama-13b/70bchat @xcite , tulu-13b-dpo @xcite , and vicuna-7b @xcite tend to mix the information of multiple entities in the same biography, and a reader without prior knowledge cannot tell that the information corresponds to different entities. non-factual as these paragraphs are, existing factuality metrics cannot correctly assess the factuality in this case. to solve this evaluation problem, we modify factscore into disambig-factscore (d-factscore), which handles entity ambiguity better. unlike factscore, which verifies each atomic fact in the paragraph independently, d-factscore splits the facts in a paragraph into groups. if the narration of the paragraph can make a reader without prior knowledge think that the two facts are about the same individual/object, they belong to the same group. next, d-factscore uses entity linking to find an entity in the knowledge source that best matches a group of facts and use this entity to verify the facts in the same group. we recruit humans to annotate the d-factscores of people biographies generated with llama-13bchat, @xcite . we show that llama and tulu mix the information of multiple entities in a non-factual way. compared to factscore, d-factscore correctly identifies those non-factual generations and gives a d-factscore 8% to 16% lower than factscore. we also propose a pipeline to estimate d-factscore automatically and show that our automatic pipeline estimates d-factscore to a reasonable degree. our contributions are summarized as follows: ## llms combine factual claims into non-factual paragraph this section shows how to prompt llms to generate people biographies full of factual claims but combined in a non-factual way due to entity ambiguity. we choose to generate people biographies for the following reasons: (1) there are extensive prior works that use biography generation to evaluate the factuality of llms @xcite . (2) given that name-based queries are common in online searches, it is likely that llm users will directly request information about specific entities from llms. ## existing factual precision metrics we discuss why some factual precision metrics cannot properly assess the factuality of paragraphs with entity ambiguity, as shown in figure 1 . ## disambig-factscore (d-factscore) we refine factscore into d-factscore to better address entity ambiguity in factuality evaluation. the definition of d-factscore is presented in section 4.1. section 4.2 outlines the human annotation process for d-factscore, with the outcomes of human evaluations detailed in section 4.3. ## conclusion and discussion we show that combining factual information yields a non-factual paragraph due to entity ambiguity, and this kind of paragraph can be easily generated by llms using rag. we further reveal that current factuality metrics cannot correctly assess the factuality of these non-factual paragraphs. to resolve the evaluation problem, we modify factscore into d-factscore by focusing on entity disambiguation. we conduct human evaluations of d-factscore to compare the factuality of different llms and show that d-factscore of open-source llms lag behind chatgpt, indicating that opensource llms cannot handle ambiguous entities in the retrieved documents to form a factual narrative. we propose a pipeline for automatic evaluation of d-factscore and show it aligns with human evaluation results to a reasonable extent. we encourage future researchers to use ambigbio and evaluate the factuality of generated paragraphs using d-factscore. the ambiguous name collection ambigbio and the codes for d-factscore are available at https://github.com/d223302/ merging-facts-crafting-fallacies . our findings underscore llms' difficulties in generating accurate content when retrieving from wikipedia with entity ambiguity. the scenario in our paper is simplified, given that wikipedia is not the sole information source about entities. even using high-quality contents from wikipedia, opensource models struggle to differentiate between entities accurately. in more realistic situations where llms draw from a broader and more nuanced content pool on the web, and the named entities used to query llms may not appear in the training data, distinguishing between different entities becomes even more challenging. overcoming this issue is vital for the reliability of llms with rag.