Atomic Self-Consistency for Better Long Form Generations (2024)

Abstract: Recent work has aimed to improve LLM generations by filtering out hallucinations, thereby improving the precision of the information in responses. Correctness of a long-form response, however, also depends on the recall of multiple pieces of information relevant to the question. In this paper, we introduce Atomic Self-Consistency (ASC), a technique for improving the recall of relevant information in an LLM response. ASC follows recent work, Universal Self-Consistency (USC) in using multiple stochastic samples from an LLM to improve the long-form response. Unlike USC which only focuses on selecting the best single generation, ASC picks authentic subparts from the samples and merges them into a superior composite answer. Through extensive experiments and ablations, we show that merging relevant subparts of multiple samples performs significantly better than picking a single sample. ASC demonstrates significant gains over USC on multiple factoids and open-ended QA datasets - ASQA, QAMPARI, QUEST, ELI5 with ChatGPT and Llama3. Our analysis also reveals untapped potential for enhancing long-form generations using the approach of merging multiple samples.

## introduction large language models (llm) with their ability to perform mathematical reasoning @xcite , planning @xcite , and generating humanlike text @xcite have become an integral component of many ai systems. long-form question answering (lfqa) is an important benchmark task whose performance reflects the reliability of these ai systems at providing comprehensive and accurate responses to user queries. in lfqa, each response comprises multiple pieces of information, described in the literature as atomic facts @xcite , that collectively contribute to the overall correctness of the answer. despite various improvements, llms are still very q: what are the main causes of climate change? a1: the primary causes of climate change include human activities like burning fossil fuels, deforestation, and industrial processes. a2: the primary causes of climate change include human activities like burning fossil fuels, deforestation, and industrial processes. natural factors such as volcanic eruptions and changes in solar radiation also contribute. prone to producing hallucinatory content such as incorrect atomic facts, especially when the responses are longer @xcite . recent works on mitigating hallucinations have primarily involved the removal of inaccurate atomic facts from the generated content @xcite @xcite . while these methods produce responses with high precision over atomic facts, the correctness of the response also depends on the inclusion of all information relevant to the question, i.e., the recall of atomic facts relevant to the question. e.g. in fig. 1 , a 1 is a precise response. a 2 is a more complete high recall response to q. on the other hand, in qa with closed-form answers (such as a math problem with a numeric answer), remarkable improvements in response quality have been achieved by stochastically sampling multiple model responses and then using consistency criteria to select one as the final answer @xcite . recently, similar efforts were extended to long-form generation. universal self consistency (usc) @xcite , is one such example which uses llms to determine consistency between model responses. the output of usc is the single most consistent generation among multiple candidate samples from the model. however, picking a single final answer among the candidate generations might miss out on relevant atomic facts from other candidates and not optimize the recall of information. further, it is still prone to some atomic hallucinations within the final selected candidate. to overcome these challenges, we propose a simple approach called atomic self-consistency (asc), which combines authentic atomic facts from multiple candidate responses to generate a superior composite response. to motivate the readers on the potential benefits of this approach, fig. 2 shows the oracle performance (best possible performance) of picking one single generation vs merging subparts of multiple generations on the asqa dataset (details in §4.7). merging answers from multiple samples have significant performance potential over picking a single answer. usc would not be able to capture this potential. fig. 2 also shows the performance of asc and other baselines. asc matches the ceiling performance possible of picking the best sample. key contributions 1. we introduce a simple and efficient method, asc, that: (i) clusters atomic parts of multiple candidate generations, (ii) uses a consistency-based criterion to pick the best clusters relevant to the question, and (iii) finally combines them all into a single superior answer. asc operates in black box mode and does not require access to llm weights or logits. 2. we systematically establish the superiority of combining multiple generations over picking a single generation. 3. we extensively evaluate and show significant performance improvements of asc over usc and other hallucination reduction baselines on four diverse long-form qa tasks-asqa, qampari, quest and eli5. we justify the benefits of asc through rigorous ablations. 4. we show strong empirical evidence for minimizing the number of stochastic samples required by asc. 5. finally, our analysis reveals untapped potential for enhancing llms by merging multiple generations into a superior composite output. this insight underscores a promising avenue for advancing llm performance further. ## related work we discuss key methodologies that are used to identify hallucinations and alleviate hallucinations. we further talk about the importance of consistency as a measure for correctness and finally talk about how this is used to improve llm response. we also discuss concurrent work using stochastic samples to improve llm generation quality. ## methodology given a question q, our task is to use an llm l to produce an answer which answers the question both accurately (i.e., with high precision) and comprehensively (i.e., with high recall). we describe the specific metrics used to measure this for each dataset in §4. let a 1 , a 2 , .., a m be m independent samples directly generated by l when query q is fed to it in a prompt p . this work merges consistent subparts from the multiple samples to produce a final answer a asc in a four-step process. 1. split: split each generation into its constituent atomic facts. 2. cluster: grouping atomic facts for efficiency 3. filter: selecting the most consistent clusters. 4. summarize: combine the selected cluster representatives to generate a final answer. ## conclusion in this work, we propose asc, a simple way of merging subparts of multiple answer samples produced by an llm. through extensive experiments and ablations, we show the 1. benefits of merging subparts of multiple answers over picking one single answer. 2. strength of consistency as a measure for improving correctness. asc significantly outperforms usc, a strong baseline for generating long-form answers. we show empirical evidence for minimizing the number of samples required by asc. finally, our analysis also reveals untapped potential for enhancing long-form generations using our approach of merging multiple responses. multiple samples still need to be generated a general limitation of most stochastic sample based methods. these methods rely on generating multiple samples and picking the final answer among them. however, this might be slightly expensive. speculative decoding @xcite has recently made great strides in reducing the amount of compute required to sample from large language models. speculative decoding can be used to significantly reduce the compute required by asc and usc. replicability: sourcecode: https://github.com/raghavlite/asc broader impact and discussion of ethics: while our model is not tied to any specific applications, it could be used in sensitive contexts such as health-care, etc. any work using our method is requested to undertake extensive quality-assurance and robustness testing before applying in their setting. to the best of our knowledge, the datasets used in our work do not contain any sensitive information. license: refer to the licenses of individual training datasets used @xcite , @xcite , @xcite , @xcite and llm models used @xcite , @xcite . @xcite had generated three subclaims from each golden answer and showed that an answer's entailment score over these sub-claims provides a more accurate measure of its correctness. we use this same 'claim-recall' to measure the correctness of a generated answer in this work. similar to str_em in asqa, this again is very related to the recall of atomic facts relevant for the question. we use the same randomly sampled 1000 questions from the eval set as @xcite . we use the test sets for qampari, quest and validation sets for asqa, eli5.