Groundedness in Retrieval-augmented Long-form Generation: An Empirical Study (2024)

Abstract: We present an empirical study of groundedness in long-form question answering (LFQA) by retrieval-augmented large language models (LLMs).In particular, we evaluate whether every generated sentence is grounded in the retrieved documents or the model’s pre-training data.Across 3 datasets and 4 model families, our findings reveal that a significant fraction of generated sentences are consistently ungrounded, even when those sentences contain correct ground-truth answers.Additionally, we examine the impacts of factors such as model size, decoding strategy, and instruction tuning on groundedness. Our results show that while larger models tend to ground their outputs more effectively, a significant portion of correct answers remains compromised by hallucinations. This study provides novel insights into the groundedness challenges in LFQA and underscores the necessity for more robust mechanisms in LLMs to mitigate the generation of ungrounded content.

## introduction one of the most significant challenges to the safe deployment of large language models (llms) is their propensity to generate hallucinated content @xcite @xcite . the risk of hallucinating increases when llms are tasked with generating long content (i.e., more than a single sentence) @xcite . this is problematic because generating long-form text is a critical component of a number of important tasks, such as disambiguating complex topics, explicit problem decomposition and reasoning, question answering, and synthesis of information from multiple sources. as a step towards mitigating hallucination, a number of studies have measured the groundedness of llm generations (for a recent survey, see d 🤖 […] cabo san lucas where the pacific ocean and gulf of california meet. frozen lm m query q 🕵 🔍 pre-training corpus c compared to the west side, the sea of cortez has warmer weather […] ## experimental procedure in this section, we detail our experimental setup, including how correctness and groundedness are measured, as well as the datasets used. we begin by defining notation. ## how frequently are generations grounded? we begin our analysis by measuring the rate at which models of various sizes and families generate sentences that are grounded in the retrieved documents as well as the pre-training data. as an example, consider figure 2 , which provides a visualization of the sentences generated by the pythia 12b model on the asqa dataset divided into 8 sectors. each sector represents one group in the cross-product of the following categories: we expect that most sentences that are part of em + will be grounded in either the retrieved documents or the pre-training corpus (or both), since those documents represent the source of the model's correctness. for sentences that are part of em 0 , we make no assumptions on the frequency of grounding. that is because incorrect answers could arise none, em 0 none, em + pt corpus, em 0 pt corpus, em + ret. docs, em 0 ret. docs, em + both, em 0 both, em + 33.1% 22.7% 3.2% 2.8 % 4 .5 % 6 . 1 1 % 11.6% 15.9% attribution scores for pythia 12b on asqa from the model emitting ungrounded sentences, or grounded sentences that are off-topic or otherwise incorrect. figure 2 reveals that overall, for pythia 12b, ∼44% of the sentences generated are grounded in at least one of the two sources considered (i.e., blue, green, and orange sectors), and that ∼48% belong to em + (i.e., the darker-color sectors). interestingly, we observe that nearly half of the generated sentences belonging to em + cannot be grounded in the retrieved documents or pre-training corpus. this is unexpected since, by virtue of containing at least some of the ground-truth answers, the model demonstrates that it may have access to relevant information. upon inspection, we find that the high proportion of ungrounded sentences in these answers contain ground-truth named entities or text snippets that are presented in a nonsensical or factually incorrect manner. we provide examples of such outputs in section 6. we note that sentences that cannot be grounded via our methods could be the result of sub-optimal retrieval in the pretraining corpus, or errors from the true (grounding) model. we elaborate more on these concerns in appendices b and c. additionally, figure 2 also shows that roughly one-fourth of the generations can be grounded in both the pre-training and retrieved document. this is likely due to overlap between the pre-training and retrieval corpora, or the appearance of common knowledge present in both corpora. does ungrounded content appear consistently? for the remainder of this analysis, we focus on outputs that contain ground-truth answers (i.e., em + ), as they might represent subtler and more interesting cases of undetected hallucination. figure 3 visualizes of frequency of grounded sentences in correct and partially correct answers on asqa, hotpotqa, and strategyqa for four models from the four families considered: mpt 7b, falcon 7b, silo 1.3b, and pythia 12b. note that the height of each bar represents the fraction of generated sentences that belong to long-form generations containing at least 1 ground-truth answer. on all three datasets and all models considered, we observe that a substantial fraction of the outputs in em + are not grounded in the retrieved documents or in the pre-training data. 1 this consistency across different models and datasets indicates a prevalent pattern where models are able to generate correct answers that are found in sentences that are not directly supported by the retrieved documents or pre-training data. in this section, we study the interplay between the tendency of models to generate grounded content and three factors: the size of the model (section 5.1), the decoding strategy (section 5.2), and instruction-tuning (section 5.3). p y t h i a 7 0 m p y t h i a 1 6 0 m p y t h i a 4 1 0 m p y t h i a 1 b p y t h i a 1 . 4 b p y t h i a 2 . 8 b p y t h i a 6 . 9 b p y t h i a 1 2 b as before, the height of each bar represents the fraction of generated sentences that belong to partially correct generations. increased model size correlates with an increase in the number of sentences in em + , but also an increase in groundedness. ## examples of model output in consider the example in the first row of ## conclusion this study analyzes the rate at which the long-form output produced by retrieval-augmented llms is grounded in retrieved documents and pre-training data. through empirical analysis across various models and datasets, we highlight the propensity of llms to blend correct information with hallucinated content. our findings indicate that this tendency is prevalent across different model sizes and persists even in the largest models available. our analyses reveal that while larger models generally produce more grounded content, they are not immune to generating ungrounded information. we observed that instruction tuning and beam search decoding reduce ungrounded sentence generation. aligned with the results of concurrent research @xcite , our findings point to specialized decoding algorithms being good candidates for significantly reducing hallucination.