Localizing and Mitigating Errors in Long-form Question Answering (2025)

Abstract: Long-form question answering (LFQA) aims to provide thorough and in-depth answers to complex questions, enhancing comprehension. However, such detailed responses are prone to hallucinations and factual inconsistencies, challenging their faithful evaluation. This work introduces HaluQuestQA, the first hallucination dataset with localized error annotations for human-written and model-generated LFQA answers. HaluQuestQA comprises 698 QA pairs with 1.8k span-level error annotations for five different error types by expert annotators, along with preference judgments. Using our collected data, we thoroughly analyze the shortcomings of long-form answers and find that they lack comprehensiveness and provide unhelpful references. We train an automatic feedback model on this dataset that predicts error spans with incomplete information and provides associated explanations. Finally, we propose a prompt-based approach, Error-Informed Refinement, that uses signals from the learned feedback model to refine generated answers, which we show reduces errors and improves the quality of the answers across multiple models. Furthermore, humans find the answers generated by our approach comprehensive and highly prefer them (84%) over the baseline answers.

## introduction long-form question answering (lfqa) provides comprehensive, user-friendly, and in-depth responses to complex questions by leveraging stateof-the-art large language models (llms) and retriever components @xcite . while llms generate plausible and convincing answers, they also hallucinate and produce factually inconsistent, irrelevant, and incomplete content @xcite @xcite , which are difficult to detect for both humans and machines. traditional evaluation metrics of answer quality, such as bleu @xcite , rouge @xcite , and bertscore @xcite yield only a single score, obscuring the error type, severity, and location in the answer. we take inspiration from machine translation, which moved beyond this simplistic evaluation paradigm by localizing and categorizing errors @xcite , resulting in higherquality and more interpretable evaluations. we make a similar contribution to the field of lfqa by asking human annotators to identify spans from the answers that correspond to errors and categorize each span into an error schema that we design. our work is the first to explore error localization in lfqa, offering a more detailed and interpretable evaluation of answers. llms make many errors for lfqa that require targeted evaluation. @xcite highlight that key aspects such as factuality, relevance, completeness, structure, references, and accessibility are essential to evaluate long-form answers. other recent studies have also validated the importance of aspects such as question misconception @xcite , factuality @xcite @xcite , relevance @xcite , and completeness @xcite in determining answer quality. while prior work has mainly focused on evaluating factuality @xcite @xcite and faithfulness @xcite in long-form text generation, other aspects of evaluation, such as response completeness and relevance (which can particularly mislead users), have received less attention. our work addresses this gap by introducing haluquestqa, a dataset of long-form answers annotated at the span level with five error types: question misconception, factuality, completeness, relevance, and references. expert annotators provide these annotations and preference judgments, as shown in figure 1 . next, we train an automatic feedback model on this dataset to predict erroneous answer spans that lack key details to address the question comprehensively. the feedback model provides fine-grained feedback, identifying error locations (sentence level), error justification, and a confidence score, all without relying on reference texts @xcite . finally, we propose error-informed refinement, a prompt-based approach that uses signals from the feedback model to refine generated answers @xcite , reducing errors and improving answer quality across multiple llms. our contributions are summarized as follows: human evaluation. prior work @xcite has shown that human evaluation for lfqa tasks is challenging due to long answer lengths, and expert annotators are required to evaluate them effectively. @xcite hire expert annotators and identify nine multi-faceted aspects for meaningful lfqa evaluation. while some of these finegrained aspects, such as factuality @xcite , coherence @xcite , and completeness @xcite , have been used to investigate errors in summarization tasks, ours is amongst the first works to study lfqa-centric errors at the span level. to this end, we collect span-level annotations of lfqa errors, enabling high-quality and interpretable evaluations that can be used to improve answer quality. while this has been done for machine translation @xcite , it has not yet been applied to long-form question answering. automatic evaluation. increasing focus on the reliability of llms has led to the development of explainable evaluation metrics @xcite to detect errors in llm generations. @xcite present instructscore, an explainable metric based on llama @xcite , to obtain detailed error analysis for llm-generated text. however, most of the current evaluation metrics require hard-to-obtain gold references. @xcite propose a reference-free evaluation metric, tigerscore, that can locate, categorize, and explain errors across various text generation tasks, including lfqa. while llmbased metrics can detect diverse errors, they are prone to hallucinations due to training data quality. in this study, we collect expert-annotated data on fine-grained lfqa errors and train a feedback model for accurate error detection. ## error mitigation in §3.4, we have shown that the lfqa answers lack completeness and omit useful information. therefore, we train a feedback model to identify erroneous answer spans with incomplete information and provide free-form error justifications. our approach error-informed refinement, uses this feedback to refine the answers and improve their overall quality without human intervention. ## results we explore several research questions: (1) can our learned feedback model detect errors in lfqa systems and help in downstream answer refinement task? (2) does fine-grained feedback produce better quality lfqa answers than coarse-grained feedback? (3) does fine-grained feedback help mitigate errors and improve the comprehensiveness of lfqa answers? (4) are comprehensive answers from our approach preferred by humans? ## conclusion we introduce haluquestqa, a dataset of expert judgments on fine-grained errors in lfqa. using our dataset, we analyze the pitfalls of human and model long-form answers, identifying issues with comprehensiveness and unhelpful references. to address these, we propose error-informed re-finement, an approach that uses signals from our learned feedback model to refine llm responses. our feedback model outperforms baseline feedback models and expert human feedback in guiding answer refinement and reducing errors. a human evaluation confirms the effectiveness of our approach, with participants finding our refined answers more comprehensive and preferable to baseline outputs.