F in LFQA : Evaluating Attributed Text Generation of LLM s in Financial Long-Form Question Answering (2025)

Abstract: Large Language Models (LLMs) frequently hallucinate to long-form questions, producing plausible yet factually incorrect answers. A common mitigation strategy is to provide attribution to LLM outputs. However, existing benchmarks primarily focus on simple attribution that retrieves supporting textual evidence as references. We argue that in real-world scenarios such as financial applications, attribution goes beyond reference retrieval.We introduce FinLFQA, a benchmark designed to evaluate the ability of LLMs to generate long-form answers to complex financial questions with reliable and nuanced attributions. FinLFQA evaluates three critical aspects of attribution through human annotations: (1) supporting evidence extracted from financial reports, (2) intermediate numerical reasoning steps, and (3) domain-specific financial knowledge that informs the reasoning process.We further provide an automatic evaluation framework covering both answer quality and attribution quality. Through extensive experiments on eight LLMs across multiple attribution-generation paradigms, we find that fine-grained metrics are important to distinguish model capabilities, that end-to-end generation achieves comparable performance to post-hoc approaches, and that iterative refinement only helps when guided by external feedback.

## introduction long-form question answering (lfqa) over documents poses a substantial challenge for current large language models (llms) because it demands the ability to process and retain information across lengthy contexts, perform multi-step reasoning, and generate factually accurate responses @xcite @xcite . a key challenge in lfqa is hallucination, where llms * equal contributions. correspondence: yilun zhao (yilun.zhao@yale.edu), chen zhao (cz1285@nyu.edu) produce output that is not grounded in the source content, which can severely compromise user trust @xcite . to address this challenge, researchers have demonstrated increasing interest in attributed text generation @xcite , which aims to improve the trustworthiness of generated content by providing supporting evidence for model outputs. this attribution enables users to verify claims and assess the reliability of responses. most current approaches focus on tasks such as fact-checking @xcite and summarization @xcite , where evidence attribution is comparatively straightforward. however, existing benchmarks @xcite primarily evaluate attribution through a single lens: evidence retrieval. while this is a necessary component, attribution in critical domains like finance extends well beyond locating supporting passages. as shown in figure 1 , two additional forms of attribution are important in financial domain. first, financial texts are rich in numerical data, requiring models not only to identify evidence but also to execute precise numerical reasoning to derive trustworthy conclusions @xcite . second, reliable answers demand integration of specialized financial knowledge, where models must grasp intricate domain concepts and relationships @xcite . to address these gaps, we introduce finlfqa, a comprehensive benchmark for evaluating longform qa and attributed generation by llms. finlfqa consists of 1,008 expert-annotated instances. as illustrated in figure 1 , it is designed to test llms' ability to apply financial knowledge, perform analytical reasoning, and generate detailed answers grounded in the financial reports of two relevant companies. in contrast to prior work that focuses primarily on evidence attribution, finlfqa evaluates three distinct forms of attribution in generated responses: supporting evidence, intermedithe us took the first step towards gaining independence from gb when it declared independence on july 2, 1776 (although the event is now commemorated on july 4, 1776, the date when the declaration of independence was officially adopted by congress). ate reasoning steps, and domain-specific knowledge. these attribution types are essential in the financial domain and broadly relevant to other highstakes applications. ## finlfqa benchmark in this section, we first formulate the task, then present the statistical analysis of the dataset and describe our data annotation process. ## automated evaluation system evaluating financial long-form question answering is challenging because it requires assessing textual accuracy, numerical correctness, and domain knowledge. to address these challenges, in addition to traditional overlap-based metrics (i.e., rouge and bertscore), we propose fine-grained metrics that jointly capture these dimensions. ## experiments this section describes the experimental setup, models evaluated, main results, and error analysis. ## conclusion this paper introduces finlfqa, a comprehensive benchmark designed to evaluate the ability of llms to generate factually grounded and wellattributed answers in long-form question answering. we design fine-grained automatic evaluation methods that measure answer quality at both token and semantic levels, assess numerical correctness, and evaluate attribution quality across evidence, code, and professional knowledge. our extensive experiments compare different generation strategies, including post-hoc, end-to-end generation and iterative refinement. experiment results show that gpt-4o achieves the highest overall performance, while open-source models demonstrate strong potential. additionally, finlfqa underscores the importance of fine-grained metrics in financial long-form question answering task. experiments on finlfqa indicate that post-hoc and end-to-end generation perform similarly, as frontier llms can handle complex attribution tasks effectively. furthermore, iterative self-feedback does not significantly improve overall performance, except in execution-based tasks where explicit feedback from generated outputs aids refinement. our analysis also shows that when external feedback is introduced, models with sufficient capacity can leverage it effectively, and feedback from domain-specific financial models further improves outcomes.