Post-Hoc Answer Attribution for Grounded and Trustworthy Long Document Comprehension: Task, Insights, and Challenges (2024)

Abstract: Attributing answer text to its source document for information-seeking questions is crucial for building trustworthy, reliable, and accountable systems. We formulate a new task of post-hoc answer attribution for long document comprehension (LDC). Owing to the lack of long-form abstractive and information-seeking LDC datasets, we refactor existing datasets to assess the strengths and weaknesses of existing retrieval-based and proposed answer decomposition and textual entailment-based optimal selection attribution systems for this task. We throw light on the limitations of existing datasets and the need for datasets to assess the actual performance of systems on this task.

## introduction users now benefit from the help of automatic question-answering (qa) systems on a day-to-day basis when faced with an information need. such systems are integrated into search engines (e.g., bingai 1 ) and digital assistants (e.g., chatgpt). however, such systems are prone to generating answers lacking sufficient grounding to knowledge sources @xcite , leading to the risks of misinformation and hallucination @xcite @xcite . therefore, attributing the generated answers to the respective sources is crucial for building trustworthy, reliable, verifiable, and accountable systems @xcite @xcite ; by allowing users to verify outputs. existing works mainly consider generating attributed text in open-ended settings. these attributions are generated along with the answers either one per answer paragraph @xcite input question: when does the next assasins creed come out? document: [1] ubisoft has announced that its next assassin's creed game will be revealed in september 2022. ## evaluation as answer sentence attribution to sentences in the source document could also be considered as an information retrieval task, we benchmark the performance of a range of retrieval-based systems: adiosaa -d considers an answer sentence as the information unit instead of decomposing it. this system establishes the importance of the answer decomposer in adiosaa. adiosaa -os decomposes each answer sentence into its information units, and then ranks source sentences based on their entailment probabilities from the attr(p,h) for each information unit. to obtain attributions for each sentence of the answer, it deduplicates and reorders the attributions for all the information units of the sentence based on the entailment probabilities. adiosaa -d -os neither uses the answer decomposer or the optimal selection algorithm rather for each sentence in the answer, it ranks source sentences based on their entailment probabilities from the attr(p,h). this system demonstrates the effectiveness of both the components in adiossa. evaluation measures we report precision (p), recall (r), and f1@@xmath0, 2, 4} predicted attributions per sentence of an answer foot_1 for the test set of verifiability dataset and development set of the hagrid dataset (as no test set is available). we tune the threshold for attributor's entailment probability (=0.5) and δ (=0.3) in algorithm 1 based on the verifiability development set. ## results and discussion while monot5-based retrieval system outperforms ( good performance of retrieval-based systems indicate that the existing datasets are less abstractive for long-form comprehension, suggesting the need for research in creating more challenging datasets to foster the development of trustworthy, reliable, and accountable systems that can be used in realworld information-seeking scenarios. ## conclusion we introduce a task of post-hoc answer attribution for long document comprehension, reformulate existing datasets, and asses the feasibility of existing textual entailment and retrieval-based systems in performing this task. evaluation shows that retrieval-based systems are good at top one prediction however, our proposed answer decomposition and textual entailment-based optimal selection system, adiosaa, performs better when more than one predictions are considered. this further indicates the need for highly abstractive long-form reading comprehension datasets that can foster the development and evaluation of more sophisticated attribution systems.