Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering (2024)

Abstract: With the enhancement in the field of generative artificial intelligence (AI), contextual question answering has become extremely relevant. Attributing model generations to the input source document is essential to ensure trustworthiness and reliability. We observe that when large language models (LLMs) are used for contextual question answering, the output answer often consists of text copied verbatim from the input prompt which is linked together with “glue text” generated by the LLM. Motivated by this, we propose that LLMs have an inherent awareness from where the text was copied, likely captured in the hidden states of the LLM. We introduce a novel method for attribution in contextual question answering, leveraging the hidden state representations of LLMs. Our approach bypasses the need for extensive model retraining and retrieval model overhead, offering granular attributions and preserving the quality of generated answers. Our experimental results demonstrate that our method performs on par or better than GPT-4 at identifying verbatim copied segments in LLM generations and in attributing these segments to their source. Importantly, our method shows robust performance across various LLM architectures, highlighting its broad applicability. Additionally, we present Verifiability-granular, an attribution dataset which has token level annotations for LLM generations in the contextual question answering setup.

## introduction the surge in the capabilities of large language models (llms) has revolutionized natural language understanding. their ability to comprehend and generate human-like text has resulted in their widespread adoption across various industries. a prominent and pivotal application of these models 1 dataset is available at https://github.com/ anirudh-phukan/verifiability-granular . despite the remarkable strides made in contextual question answering, challenges persist within llms. while llms excel in generating informative responses, they often fall short in providing explicit references or attributions to the specific sections or sources within the context from which their answers derive @xcite . the absence of this attribution impedes the ability to verify the authenticity and accuracy of the generated information. attribution not only enables verification @xcite but also increases user trust and confidence in the responses generated by these models, thus fostering their broader acceptance and utilization across diverse domains @xcite . existing methods for attribution typically align within three primary categories @xcite : systems fine-tuned or trained explicitly for attribution tasks @xcite , retrieve-then-read @xcite and approaches relying on post-generation attribution @xcite . however, each of these approaches encounters substantial challenges in achieving accurate and granular attribution, thereby limiting their effectiveness. the initial category, which consists of systems tailored or explicitly trained for attribution tasks, poses a significant challenge @xcite . this challenge primarily arises because the training process demands extensive resources, including time, computational power, and a large corpus of annotated data. furthermore, the continuous development of more sophisticated models requires retraining, perpetuating an unending cycle of adaptation. additionally, the need for regression testing to ensure ## related work researchers have employed different approaches for the attribution of generated or identified text spans. a shared task was organized by @xcite to encourage researchers to build systems capable of performing fact verification and attribution to the source texts. @xcite emphasized the role of source attribution in fostering truthful and responsible ai. the growing concern of fake news detection in ai-based news generation has also been considered as an attribution task @xcite . in addition to automatic attribution, studies on manual attribution have been performed by domain specific individuals (borel, 2023; @xcite . while we discuss about fact verification as an attribution task, it is important to note that user interactions have also been found to require attribution @xcite . @xcite has highlighted the fact that wikipedia articles needs validation and that the citations need to be attributed. on the other hand @xcite introduced a python library based on gpt-2 capable of identifying feature attributions generated from the insequence model; capable of performing in multilingual settings @xcite . recently, researchers performed attribution task on multimodal systems as well @xcite @xcite . however, what is missing in prior art is the utilisation of the contextual guidance inherently present during generation for the attribution task. in our work, we tackle this problem by utilising the contextual information encoded during generation by the llm and attribute spans to semantically relevant parts of the source document. ## problem statement we observe a phenomenon in contextual question answering using llms which results in a comprehensive answer typically characterized by factual spans, replicated verbatim from various segments of the provided context, interwoven with "glue text". an example of this phenomenon is shown in figure 2 . @xcite also note this pattern and use references to losslessly speed up llm inference. building upon the aforementioned observations, we dissect the challenge of attribution in the contextual generation setting into two distinct yet interconnected sub-problems. the first sub-problem involves the identification of tokens within the output that have been directly copied from the provided context. the second sub-problem delves deeper into the attribution of these identified tokens. this step involves mapping these tokens back to their original positions within the document. in essence, this implies tracing back the tokens to the exact sections within the document from which they were copied verbatim. by doing this, we aim to establish a clear flow of information from the source document to the generated output, thereby enabling a more nuanced understanding of the context generation process. the task at hand involves two distinct sub-tasks within the context of attributing the output of a large language model (llm). we are given a document d represented as a sequence of tokens (d 1 , d 2 , . . . , d n ), a question q, and an answer a represented as a sequence of tokens (a 1 , a 2 , . . . , a m ). our objective comprises two interconnected subtasks: subtask 1: token attribution identification. this subtask involves identifying a subset of tokens in a, denoted as a attr ⊆ a, that require attribution to tokens in d. mathematically, this can be expressed as selecting tokens a @xmath0. in our current study, we restrict the scope to tokens that are verbatim copied from d. subtask 2: corresponding token mapping. for each token in a attr , this subtask aims to find a mapping function f : a @xmath1, there exists a corresponding token or sequence of tokens in d to which a i can be attributed. formally, the mapping function can be expressed as: in this formulation, the function f establishes a connection between the tokens in the answer requiring attribution and their corresponding tokens or token sequences in the document d. ## experimental setup in this section, we outline the details of our experimental setup; the dataset, the evaluation metrics; and the baselines. ## results and analysis we perform experiments for our approach ( § 4.2) with the following llms: llama-7b @xcite , llama-70b @xcite , mistral-7b (jiang et al., 2023), yi-6b @xcite and opt-350m @xcite . note that opt-350m has been excluded from comparision on verifiability-granular dataset due to its limitation of 2048 token context length and the dataset contains samples upto 4096 tokens. all of our experiments were run on a a100 machine with 4 80gb gpus. ## discussion & conclusion through this study, we introduce a novel, efficient solution to the challenge of attribution in contextual question answering. our method leverages the hidden state representations of llms, providing detailed, granular attributions without requiring extensive model retraining. our approach's ability to perform token-level attribution easily lends itself to end-user applications, and we have included an actual system input-output in the appendix (section a.6) depicting the same. our experimental results, across various model configurations and layers, demonstrate the efficacy of our approach. notably, our method performs on par or better than training-free baselines in both identifying tokens in an answer that are directly copied from the context and attributing these tokens to their original positions in the document. additionally we make available verifiability-granular dataset, which contains token-level attributions for llm generations in the contextual question answering setup. perhaps one of the most interesting findings from our study is that the ability to identify extracted tokens and attribute them back to the source is not exclusive to a specific model family. this suggests that our approach can be applied broadly across different llm architectures, underscoring its versatility.