Adaptive Question Answering: Enhancing Language Model Proficiency for Addressing Knowledge Conflicts with Source Citations (2024)

Abstract: Resolving knowledge conflicts is a crucial challenge in Question Answering (QA) tasks, as the internet contains numerous conflicting facts and opinions. While some research has made progress in tackling ambiguous settings where multiple valid answers exist, these approaches often neglect to provide source citations, leaving users to evaluate the factuality of each answer. On the other hand, existing work on citation generation has focused on unambiguous settings with single answers, failing to address the complexity of real-world scenarios. Despite the importance of both aspects, no prior research has combined them, leaving a significant gap in the development of QA systems. In this work, we bridge this gap by proposing the novel task of QA with source citation in ambiguous settings, where multiple valid answers exist. To facilitate research in this area, we create a comprehensive framework consisting of: (1) five novel datasets, obtained by augmenting three existing reading comprehension datasets with citation meta-data across various ambiguous settings, such as distractors and paraphrasing; (2) the first ambiguous multi-hop QA dataset featuring real-world, naturally occurring contexts; (3) two new metrics to evaluate models’ performances; and (4) several strong baselines using rule-based, prompting, and finetuning approaches over five large language models. We hope that this new task, datasets, metrics, and baselines will inspire the community to push the boundaries of QA research and develop more trustworthy and interpretable systems.

## introduction knowledge-enhanced large language models (llms) have demonstrated remarkable questionanswering (qa) capabilities, partially due to their ability to reason over a substantial number of tokens @xcite some work has shown that llms do not fully utilize long sequences @xcite , an issue that arises from the context itself is that knowledge is dynamic and is constantly changing, and hence, conflicting facts and opinions may exist within it @xcite . for example, since politicians change, there exist documents each expressing that a different person is the current president of the united states. this is especially problematic for models that can handle long contexts, such as existing state-ofthe-art models @xcite @xcite @xcite and retrieval-augmented generation (rag) systems @xcite @xcite , as the greater the length of the context, the higher the probability of encountering conflicting information from various sources, domains, or even time periods within the same source or domain. existing work on qa in contextual foot_0 knowledge conflicts 2 setting mitigate this issue by either predicting all valid answers @xcite , aggregating all answers @xcite , asking clarification questions @xcite , and other methods @xcite . however, these methods burden users with the task of extensively evaluating the factuality of each answer @xcite @xcite , as they do not cite the source of the answer. hence, it is crucial to develop systems that not only generate all possible answers, but produce a distinct response for each conflicting information while also citing their sources, as shown in figure 1 . this, in turn, will also lead to an increase in users' trust and interpretability @xcite . and while some existing work develop models that cite their sources, they only focus on unambiguous setting, where only one answer exists @xcite @xcite . furthermore, none of the existing work on contextual conflicts or citation generation focus on complex qa settings, which require multi-hop reasoning and many answers, and resemble a more realistic real-world setting @xcite @xcite we bridge the gap between ambiguous qa and citation generation by proposing the novel task of qa with source citation in ambiguous settings, where multiple valid answers exist. to facilitate research, we provide a comprehensive framework featuring: five novel datasets with citation metadata, the first ambiguous multi-hop qa dataset, two new evaluation metrics, and strong baselines. our goal is to inspire the community to push the boundaries of qa research and develop more trustworthy and interpretable systems. knowledge conflicts prior work on qa in knowledge conflicts settings falls into two categories: ambiguous questions, where answers vary depending on the question phrasing @xcite @xcite , and ambiguous context, where answers vary depending on the context provided @xcite @xcite . however, existing work lacks two key aspects: 1) source citation, and 2) evaluation in complex, real-world settings requiring multi-hop reasoning. citations while llms contain various types of factual knowledge within their parameters @xcite @xcite @xcite , they often suffer from hallucinations, generating non-factual text @xcite @xcite . to address this, researchers have developed models that can provide citations @xcite @xcite @xcite . however, these models have not been tested in complex scenarios, such as ambiguous qa settings with multiple possible answers or multi-hop reasoning. ## experiments in our experiments, each dataset consists of triples (q, [c 1 , ..., c n ], [a 1 , ..., a k ]), where q is a question, [c 1 , ..., c n ] are multiple context documents, and [a 1 , ..., a k ] are at least two conflicting answers. we follow prior work @xcite @xcite by concatenating the question and contexts into a single string, which is then input to each model. ## results example model generations are shown in ## real-world usage in our comprehensive analysis, we evaluate three main approaches to improve llms' ability to answer ambiguous questions with source citations: 1) prompt-based; 2) rule-based, and 3) fine-tuningbased. notably, while the rule-based approach outperforms the other two in some occasions, as discussed in section 3.4.2, it is incapable of answering questions that require complex reasoning across multiple documents, as it only sees one document at a time. to that end, we do not recommend using this approach when it is known that the data is of complex nature. but, to use it, users need to split the retrieved documents into chunks of one document at a time, which are sent to the model, followed by an aggregation of the answers. with regards to the other two approaches, the prompt-based approaches can be incorporated into most llms with a simple addition of a prompt, as shown in appendix a. however, it is worth mentioning that the fine-tuning approach outperforms the prompting approach on multihop reasoning, but also results in a large performance decrease when no ambiguity exists, as discussed in section 4.4. we also showed that lora-based fine-tuning is sufficient to improve llms' abilities in this task greatly over the baseline, highlighting the usability for real-users that do not have large computational resources. ## future work having established the novel task of qa with source citation in ambiguous settings, and proposing strong baselines across our newly created datasets, several promising directions for future research emerge. future work can build upon our contributions by: 1) investigating the impact of finetuning on datasets with distractors, as our current experiments only focused on the no-distractor setting; 2) extending our finetuning data to include both conflicting and non-conflicting instances, which could lead to more robust models capable of handling varying levels of ambiguity; 3) exploring the effectiveness of finetuning on paraphrased data instead of duplications, which may provide additional insights into the model's ability to generalize across different linguistic formulations; 4) designing alternative prompts that emphasize the importance of citations or exploring other citation methods to further enhance the model's performance; 5) creating a more diverse range of datasets that capture different aspects of ambiguity, beyond the three flavors (paraphrasing, distractors, and duplications) we experiment with in this work; 6) investigating the potential benefits of employing different architectures or larger language models to tackle this challenging task; and 7) pretraining models on conflicting data and citations to potentially improve their ability to resolve knowledge conflicts and provide trustworthy answers. ## conclusion we address a significant gap in qa research by introducing the novel task of qa with source ci-tation in ambiguous settings. this task combines the complexities of ambiguous qa with the importance of providing source citations, enabling users to evaluate the factuality of each answer. to facilitate research in this area, we provide a comprehensive framework consisting of novel datasets, new evaluation metrics, and strong baselines using various approaches over 5 llms. our work aims to inspire the development of more trustworthy and interpretable qa systems, bridging the gap between ambiguous answer resolution and citation generation. by exploring this new task, we hope to pave the way for more reliable and transparent qa systems that can accurately resolve knowledge conflicts and provide users with credible sources to support their answers.