Model Analysis & Evaluation for Ambiguous Question Answering (2023)

Abstract: Ambiguous questions are a challenge for Question Answering models, as they require answers that cover multiple interpretations of the original query. To this end, these models are required to generate long-form answers that often combine conflicting pieces of information. Although recent advances in the field have shown strong capabilities in generating fluent responses, certain research questions remain unanswered. Does model/data scaling improve the answers’ quality? Do automated metrics align with human judgment? To what extent do these models ground their answers in evidence? In this study, we aim to thoroughly investigate these aspects, and provide valuable insights into the limitations of the current approaches. To aid in reproducibility and further extension of our work, we open-source our code.

## introduction question answering (qa) has been subject to great progress in the past years, largely thanks to the representational capabilities of modern architectures like the transformer @xcite , but also due to the curation of large, high-quality datasets that enabled the effective training of these models @xcite . at the same time, the presence of ambiguous questions has been a challenging aspect of qa. in order to answer such questions, models are required to generate long answers with fluency and cohesion, which is often referred to in the literature as long-form question answering (lfqa). to tackle this, @xcite curated the ambigqa dataset, which contains disambiguations for various questions that were present in popular benchmarks. extending this work, @xcite selected a subset of the questions and crowd-sourced gold answers that cover all possible interpretations of each question, resulting in the asqa dataset. recently, @xcite performed a case study on eli5 @xcite , one of the largest collections available for lfqa, and pointed out several issues that complicate the development and evaluation of suitable models. in particular, the authors questioned whether the retrieved documents are considered by the generative models when producing an answer, and the correlation of the common evaluation metrics with human judgment. in this work, we aim to investigate whether the baselines set on asqa by @xcite suffer from the issues pointed out by @xcite , but also to analyze the modeling choices that contribute to performance. concretely, we set out to answer the following research questions: rq1 does scaling the size of the generative models affect the quality of the generated answers? rq2 can an intermediate round of fine-tuning on non-ambiguous lfqa collections improve performance in ambiguous qa? rq3 when comparing models head-to-head, does human judgment reflect the difference in the automated evaluation metrics? rq4 do models base their answers on the retrieved evidence, or could they be hallucinating? ## methodology we design a standard retrieval-augmented system, to identify the dimensions that contribute to disambiguating questions and generating factual answers. ## experimental setup we aim to assess whether lfqa systems can generate concise answers that disambiguate the provided questions. in this section, we present the datasets and models used, as well as the evaluation metrics. ## conclusion in conclusion, by exploring the lfqa field in answering ambiguous questions, we find that the asqa dataset is a good foundation to develop and evaluate models. we notice that larger generative models produce better answers, with semantic matching for retrieval having a positive impact. to compensate for the small size of the dataset, we experiment with intermediate fine-tuning on larger collections and find that doing so only marginally improves the results. by comparing the performance when using relevant versus random documents, we show the models' dependency on the provided context. our human evaluation confirms the general trends we observed using the automated metrics, giving credit to their disambiguating ability.