Answer Consolidation: Formulation and Benchmarking (2022)

Abstract: Current question answering (QA) systems primarily consider the single-answer scenario, where each question is assumed to be paired with one correct answer. However, in many real-world QA applications, multiple answer scenarios arise where consolidating answers into a comprehensive and non-redundant set of answers is a more efficient user interface. In this paper, we formulate the problem of answer consolidation, where answers are partitioned into multiple groups, each representing different aspects of the answer set. Then, given this partitioning, a comprehensive and non-redundant set of answers can be constructed by picking one answer from each group. To initiate research on answer consolidation, we construct a dataset consisting of 4,699 questions and 24,006 sentences and evaluate multiple models. Despite a promising performance achieved by the best-performing supervised models, we still believe this task has room for further improvements.

## introduction open-domain question answering (qa) systems @xcite aim to answer natural language questions using large collections of reference documents, contributing to real-world applications such as intelligent virtual assistants and search engines. current qa systems @xcite usually adopt a three-stage pipeline consisting of: (1) a passage retriever @xcite ) that selects a small set of passages relevant to the question, (2) a machine reader that examines the retrieved passages and extracts @xcite or abstracts @xcite the candidate answers, and (3) an answer reranker @xcite ) that fuses features from previous stages to either select one final answer or return the top-ranked answer from the previous stage. current qa research @xcite primarily examines the case where each question is assumed to have a single correct answer. however, in practice, many questions can have multiple correct answers. for example, the question "is coffee good for your health?" can be answered with respect to different aspects (e.g., "coffee can help you with weight loss", "coffee can cause insomnia and restlessness"). to correctly identify different aspects of answers to the same question while mitigating aspect-level redundancy, it is important to consolidate the answers. answer consolidation is particularly desirable for applications such as intelligent assistants, where responses are desired to be both comprehensive and concise. additionally, in scenarios where qa is used for knowledge extraction @xcite or claim verification @xcite , consolidation is also an essential step to identify salient knowledge or evidence while mitigating duplication. to effectively recognize multiple aspects of answers in qa systems, our first contribution is to introduce and formalize the answer consolidation problem. specifically, given a question paired with multiple answer snippets, answer consolidation first partitions the snippets into groups where each group represents a single aspect within the answer space. once partitioned, the final answer set is produced by returning a representative snippet from each group. in this formulation, the answer consolidation task is a post-processing stage that takes predicted answer-mentioning snippets (in this work, sentences) from previous qa stages and produces an answer set that maximizes answer aspect coverage while minimizing answer duplication. to foster research on the answer consolidation problem, our second contribution is the collec-tion of a new dataset, namely, quasi (question-answer consolidation). quasi consists of 4,699 questions such that each question is paired with multiple answer-mentioning sentences grouped according to different aspects. starting with a quorabased question source @xcite , noting the potential for multi-aspect answers, we first retrieve 10 relevant answer sentences from the web for each question. these sentences are then examined by three crowd-sourced workers, who exclude sentences that do not contain an answer and group the remaining ones. finally, individual sentence groupings from different workers are aggregated into a single partitioning. quasi consists of 24,006 sentences and 19,676 groups, corresponding to an average number of 4.18 aspects per question and 1.22 sentences per group. our third contribution is a comprehensive benchmarking for the answer consolidation problem based on quasi. specifically, we consider two evaluation settings: (1) classification, where the model predicts whether two sentences are in the same group. (2) sentence grouping, where the model groups the answer sentences. we evaluate a wide selection of zero-shot and supervised methods, including various sota sentence embedding models @xcite , cross-encoders @xcite , and a newly proposed answeraware cross-encoder model. in the supervised setting, the answer-aware cross-encoder achieves the best results based on the matthew correlation coefficient (mcc) score of 87.8% (classification setting) and an adjusted mutual information (ami) score of 68.9% (sentence grouping setting). as this performance is notably far from perfect, our findings indicate the need for future investigation on this meaningful, but challenging, task. answer summarization. questions with multiple answers are common in online communities. for example, @xcite observe that no more than 48% of best answers on yahoo! answers are unique. many efforts @xcite @xcite have been devoted to summarizing reusable answers in community qa. particularly, answer-summ @xcite proposes a dataset where different answers are rewritten to bullet points by humans. while training on summarization data may enable the model to return salient and non-redundant answers, such training only works for abstractive machine readers. a more related work is bert-ddp @xcite , which considers the problem of getting a diverse and nonredundant answer set. they construct a dataset based on yahoo! chiebukuro where workers are asked to provide an answer set given a question. however, the correct answer set is not unique when answers are equivalent. as they treat all but the annotated answer set as wrong, both training and inference are prone to false negatives. in this paper, we group the answers with respect to their aspects and provide a discriminant rule, such that the correct group assignment is unique. diverse passage retrieval. many information retrieval efforts address the problem of retrieving diverse documents for a query @xcite @xcite . in qa, @xcite examine answer diversity in passage retrieval and propose a self-supervised dynamic oracle training objective. however, as passages may contain irrelevant information to the question, the retriever faces the challenge of identifying and integrating answers in passages when assessing answer diversity. in this work, we consider a dedicated task of answer consolidation and leave the problem of identifying answers to previous qa stages. basic concepts. when answering a specific question, different answers may be given regarding different perspectives, opinions, angles, or parts of the overall answer. we regard such answers as those pertaining to different aspects. furthermore, we refer to two sentences as equivalent if they contain the same answer aspect(s) and distinct if they express different answer aspect(s). to better identify equivalent/distinct sentences, we propose the following operational discriminant rule: given two answer-mentioning sentences s 1 , s 2 for the same question q, we can rewrite the answers contained in s 1 and s 2 into yes-no questions q ′ 1 and q ′ 2 , which can be answered by yes/no/irrelevant. foot_0 then, if s 1 and s 2 give coherent answers of yes/no foot_1 to q ′ 2 and q ′ 1 , respectively, then s 1 and s 2 are considered to represent equivalent aspects. otherwise, they are considered distinct from each other. we take the following example: q: is coffee good for your health? s1: coffee can make you slim down. s2: coffee can relieve headache. s3: coffee can help with weight loss. then we rewrite the answers contained in sentences as the following questions: q'1: can coffee make you slim down? q'2: can coffee relieve headache? q'3: can coffee help you with weight loss? we can tell that s1 and s3 are equivalent, as they both give coherent answers (yes) to each others' yes-no questions. we can also tell that s2 is distinct from s1 and s3, as it gives irrelevant answers to q'1 and q'3. ## question-answer consolidation dataset (quasi) in this section, we describe the creation of our question-answer consolidation dataset (quasi) including corpus collection ( §4.1) and dataset annotation ( §4.2). we then provide statistical and quantitative analysis of quasi ( §4.3). ## approach in this section, we first tackle the classification setting of answer consolidation ( §5.1). given a question and answer-mentioning sentences, the task is to predict for a pair of sentences whether they are in the same group. we consider different types of models, including sentence embedding models, cross-encoders, and answer-aware cross-encoders. then we consider the sentence grouping setting, presenting the method of transforming pairwise predictions to sentence groups ( §5.2). for all methods, we use roberta large @xcite as the encoder, noting that other pretrained language models (plms) can easily be incorporated as part of these methods. ## experiments in this section, we present the experimental setup ( §6.1), show the main results ( §6.2), conduct an ablation study ( §6.3), and provide error analysis ( §6.4). ## conclusion in this paper, we formulate and propose the answer consolidation task that seeks to group answers into different aspects. this process can be used to construct a final set of answers that is both comprehensive and non-redundant. we contribute the question-answer consolidation dataset (quasi) for this task and evaluate various models, including sentence embedding models, cross-encoders, and answer-aware cross-encoders. while the best-performing supervised models have achieved promising performance, without that abundant an-notation, unsupervised methods still remain far from perfect. this suggests room for further studies on more robust and generalizable solutions for answer consolidation that would largely benefit real-world open-domain qa systems.