Learning Answer Generation using Supervision from Automatic Question Answering Evaluators (2023)

Abstract: Recent studies show that sentence-level extractive QA, i.e., based on Answer Sentence Selection (AS2), is outperformed by Generation-based QA (GenQA) models, which generate answers using the top-k answer sentences ranked by AS2 models (a la retrieval-augmented generation style). In this paper, we propose a novel training paradigm for GenQA using supervision from automatic QA evaluation models (GAVA). Specifically, we propose three strategies to transfer knowledge from these QA evaluation models to a GenQA model: (i) augmenting training data with answers generated by the GenQA model and labelled by GAVA (either statically, before training, or (ii) dynamically, at every training epoch); and (iii) using the GAVA score for weighting the generator loss during the learning of the GenQA model. We evaluate our proposed methods on two academic and one industrial dataset, obtaining a significant improvement in answering accuracy over the previous state of the art.

## introduction recent research on retrieval-based question answering (qa) systems has been focused on two main tasks: (i) answer sentence selection (as2) e.g., @xcite , which, given a question and a list of answer candidates, chooses the most relevant answer that correctly answers the question; and (ii) machine reading (mr) e.g., @xcite , which, given a question and a reference text, involves finding a text span that directly answers the question. while effective, both the strategies (as2 and mr) have some limitations: (i) the text might not include all the information necessary to answer a question, (ii) the text might include unnecessary, distracting information, or (iii) the text expresses the answer in a convoluted (indirect) format. additionally, the text style and sentiment may be inappropriate for answering, or might be structurally too dependent on longer discourse context to enable usage as a stand-alone answer. these drawbacks have motivated researchers to explore text generation systems for writing 'better' answers in the open-domain abstractive qa setting. for example, in the mr domain, rag @xcite ) generates an answer from a set of documents which are selected by dense passage retrieval models. for the domain of as2, previous research has focused on summarizing answers from relevant paragraphs/evidences @xcite , or synthesizing information from the top ranked answer candidates of an as2 system @xcite @xcite . the latter, termed as genqa, has shown improvements in answer generation from the perspective of both answering accuracy and style suitability. the main distinguishing feature of genqa from a generation-based mr approach is the length of the answer: the former uses an entire sentence as the target answer, while the latter in practice uses a short text (primarily targeting entity names). therefore genqa offers a more general and challenging research setting for answer generation. training effective genqa models is made challenging by the cost and difficulty of obtaining largescale, high quality training data. this typically requires human annotators to read the questions and relevant top k retrieved paragraphs/sentences, and then re-write them into a self-contained, and concise natural answer (sentence/paragraph). recent research @xcite has proposed effective automatic qa evaluation models based on transformer encoders for sentence-form answers. training these qa evaluators only requires access to question answer pairs with annotations of correctness/incorrectness of the answers. this style of data annotation is much cheaper to perform than writing high-quality answers for training for genqa models. in this work we explore the novel idea of using automatic qa evaluators for training genqa models, which enables a faster and cheaper design implementation. in this paper, we reduce the amount of data needed for training a genqa model using supervision from automatic qa evaluators. our first contribution is to propose gava: an automatic qa evaluation approach that extends ava @xcite by (i) exploiting multiple reference answers and (ii) evaluating lm-generated answers instead of extracted answers. this way, we obtain a more robust and accurate qa evaluator that can effectively supervise the training of genqa models. we propose three novel methods to use gava for refining the training of genqa. the first consists of (i) generating multiple possible answers using a baseline genqa model for questions belonging to the genqa training dataset, and (ii) then refining the set of generated answers by only retaining those with the highest gava scores (corresponding to "correct" or "high quality" answers). these generated answers are used as alternate gold standard answers (in addition to the annotators' written answers) to create additional training examples for genqa. we term this approach gava-sda (static data augmentation). the second approach extends gava-sda, performing data augmentation dynamically at every epoch instead of off-line before training. this intuitively is more effective as the genqa model continuously improves during the training. specifically, at every epoch, we use gava to score the list of generated answers along with the k input answer candidates. we then use the top scoring answer as the genqa target and the next top-k scoring answers as inputs for genqa. we term this approach gava-dda (dynamic data augmentation). the third approach uses gava as a scoring function for loss weighting during the training of genqa. specifically, we generate an answer using a genqa model for a training sample, and weight the genqa model loss of this instance using the gava score corresponding to the generated answer. intuitively, this makes the genqa model learn more from instances associated with higher gava-scoring answers (which corresponds to "correct" or "high quality" answers). we term this approach gava-lw (loss weighting). we perform empirical evaluation on two academic and one industrial qa dataset (de-identified customer questions from alexa personal assistant), and show that our three proposed techniques using gava for training a genqa model produce significant improvements in answering accuracy over a baseline genqa approach. we also show that the answers generated by these improved genqa models consistently achieve higher gava scores on average than the baseline. we will release the code along with the trained genqa and gava models at https://github.com/amazon-science/ wqa-genqa-gava to enable easy replication of our experimental results. ## related work answer generation: several research works @xcite have studied the problem of generating short answer spans (typically entity level) for mr systems. the most relevant of these works for genqa is the work of @xcite , that trains an answer generation model using the evidentiality of retrieved passages. @xcite uses decoder cross-attention patterns to generate extractive answer spans. @xcite generate answer spans by using a combination of a generative and extractive reader (aggregating information over multiple passages). an independent, but related line of research is questionbased summarization, and there have been several research works in this field: @xcite . @xcite proposed the first formulation for generating complete answer sentences using evidences retrieved by an answer sentence selection (as2) model. this model was termed genqa, and it uses the top-k most relevant answer sentence candidates for a question as input context to an encoder-decoder model to generate a natural sounding complete answer sentence for this question. @xcite extend genqa for multiple languages by using answer sentence candidates from multiple languages as input context for the genqa model. recently, @xcite propose training of genqa models using unlabeled data by leveraging weak supervision from trained as2 ranking models. this approach was shown to combine well with the supervised genqa approach @xcite to improve the answering accuracy. note that all of these approaches are different from the ones described in the previous paragraph as they aim to generate complete answer sentences, and not just short answer spans. evaluation of qa systems: token level simi-larity metrics like bleu @xcite ), rouge @xcite , meteor @xcite , etc have been shown @xcite to not extend to sentence-form qa evaluation. for mr tasks, @xcite adapt bleu and rouge metrics, but limit their evaluation to only yes-no and entity questions. @xcite uses multiple gold reference answers (extracted from knowledge bases) to be used as references for evaluating answer span extraction. there have been several learnable automatic metrics: bertscore @xcite , comet @xcite , bleurt (sellam et al., 2020), etc. that have been proposed for some tasks in nlp such as mt and summarization. these are based on transformer encoder models. chen et al. (2019) proposed extending bertscore for mr tasks using the question and paragraph context in addition to the answer. in similar line of work, @xcite propose ava which is an automatic qa evaluation metric that learns a transformer to encode the question, a reference gold answer and the target answer to be evaluated. very recently, @xcite also present similar findings as ava, by proposing bem which can be used for evaluating sentence-level extractive qa (as2). ava and bem have not been evaluated for genqa systems previously. hsu et al. ( 2021 ) and @xcite show that automatic metrics like bleu, bleurt, bertscore do not correlate well with human judgements for evaluating accuracy of genqa systems. we extend ava for our experiments as the automatic qa evaluation system. ## automatic qa evaluation using multiple references (gava) @xcite propose ava: an automatic evaluation models for qa based on a transformer encoder. it is applied to a question and a complete answer sentence to determine the correctness or incorrectness of the answer. formally, we denote the ava model with e, which takes as input a question q, a target answer a, and a reference r, i.e., gold standard (gs) answer, and outputs a correctness probability score, @xmath0, 1]. ava is trained on the same labeled data of as2, i.e., question answer pairs, where each question has multiple annotated answer candidates available. though the ava approach was empirically shown to be accurate for evaluating as2 systems @xcite , there are some limitations associated with it: (i) several questions may have diverse correct answers, e.g., "tell me a winner of the us open?", and (ii) the same answer may be expressed in very different formats, e.g., "how old is joe biden?", "biden is 80 years old" v/s. "the president has just entered his life's eighth decade". furthermore, ava does not use negative references when evaluating correctness of answers, while incorrect answers can also help refine the prediction of correctness/incorrectness. note that most as2 datasets have multiple annotated answers (combination of correct and incorrect labels), and thus it is straightforward to use them for building data to train ava with multiple positive and negative references. intuitively, a genqa system synthesizes an answer using different pieces of information spread across many relevant candidates (while suppressing any irrelevant information), aligning well with the idea of using multiple references for qa evaluation. we term this approach: gava (ava for generation-based models), which uses multiple references (combining positive and negative references) {r 1 , r 2 , . . . , r n }. fig 1 shows the gava architecture: which uses a transformer encoder, taking as input: a question q, a target answer a, and n references. the information about the nature of the positive/negative references is encoded by prepending each reference with a prompt indicating the type of reference it is. ## gava for training genqa in this section, we propose three approaches for training genqa models using gava. ## experiments 6.1 datasets for training and evaluating our models, we consider two academic and one industrial dataset representing real world customer questions. wqa web question answers (wqa) is a public dataset defined in @xcite . the dataset contains 149,513 questions, each associated with about 15 answer candidates. both questions and answers are retrieved from a large-scale web index. each qa pair is manually annotated for answer correctness by professional annotators. ms- @xcite proposed ms-marco, originally for mr tasks, comprising ∼800k queries retrieved from the bing search engine along with ∼10 labeled answer passages. following @xcite , we transform ms-marco to obtain a large dataset of qa pairs, where the answers are sentences and not passages/paragraphs. using a sota as2 model (debertav3-xl @xcite trained on the asnq @xcite ) dataset), we pick the top-2 ranked answer sentences from a positively labeled passage as positive answer candidates for the question. similar to @xcite , we randomly sub-sample 1k questions from the dev. set for evaluation (we use human annotations for our experiments and using the entire 100k dev. set would be extremely expensive to annotate). iqad industrial qa dataset @xcite ) is a large scale internal industrial qa dataset containing nonrepresentative de-identified user questions from alexa personal assistant. iqad contains ∼10k questions, each having ∼200 answer candidates retrieved using a large scale web index (over 100m documents). each question has ∼15 answer candidates with human annotations of correctness/incorrectness. results on iqad are presented relative to a baseline due to the data being internal. ## conclusion in this paper we propose a novel training paradigm of learning answer generation systems (genqa) using supervision from automatic qa evaluation met-