Modeling Exemplification in Long-form Question Answering via Retrieval (2022)

Abstract: Exemplification is a process by which writers explain or clarify a concept by providing an example. While common in all forms of writing, exemplification is particularly useful in the task of long-form question answering (LFQA), where a complicated answer can be made more understandable through simple examples. In this paper, we provide the first computational study of exemplification in QA, performing a fine-grained annotation of different types of examples (e.g., hypotheticals, anecdotes) in three corpora. We show that not only do state-of-the-art LFQA models struggle to generate relevant examples, but also that standard evaluation metrics such as ROUGE are insufficient to judge exemplification quality. We propose to treat exemplification as a retrieval problem in which a partially-written answer is used to query a large set of human-written examples extracted from a corpus. Our approach allows a reliable ranking-type automatic metrics that correlates well with human evaluation. A human evaluation shows that our model’s retrieved examples are more relevant than examples generated from a state-of-the-art LFQA model.

## introduction when an author introduces a complicated concept, they commonly follow it up with a concrete example to help clarify their intended meaning. this process, known as exemplification, occurs in diverse forms, including hypothetical examples, personal anecdotes, and analogies @xcite . exemplification is particularly common within the nlp task of long-form question answering (lfqa), where an author wants to communicate a concept unknown to the question asker. consider the following qa pair from the r/explainlikeimfive subreddit @xcite : q: how does the ground not cave in while being under the heavy weight of cities? a: it's all about what they're building on, and occasionally they get it wrong... for example, san francisco's millennium tower, built on mud and sand, has already sunk 18 inches into the ground. here, the answerer uses a specific example to emphasize the importance of building on a solid foundation. in general, explaining via exemplification is a fundamental technique in these kinds of pedagogical scenarios @xcite , and it warrants separate study due to its importance in lfqa and the challenges in evaluating it. however, existing work on building lfqa models @xcite does not give special treatment to exemplification or any other discourse phenomena, choosing instead to evaluate model outputs against reference answers using metrics such as rouge @xcite that are not meaningful for this task @xcite . in the above qa pair, any other structurallyunstable building (e.g., the leaning tower of pisa) could serve as a valid example, but an lfqa model would be unfairly penalized for generating one of these acceptable alternatives. in this paper, we first conduct a detailed study of exemplification across three different domains: wikipedia, web articles, and community answers to questions from the eli5 lfqa dataset. we extract sentences and clauses associated with exemplification by matching on explicit markers such as "for example" and "(e.g., ...)" and annotate 300 examples from this dataset. our analysis reveals significant variation in occurrence frequencies of different forms of exemplification (e.g., hypothetical vs. specific examples) across the three domains. next, we focus on improving the modeling and evaluation of the subtask of exemplification within lfqa. we propose to treat it as a retrieval problem rather than a generation problem: given a question and a prefix of a reference answer (in the above qa pair, all of the text before "for example"), a model must retrieve the ground-truth example that follows the prefix (the sentence about the millenium tower) from the set of all exemplifying sentences and clauses in the dataset. we can use retrieval metrics such as recall@k to evaluate a model's ability to select the ground-truth example, which are more informative than metrics such as rouge. we demonstrate that pretraining our retriever on a large-scale dataset of exemplifying units extracted from the books3 corpus @xcite and then fine-tuning it on eli5 examples results in substantial improvements on these ranking metrics. finally, we crowdsource human evaluation comparing our retriever's outputs with those generated by the state-of-the-art eli5 model of @xcite and find that workers prefer the retriever's outputs far more frequently than those of the generation model. we hope that our work spurs more research into the evaluation and modeling of exemplification and other complex discourse phenomena present in lfqa. to facilitate future research, we publicly release the code and trained models from our work. foot_0 ## exploring exemplification in this section, we first describe our data extraction process, which we use to mine instances of exemplification from three datasets (and domains): eli5 @xcite , natural questions @xcite and books3 @xcite . this process involves matching on a set of "exemplification markers" and collecting both the text of the matching example (a sentence or clause) as well as the surrounding context on both sides. we then conduct a fine-grained human annotation study on the extracted data, breaking exemplification down into different types and exploring how they are used across the different domains. ## retrieving examples in lfqa our annotation and analysis of the extracted exemplification dataset reveals the diversity and complexity of this discourse phenomenon. in this section, we shift our focus towards building retrievalbased models to produce examples based on a given lfqa context. first, we define our exampleretrieval task formally and introduce our evaluation metrics. then, we describe our contrastive-learning based retriever model, which closely follows the retriever implementation in @xcite , and baseline retrieval models, and report their performances. ## results & analysis we report results from baseline retrievers and our trained models in ## conclusion in this work, we present the first computational study of exemplification in long-form question answering. we perform a detailed annotation over the use of exemplification across various domains and observe different distributions over complex exemplification types and units. while existing lfqa systems are conditional language models that do not give special treatment to exemplification, we propose to retrieve examples based on their context instead of generating them. we develop egret, a simple dual encoder trained with a contrastive learning objective, that outperforms a diverse set of baselines on this task of example retrieval, which we can meaningfully evaluate using simple ranking metrics instead of unsuitable metrics like rouge. we hope that our work spurs researchers to consider separately modeling and evaluating the fine-grained linguistic and discourse phenomena found in lfqa data.