Machine Reading Comprehension using Case-based Reasoning (2023)

Abstract: We present an accurate and interpretable method for answer extraction in machine reading comprehension that is reminiscent of case-based reasoning (CBR) from classical AI. Our method (CBR-MRC) builds upon the hypothesis that contextualized answers to similar questions share semantic similarities with each other. Given a test question, CBR-MRC first retrieves a set of similar cases from a nonparametric memory and then predicts an answer by selecting the span in the test context that is most similar to the contextualized representations of answers in the retrieved cases. The semi-parametric nature of our approach allows it to attribute a prediction to the specific set of evidence cases, making it a desirable choice for building reliable and debuggable QA systems. We show that CBR-MRC provides high accuracy comparable with large reader models and outperforms baselines by 11.5 and 8.4 EM on NaturalQuestions and NewsQA, respectively. Further, we demonstrate the ability of CBR-MRC in identifying not just the correct answer tokens but also the span with the most relevant supporting evidence. Lastly, we observe that contexts for certain question types show higher lexical diversity than others and find that CBR-MRC is robust to these variations while performance using fully-parametric methods drops.

## introduction machine reading comprehension (mrc) aims to measure the ability of models to understand and reason over a specified sequence of text. in the extractive setting, the task requires a model to answer a question by reading a set of one or more passages, referred to as the context, and identifying a span of text from that context as the answer. figure 1 : overview of cbr-mrc. top: given a test question, cbr-mrc first retrieves similar cases from a memory of known (question, context, answer) triples. bottom: cbr-mrc scores each candidate answer span by comparing its contextualized representation with the answer spans of the retrieved cases to output a score. the candidate with the highest aggregate score with the answer spans from the cases is output as the prediction ("charles babbage"). current state-of-the-art machine readers @xcite @xcite @xcite use fine-tuned transformer-based models @xcite . however, the prediction mechanisms in these models are largely opaque to humans, limiting their interpretability and maintainability in realworld scenarios @xcite @xcite . this becomes particularly relevant when practitioners need to determine the cause of erroneous predictions, for instance, to patch models deployed in production systems. moreover, to build reliable qa systems, it is essential for models to not only predict the correct answer but also provide evidence to support the predictions @xcite . to address these limitations, we introduce cbr-mrc -a semi-parametric approach for mrc that makes predictions by explicitly reasoning over a set of retrieved answers for similar questions and their contexts from a non-parametric memory. our key idea is that contextualized answers to similar questions share representational similarities. we posit that a (question, passage, answer) triple captures a set of latent relations, which can be understood to represent the types or intents present within a question given the accompanying context. we use the intersection of latent relation sets between questions to learn a similarity function that can then be used to predict answers to questions with similar intents. more concretely, our approach first compares the contextualized representations of candidate answer spans in the target context with gold answer spans of similar questions from memory. then, the candidate with the highest aggregate similarity with the gold case answer spans is selected as the predicted answer. for example, in figure 1 , the question "who invented the computer?" shares the same latent relations with the question "who is the inventor of the telephone?", and the contextualized span similarity with the case answer "graham bell" is maximized when compared with the correct answer "charles babbage" and low when compared with other candidate spans such as "mechanical computer". cbr-mrc adapts the case-based reasoning @xcite paradigm from classical ai @xcite @xcite , which typically consists of four steps -retrieve, reuse, revise, and retain. cbr-mrc uses the first tworetrieval from a memory of cases, or casebase, using the embedding of the test question followed by reusing reasoning patterns encoded in the contextualized answer embeddings of the retrieved cases to select the most similar span from the test context as the predicted answer. to the best of our knowledge, we are the first to propose a novel adaptation of the cbr framework for the task of question answering over unstructured text. we empirically verify cbr-mrc on naturalquestions and newsqa, two common mrc datasets, and find our method outperforms strong baselines by up to 11.53 em. the inference procedure of cbr-mrc is not just accurate but also interpretable -we know which cases the model uses for its prediction and how much each case contributes. further, the semi-parametric nature of cbr-mrc allows for efficient and accurate domain adaptation by simply adding new cases to the casebase without the need to modify any parameters. 2 lastly, we also find that our method shows higher gains compared to fully-parametric models when the training set contains lexically diverse contexts for questions with similar latent relations, demonstrating the robustness of cbr-mrc when questions may be answered using several different passages. in summary, our contributions are as follows: charles babbage is credited with inventing the first mechanical computer in 1833. graham bell a: david alter c: dr. david alter invented the electric telegraph in 1836 a: graham bell c: graham bell is credited with patenting the 1st practical telephone q: who invented the telephone? similar question pairs and their (cached) contexts q: who invented the telegraph? david alter who invented the computer? 3. similarity matching and score aggregation to find most similar spans charles babbage mechanical computer 1833 charles babbage contextualized answer spans for inner-product case embedding latent relation span embedding query embedding computed at inference 2. candidate answer span extraction test question and context 1. case retrieval finally, the span with the highest inner-product similarity is selected as the prediction. note that the casebase questions and their corresponding answer span embeddings are pre-computed and cached. our work focuses on the extractive mrc setting, which aims to identify the correct answer span within a passage given a test question. interpretable methods for mrc have also seen much recent interest. @xcite highlights the challenges facing mrc in terms of explainability and its impact on mrc performance. several methods @xcite @xcite have attempted to improve interpretability in mrc by identifying the supporting sentence for the answer. however, these methods rely on the availability of gold-supporting sentences, which may not always be present. other methods utilize graph networks @xcite and attention mechanisms to capture the attended parts of text or relations @xcite . in this work, we improve interpretability by providing a novel inference procedure that allows us to clearly attribute predictions to previously seen instances that were used in the decision-making process. case-based reasoning. previous work @xcite @xcite has demonstrated strong performance using cbr on knowledge graph reasoning tasks. cbr has also been successfully applied to the task of semantic parsing @xcite @xcite , where methods typically retrieve the logical forms of similar cases and pass them through a seq2seq model as input along with a target query. both settings, however, operate over structured spaces, where similarities between instances can easily be determined by collecting and comparing symbolic patterns in the data, e.g., paths between nodes in a kg, or syntactic similarities in semantic parsing. in this work, however, we adapt cbr to operate over unstructured text, and it is not obvious how previous approaches can be re-used to work in this setting out-of-the-box. a concurrent work @xcite performs answer sentence selection by using graph neural networks to model interactions among questions and answers, assuming that for each questionanswer, there exist similar question-answers in the training data. we instead focus on the more finegrained task of answer span prediction from contexts and use transformer models to score questionquestion and answer-answer similarities between each retrieved case and a test example. in-context learning (icl). icl is an emergent property observed in large language models (llms) @xcite showing remarkable success in several nlp tasks @xcite @xcite . to make predictions, a model is simply prompted with demonstrations for a target task and a test instance. recent work @xcite @xcite has shown strong performance by retrieving relevant demonstrations. this setting can be understood as a specific instantiation of the case-based reasoning framework, where the "reuse" step uses icl instead of a fine-tuned model. icl, however, is sensitive to the prompt @xcite , leading to black-boxed inference and reducing reliability of predictions. problem decomposition and iterative prompting methods, such as chain-of-thought @xcite and least-to-most @xcite , attempt to address this issue by converting complex problems into intermediate steps. in contrast to methods reliant on emergent pre-training behavior, our work explicitly encodes the relational similarity hypothesis in a trained model to reuse exemplars for making predictions. ## method we now formally describe cbr-mrc (figure 2 ). a case in cbr consists of a problem and its solution. formally, we define a case c as a tuple {q, a, p}, representing a question, its corresponding answer set foot_1 , and the passage context that contains each element in the answer set. the context p is a span of text that not only contains all @xmath0. we refer to a collection of all cases as the casebase b, which is typically set to the full training data and may continually be augmented to incorporate new data patterns. in the mrc setting, annotations for p for all @xmath1. ## experiments we evaluate cbr-mrc on two machine reading comprehension tasks -supervised answer span extraction and few-shot domain adaptation. for three mrc datasets, we use the training data to both construct the casebase as well as train our model and report performance on the in-domain test sets. we report the few-shot domain adaptation performance on two additional datasets. ## conclusion we present cbr-mrc, a semi-parametric model for machine reading comprehension that is simple, accurate, and interpretable. our model stores a collection of cases, retrieves the most relevant cases for a given test question, and then explicitly reuses the reasoning patterns encoded in the embeddings of these cases to predict an answer. we show that our model performs well for both extracting answers and identifying supporting evidence on several mrc tasks compared to fully-parametric baselines. we also demonstrate the ability of our model to transfer to new domains with limited labeled data. finally, we analyze our model under varying conditions of lexical diversity and find that it is robust to high lexical diversity, whereas fullyparametric models show a drop in performance.