T ext G raphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration (2021)

Abstract: The Shared Task on Multi-Hop Inference for Explanation Regeneration asks participants to compose large multi-hop explanations to questions by assembling large chains of facts from a supporting knowledge base. While previous editions of this shared task aimed to evaluate explanatory completeness – finding a set of facts that form a complete inference chain, without gaps, to arrive from question to correct answer, this 2021 instantiation concentrates on the subtask of determining relevance in large multi-hop explanations. To this end, this edition of the shared task makes use of a large set of approximately 250k manual explanatory relevancy ratings that augment the 2020 shared task data. In this summary paper, we describe the details of the explanation regeneration task, the evaluation data, and the participating systems. Additionally, we perform a detailed analysis of participating systems, evaluating various aspects involved in the multi-hop inference process. The best performing system achieved an NDCG of 0.82 on this challenging task, substantially increasing performance over baseline methods by 32%, while also leaving significant room for future improvement.

## introduction multi-hop inference is the task of aggregating more than one fact to perform an inference. in the context of natural language processing, multi-hop inference is typically evaluated using auxiliary tasks such as question answering, where multiple sentences from external corpora need to be retrieved and composed figure 1: the motivating example provided to participants. given a question and correct answer (top), the explanation regeneration task requires participating models to find sets of facts that, taken together, provide a detailed chain-of-reasoning for the answer (bottom). this 2021 instantiation of the shared task focuses on the subtask of collecting the most relevant facts for building explanations. to form reasoning chains that support the correct answer (see figure 1 ). as such, multi-hop inference represents a crucial step towards explainability in complex question answering, as the set of supporting facts can be interpreted as an explanation for the underlying inference process @xcite . constructing long inference chains can be extremely challenging for existing models, which generally exhibit a large drop in performance when composing explanations and inference chains requiring more than 2 inference steps @xcite @xcite @xcite . to this end, this shared task on multi-hop inference for explanation regeneration @xcite ) has focused on expanding the capacity of models to compose long inference chains, where participants are asked to develop systems capable of reconstructing detailed explanations for science exam questions drawn from the worldtree explanation corpus @xcite , which range in compositional complexity from 1 to 16 facts (with the average explanation including 6 facts). large explanations are typically evaluated on two dimensions: relevance and completeness. relevance refers to whether each fact in an explanation is relevant, topical, and required to complete the chain of inference that moves from question to correct answer. conversely, completeness evaluates whether the entire set of facts in the explanation, together, composes a complete chain of inference from question to answer, without significant gaps. in practice, both of these are challenging to evaluate automatically @xcite , given that multi-hop datasets typically include a single example of a complete explanation, in large part due to the time and expense associated with generating such annotation. underscoring this difficulty, post-competition manual analyses on participating systems in the previous two iterations of this shared task showed that models may be performing up to 20% better at retrieving correct facts to build their explanation from, highlighting this significant methodological challenge. this 2021 instantiation of the shared task on explanation regeneration focuses on the theme of determining relevance in large multi-hop explanations. to this end, participants were given access to a large pre-release dataset of approximately 250k explanatory relevancy ratings that augment the 2020 shared task data (jansen and ustalov, 2020), and were tasked with ranking the facts most critical to assembling large explanations for a given question highest. similarly to the previous instances of our competition, the shared task has been organized on the codalab platform. 1 we released train and development datasets along with the baseline solution in advance to allow one to get to know the task specifics. 2 we ran the practice phase from february 15 till march 9, 2021. then we released the test dataset without answers and ran the official evaluation phase from march 10 till march 24, 2021. after that we established postcompetition phase to enable long-term evaluation of the methods beyond our shared task. participating systems substantially increased task performance compared to a supplied baseline system by 32%, while achieving moderate overall absolute task performance -highlighting both the success of this shared task, as well as the continued challenge of determining relevancy in large multi-hop inference problems. ## task description following the previous editions of the shared task, we frame explanation generation as a ranking problem. specifically, for a given science question, a model is supplied both the question and correct answer text, and must then selectively rank all the atomic scientific and world knowledge facts in the knowledge base such that those that were labelled as most relevant to building an explanation by a human annotator are ranked the highest. additional details on the ranking problem are described in the 2019 shared task summary paper @xcite . ## training and evaluation dataset questions and explanations: the 2021 shared task adopts the same set of questions and knowledge base included in the 2020 shared task (jansen and ustalov, 2020), with additional relevance annotation described below. the questions and explanations are drawn from the worldtree v2 explanation corpus @xcite , a set of detailed multi-fact explanations to standardized elementary and middle-school science exam questions drawn from the aristo reasoning challenge (arc) corpus @xcite . worldtree v2 contains 2207 train, 496 development, and 1665 held-out test questions and explanations. relevancy ratings: the worldtree v2 dataset used in previous iterations of the shared task includes a single complete explanation per question, supplied as a list of binary classifications that describe which facts are included in the gold explanation for a given question. this 2021 edition of the shared task augments these original worldtree explanations with a pre-release dataset foot_2 of approximately 250,000 manual relevancy ratings. specifically, for each question in the corpus, a set of 30 facts determined to be the most likely facts relevant to building an explanation were manually assigned relevancy ratings by annotators. ratings are on a 7-point scale (0-6), where facts rated as a 6 are the most critical to building an explanation, while facts rated as 0 are unrelated to the question. an example of these relevance ratings is shown in evaluation metrics: historically, performance on the explanation regeneration task was evaluated using mean average precision (map) , using the binary ratings (gold or not gold) associated with each fact for a given explanation. to leverage the new graded annotation schema, here we switch to evaluate system performance using normalized discounted cumulative gain (ndcg) @xcite . ## system descriptions and performance the 2021 shared task received 4 submissions, with 3 teams choosing to submit system description papers. the performance of the submitted systems are shown in baseline (tf.idf). we adopt a term frequencyinverse document frequency (tf.idf) baseline (see, e.g. @xcite . specifically, given a question and its correct answer, the baseline calculates the cosine similarity between a query vector (representing the question and correct answer) and document vectors (representing a given fact) for each fact in the knowledge base. the model then adopts the tf.idf weighting scheme to rank each fact in the knowledge base for a given question-answer pair. this baseline achieves a ndcg score of 0.501 on the test set. deepblueai. the model presented by @xcite represents the top-performing system in this edition of the shared task with a ndcg score of 0.820 -representing a substantial 32% improvement when compared to the tf.idf baseline. the model employs a two step retrieval strategy. in the first step, a pre-trained language model is finetuned to retrieve the top-k (@xmath0. subsequently, the same architecture is adopted to build a re-ranking model to refine the list of the top-k candidate facts. the authors propose the use of a triplet loss for the fine-tuning of the model. specifically, the triplet loss minimizes the distance between an anchor and a positive example, while maximizing the distance between the same anchor and a negative example. the team treats question and correct answer as the anchor, while the facts annotated with high ratings are adopted as positive examples. different experiments are conducted with three negative sampling strategies for retrieval and re-ranking. the best results are obtained when sampling negative examples from the same tables of highly relevant facts. the authors find that the best performance is obtained when averaging the results from roberta @xcite and ernie 2.0 @xcite with different random seeds. google-bert. @xcite propose a framework composed of three main steps. in the first step, the model adopts a simple tf.idf model with cosine similarity to retrieve the top-k relevant explanation sentences (@xmath1. in the second step, the authors employ an autoregressive model which selects the most relevant facts in a iterative manner. specifically, the authors propose the adoption of a bert-based model @xcite that selects the facts at iteration n given the facts retrieved in the previous step. the model uses up to 4 iterations. finally, the authors employ a re-ranking module to re-score the retrieved candidate explanations computing the relevance between each fact and the question-answer pairs. the re-ranking model is implemented using a bert model for binary classification. the ablation study shows that the first two steps allow achieving a performance of 0.679 ndcg, that is improved up to 0.700 ndcg using the re-ranking model. moreover, the experiments show that the best performance is achieved when the re-ranking model is adopted to re-score the top @xmath2. ## detailed analysis in order to better understand the behavior and contribution of the proposed systems, we perform a detailed analysis by grouping the explanatory facts in the supporting knowledge base in different categories. specifically, we adopt categories that cover various aspects of the multi-hop inference process, ranging from different kinds of knowledge to different degrees of explanatory relevance and lexical overlap, to analyse the performance of each model beyond the overall explanation regeneration score. ## conclusion the 2021 edition of the shared task on multi-hop inference for explanation regeneration was a success, with 4 participating teams each substantially improving performance over the baseline model. the best performing team, deepblueai, produced a system that improves absolute performance by 32%, up to 0.820 ndcg, bringing overall state-ofthe-art performance at this relevancy ranking aspect of multi-hop inference to a moderate level. we hope that future systems for many-hop multi-hop inference that aim to build large detailed explanations for question answering will be able to leverage these results to build strong relevancy retrieval subcomponents to augment their compositional inference algorithms. award #1815948, "explainable natural language inference"). this edition of the shared task would not have been possible without the hard work of a number of relevance annotators, and their generous offer to anonymously use their data while their work is under review. a special thanks to andré freitas for the helpful discussions. additionally, we would like to thank the computational shared facility of the university of manchester for providing the infrastructure to run our experiments.