T ext G raphs 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration (2020)

Abstract: The 2020 Shared Task on Multi-Hop Inference for Explanation Regeneration tasks participants with regenerating large detailed multi-fact explanations for standardized science exam questions. Given a question, correct answer, and knowledge base, models must rank each fact in the knowledge base such that facts most likely to appear in the explanation are ranked highest. Explanations consist of an average of 6 (and as many as 16) facts that span both core scientific knowledge and world knowledge, and form an explicit lexically-connected “explanation graph” describing how the facts interrelate. In this second iteration of the explanation regeneration shared task, participants are supplied with more than double the training and evaluation data of the first shared task, as well as a knowledge base nearly double in size, both of which expand into more challenging scientific topics that increase the difficulty of the task. In total 10 teams participated, and 5 teams submitted system description papers. The best-performing teams significantly increased state-of-the-art performance both in terms of ranking (mean average precision) and inference speed on this challenge task.

## introduction multi-hop inference is the task of combining two or more facts to make an inference. in the context of natural language processing, this is often studied in terms of question answering tasks, where a model must combine multiple textual facts (typically retrieved from different books, web pages, or other documents) to answer a question correctly. with the recent field-wide push towards building machine learning models that are able to explain the reasons behind their inferences, multi-hop inference has garnered a renewed interest, as the set of connected facts used to perform the inference can be supplied to the user as a form of human-readable explanation for why the inference is correct. multi-hop inference can be extremely challenging, particularly as the number of facts required to perform an inference increases, which typically causes large drops in performance @xcite and places strong limits on inference capacity @xcite . moreover, a body of recent work suggests that, in spite of steadily increasing performance on multi-hop benchmarks, much of this performance may be due to strong retrieval baselines rather than methods that are explicitly performing compositional inference @xcite @xcite . the shared task on multi-hop inference for explanation regeneration aims to address some of these contemporary challenges in multi-hop inference by asking participants to develop systems that can construct very large multi-fact explanations for science exam questions that contain up to 16 facts. the task simplifies the question answering problem, supplying both question and correct answer a given model, allowing that model to squarely focus on the explanation construction task. for a given question, a model must pick a complete set of explanatory facts from a knowledge base of approximately 10,000 semi-structured facts that span core scientific knowledge as well as detailed common sense or world knowledge. these model-generated explanations are then evaluated against hand-authored explanations generated by skilled human annotators whose goals in authoring were both explanatory completeness and having a high level of explanatory depth. an example question, answer, and short explanation graph is shown in figure 1 . in the context of contemporary datasets for multi-hop inference, the worldtree v2 corpus @xcite used in this shared task has both substantially larger multi-hop inference problems and substantially less training data than other multi-hop inference datasets, making this shared task extremely challenging. for example, the frequently used hotpotqa dataset @xcite requires aggregating sentences from 2 paragraphs in wikipedia, and has seen contemporary models reach nearly 90% performance on its analogous supporting-fact selection task foot_0 . similarly, qasc @xcite , a science-domain dataset foot_1 similar to worldtree that requires selecting two supporting facts from a corpus, has also seen accuracy reach 90% @xcite . in contrast, the best-performing model in this 2020 shared task was able to achieve a map of 0.60 on the 1-to-16 fact multi-hop inference problems in worldtree v2, highlighting the difficulty of this challenge task, and showcasing that there is still plenty of room to grow. an example 13-fact explanation from this shared task is shown in figure 2 , illustrating the difference in difficulty between generating 2-fact explanations and many-fact explanations that include detailed world knowledge. this is the second iteration of the shared task on multi-hop inference for explanation regeneration, which keeps the task identical to the first iteration run in 2019 @xcite , but more than doubles the available training data, evaluation data, and supporting knowledge base. this increase in size is largely due to expanding the explanation corpus to more advanced years of standardized science exams, causing the task to also become significantly more challenging. in this regard, baseline tf.idf performance decreased from 0.30 map in 2019 to 0.23 map in 2020, a drop of more than 20% in performance, highlighting the challenging nature of this additional data. participating teams used a wide variety of approaches, typically with large language models featuring prominently, augmented with graph neural networks, integer linear programming, or iterative scoring methods for the multi-hop inference task. in spite of the dataset being substantially more challenging than the 2019 shared task, participants made substantial increases both in overall task performance as well as in the speed of training and inference. our shared task has been organized on the codalab platform. 3 we released train and development datasets along with the baseline solution in advance to allow one to get to know the task specifics. we ran the practice phase from march 1 till april 5, 2020. then we released the test dataset without answers and ran the official evaluation phase from april 6 till september 21, 2020. after that we established post-competition phase to enable long-term evaluation of the methods beyond our competition. in this shared task summary paper we first highlight some of the contemporary challenges in multi-hop inference. we then describe the explanation regeneration task (framed as a ranking problem), the details of the training and evaluation dataset used for the shared task, followed by competition details and system descriptions for participating teams. ## contemporary challenges in multi-hop inference a number of contemporary challenges exist in performing multi-hop inference for question answering, with several highlighted below. for a more in-depth survey of contemporary challenges and methods for multi-hop inference, see @xcite . semantic drift. semantic drift is the tendency for inference algorithms based on graph traversal to traverse from highly-relevant facts (nodes) towards irrelevant nodes based on noisy signals. for example, when answering a question about popular varieties of orchard apples, without mechanisms to control for semantic drift, a given algorithm might traverse to facts about popular apple computers because common signals for traversal (such as two facts having one or more of the same words) are often noisy and lack context. semantic drift has been observed across a wide variety of representations and traversal methods figure 1 : an example of the explanation regeneration task. a model is supplied with a question and correct answer (top), and from this must construct an explanation from facts in the supporting knowledge base. an example 3-fact gold explanation is shown (bottom). facts are connected to question, answer, and/or other facts by explicit lexical overlap (edges). many-hop multi-hop training data. large, high-quality datasets for training multi-hop inference models generally were not available until recently @xcite @xcite . even as such, pragmatic challenges in dataset construction necessitate that each dataset will have limitations. the ideal dataset would be: (1) large in terms of the number of training and evaluation examples, (2) use natural rather than artificial questions, (3) use found (retrieved) rather than authored facts, (4) contain large many-fact multi-hop inference problems, (5) include explicit details of the inference required to be made, including any common sense or world knowledge that may otherwise be implicit and inaccessible to a model. in general, existing datasets tend to compromise on at least several of these (and other) desiderata for technical or pragmatic reasons (such as cost or scalability). ## training and evaluation dataset this 2020 shared task on explanation regeneration transitions from worldtree v1 @xcite to the new worldtree v2 explanation corpus @xcite . similar to the first worldtree corpus, the second version contains large multi-hop inference problems that require models to construct large, detailed, multi-fact explanations for standardized science exam questions from the united states -but at a greater scale. worldtree v1 consists of approximately 1.7k questions paired with explanations constructed from a knowledge base of approximately 5k facts, while v2 expands this to 4.4k questions and a knowledge base of 10k facts, more than doubling the amount of training and evaluation data available. questions. questions consist of standardized science exam questions drawn from the aristo reasoning challenge (arc) corpus @xcite , a set of 4-choice multiple choice questions primarily drawn from 12 us states over the past two decades. where the 2019 shared task consisted entirely of elementary science questions (intended for students typically between 9 to 10 years in age), the additional questions in the 2020 shared task primarily expand into middle school science (intended for students between 13 and 14 years of age). these questions typically require broader knowledge and more complex forms of reasoning than their elementary counterparts, increasing the overall difficulty of the task. explanations. each question in worldtree is paired with a detailed explanation for why the answer to that question is correct. explanations take the form of a set of atomic facts that consist of both core scientific knowledge (e.g. "cellular respiration is when a cell converts from oxygen and carbohydrates into carbon dioxide, water, and energy") as well as common-sense/world knowledge (e.g. "if a container contains something, then that container touches that something"), with the goal of authoring explanations at sufficient explanatory depth to meet the informal goal of making them "meaningful to a 5 year old". each explanation ranges in length from 1 to over 16 facts (with the average explanation containing 6 facts). the explanations were authored such that key terms in the question, answer, and explanatory facts must be explicitly linked to each other to help make the inference process more explicit -and as such, while the set of facts for each explanation are unordered from the perspective of forming a narrative, they form a lexically-connected "explanation graph" describing how the knowledge interconnects. each fact in an explanation contains an additional rating describing that fact's "explanatory role" in the explanation, which can be either central, grounding, or lexical glue: question. a student placed an ice cube on a plate in the sun. ten minutes later, only water was on the plate. which process caused the ice cube to change to water? answer candidates. (a) condensation (b) evaporation (c) freezing (*d) melting gold explanation from worldtree corpus. explanatory role fact (table row) central melting means changing from a solid into a liquid by adding heat energy grounding an ice cube is a kind of solid grounding water is a kind of liquid central water is in the solid state, called ice, for temperatures between -273c and 0 c lexglue heat means heat energy lexglue adding heat means increasing temperature central if an object absorbs solar energy then that object will increase in temperature central if an object is in the sunlight then that object will absorb solar energy central the sun is a source of (light ; light energy) called sunlight lexglue to be in the sun means to be in the sunlight central melting is a kind of process explanation regeneration task (ranking). rank gold fact (table row) 1 melting is a kind of process 2 thawing is similar to melting 3 melting is a kind of phase change 4 melting is when solids are heated above their melting point 5 amount of water in a body of water increases by (storms ; rain ; ice melting) 6 an ice cube is a kind of object 7 an ice cube is a kind of solid 8 freezing point is similar to melting point 9 melting point is a property of a (substance ; material) 10 glaciers melting has a negative impact on the glaicial environment 11 plate tectonics is a kind of process 12 sometimes piles of rock are formed by melting glaciers depositing rocks 13 melting point can be used to identify a pure substance 14 ice crystals means ice 15 the (freezing point of water ; melting point of water) is 0c 16 the melting point of iron is 1538c 17 the melting point of oxygen is -218.8c 18 melting means changing from a solid into a liquid by adding heat energy 19 adding salt to a liquid decreases the melting point of that liquid 20 knowledge base. each fact in worldtree takes the form of a row in one of 80 semi-structured tables. each table is centered around a particular kind of knowledge (e.g. taxonomic, part-of, properties, changes, causality, if-then relationships, coupled relationships, affordances, etc.), and contains between 2 and 16 content columns that allow each fact to form a semi-structured n-ary relation. the table topics and structure was empirically derived using prior studies in the science domain as a starting point @xcite . each fact in worldtree can be used either in this semi-structured form, or read off directly as a plain text sentence, allowing both structured and free-text inference methods to use the same knowledge base, and be directly compared. ## system descriptions and performance the 2020 shared task received 10 submissions, nearly doubling submissions from the previous year. system performance is shown in baseline (tf.idf). a term frequency baseline. for a given question and answer pair, the model calculates the cosine similarity between a query vector (composed of term frequencies from either the question or correct answer) and document vectors (composed of term frequencies from a given fact in the knowledge base) for each fact in the knowledge base. this baseline uses the tf.idf weighting scheme when calculating cosine similarity (e.g. see @xcite ). for each (question, answer, fact) tuple, two cosines are calculated -one between question and fact, the other between answer and fact, and these two scores are combined into a linear model using the svm rank ranking classifier @xcite ), foot_4 to exhaustively rank each fact in the knowledge base for a given (question, answer) pair. baidu pgl. this best-performing system by @xcite at baidu combines language models and graph neural networks in a reranking framework. first, the ernie 2.0 language model @xcite , which achieves over 90% performance on the glue benchmark @xcite , is used to provide an initial ranking of facts in the knowledge base. the team notes that this nearly doubles oracle ceiling performance compared to a tf.idf model, ranking an average of 92% of gold facts within the top 100, while the initial ranking itself provides a comparatively strong 0.48 map on explanation reconstruction as a stand-alone retrieval model. a second ernie 2.0-based module then reranks the shortlist from the initial ranker, dramatically increasing performance to 0.59 map. a gnn based on graphsage @xcite is then used to help learn to aggregate facts in a multi-hop fashion, which increases performance by approximately 0.01 map. finally, an ensemble model of the full model is constructed, raising performance 0.02 map to reach 0.62 on the development set, while evaluating at 0.60 map on the unseen test set. ## extended evaluation and analysis the worldtree corpus is designed to instrument various aspects of the multi-hop inference process, and here as in the first shared task we provide an extended analyses of shared task participant performance beyond the final measure of explanation regeneration quality. as these analyses are identical to those in the 2019 shared task, please see @xcite for a full description of the analysis metrics and procedures. ## conclusion the 2020 shared task on multi-hop inference for explanation regeneration successfully achieved a new state-of-the-art performance on the explanation regeneration task using the benchmark worldtree v2 dataset. participating teams used a wide variety of methods, typically combining large pre-trained language models with task-specific modules for performing the multi-hop inference task, and improved both explanation regeneration accuracy and speed. additional analyses show that models with similar downstream performance can show different performance profiles on specific aspects of the task in general, and on multi-hop performance in particular, emphasizing the need to report detailed performance profiles when working on multi-hop inference tasks.