Textgraphs-15 Shared Task System Description : Multi-Hop Inference Explanation Regeneration by Matching Expert Ratings (2021)

Abstract: Creating explanations for answers to science questions is a challenging task that requires multi-hop inference over a large set of fact sentences. This year, to refocus the Textgraphs Shared Task on the problem of gathering relevant statements (rather than solely finding a single ‘correct path’), the WorldTree dataset was augmented with expert ratings of ‘relevance’ of statements to each overall explanation. Our system, which achieved second place on the Shared Task leaderboard, combines initial statement retrieval; language models trained to predict the relevance scores; and ensembling of a number of the resulting rankings. Our code implementation is made available at https://github.com/mdda/worldtree_corpus/tree/textgraphs_2021

## introduction complex question answering often requires reasoning over many evidence documents, which is known as multi-hop inference. existing datasets such as wikihop @xcite , open-bookqa @xcite , qasc @xcite , are limited due to artificial questions and short aggregation, requiring less than 3 facts. in comparison, the textgraphs shared task @xcite ) makes use of worldtree v2 @xcite which a large dataset of over 5,000 questions and answers, as well as detailed explanations that link them. the 'gold' explanation paths require combining an average of 6 and up to 16 facts in order to generate an full explanation for complex science questions. the worldtree dataset was recently supplemented with approximately 250,000 expertannotated relevancy ratings for facts that were highly ranked by models in previous shared task iterations, based on the same consistent set of question and answers. in previous years, the emphasis of the shared task has been on creating 'connected explanations' as completely as possible, which is difficult because of the large branching factor along an explanation path, in conjunction with semantic drift @xcite . in contrast, the scoring function for the 2021 shared task required participants to rank explanation statements according to their relevance to explaining the science situation, rather than whether they are in the single gold explanation path. specifically, participants were required to provide ordered lists of explanation statements for each question, and the normalized discounted cumulative gain measure @xcite was used as a scoring function. the main contributions of this work are: ## models neural information retrieval models such as dpr @xcite , rag @xcite , and colbert @xcite ) that assume query-document independence use a language model to generate sentence representations for the query and document separately. the advantage of this late-interaction approach is efficient ## experiments our system comprised three stages, and we present results of the experiments used to validate our choices at each stage, with the overall results being compiled in ## discussion in previous versions of the textgraphs shared task, the goal was essentially to obtain the single 'gold explanation' that perfectly matched an expertly crafted graph of explanation statements, with the scoring being based on a ranking metric that rewarded participants for finding these gold explanation statements. this task was challenging due to the semantic drift issue previously mentioned, and the sensitivity of the scoring to choosing the same explanation path as the original annotators. foot_0 paradoxically, instead of tackling the problem with logic-oriented graph planning methods, the dominant techniques tended to rely on large language models which could maximise the ranking scores without 'understanding the bigger picture'. the change of scoring metric in this current shared task, to incorporate all statements that are relevant to the question and answer, appears to target the capturing of 'bigger picture' ideas. however, this seems to have once again promoted the use of large language models, since they provide a system component that can bring the most 'common sense' into the multi-step reasoning domain, without getting tangled in the logical weeds that go into producing the gold explanations. while the addition of the expert ratings on the explanation statements is undoubtedly positive for the shared task dataset, it is not clear to what extent it helps address the multi-hop nature of the challenge -on which significant progress had already been made (and will hopefully continue based on other promising directions have been identified by previous iterations of the shared task.) ## conclusion our shared task submissions showed that ensembles of language models trained on a regression basis to predict expert ratings obtain highly competitive results. we look forward to achieving further progress on the multi-hop reasoning task in the future.