Encoding Explanatory Knowledge for Zero-shot Science Question Answering (2021)

Abstract: This paper describes N-XKT (Neural encoding based on eXplanatory Knowledge Transfer), a novel method for the automatic transfer of explanatory knowledge through neural encoding mechanisms. We demonstrate that N-XKT is able to improve accuracy and generalization on science Question Answering (QA). Specifically, by leveraging facts from background explanatory knowledge corpora, the N-XKT model shows a clear improvement on zero-shot QA. Furthermore, we show that N-XKT can be fine-tuned on a target QA dataset, enabling faster convergence and more accurate results. A systematic analysis is conducted to quantitatively analyze the performance of the N-XKT model and the impact of different categories of knowledge on the zero-shot generalization task.

## introduction contemporary question answering (qa) is evolving in the direction of addressing more abstractive reasoning tasks @xcite @xcite , supported by multi-hop inference @xcite and explanatory scientific facts @xcite @xcite . this trend of aiming to address more complex, multi-evidence and chained inference is pushing the envelope for novel representation and architectural patterns @xcite @xcite @xcite , which are moving from modelling meaning from immediate distributional semantics patterns into deeper abstractive capabilities. this poses a paradigmatic challenge on the design of qa architectures, which need to operate over high-level semantic patterns and acquire the necessary knowledge to perform abstraction @xcite . at the same time, the design of new strategies to incorporate explanatory knowledge into neural representation has the potential to address fundamental data efficiency problems and promote zero-shot generalisation on out-of-distribution examples. explanation-based science qa @xcite provides a rich framework to evaluate these emerging requirements, as the task typically requires multi-hop reasoning through the composition of explanatory facts. while existing approaches in the field mainly focus on the construction of natural language explanations @xcite , this work aims to explore the impact of explanatory knowledge on zero-shot generalisation. in this paper, we argue that explanation-centred corpora can serve as a resource to boost zero-shot capabilities on question answering tasks which demand deeper inference. to this end, we explore the adoption of latent knowledge representations for supporting generalisation on downstream qa tasks requiring multi-hop inference. our hypothesis is that explanatory scientific knowledge expressed in natural language can be transferred into neural network representations, and subsequently used to achieve knowledge based inference on scientific qa tasks. to validate this hypothesis, this paper proposes a unified approach that frames question answering as an explanatory knowledge reasoning problem. the unification between the two tasks allows us to explore the adoption of pre-training strategies over explanatory knowledge bases, and subsequently leverage the same paradigm to generalise on the question answering task. an empirical evaluation is performed on transformers-based architectures adopting the worldtree corpus as a knowledge base @xcite and measuring generalisation on arc @xcite and openbookqa @xcite . the main contributions of this paper are as follows: an ablation analysis on different types of knowledge facts is performed to measure the impact of different knowledge categories. ## related work in this section we describe several works related to knowledge-based scientific qa. explanation bank explanation bank foot_0 is a core component of the worldtree corpus @xcite . the dataset provides explanations for multiple-choice science questions in the form of graphs connecting questions and correct answers, where multiple sentences from a knowledge base (kb) are aggregated through lexical overlap between terms. the background knowledge used for the explanations is grouped in semi-structured tables, whose facts range from common-sense to core scientific statements. explanation bank has been proposed for the task of explanation regeneration (jansen and ustalov, 2019) -i.e. given a multiple-choice science question, regenerate the gold explanation supporting the correct answer. the explanation regeneration task has been framed as an information retrieval (ir) problem @xcite . in this paper, we aim to leverage the knowledge expressed in the explanations to enhance generalisation and zero-shot capability on multiple-choice scientific question answering. bidirectional encoder representations from transformers bert represents the foundation which defines the state-of-the-art in several nlp tasks @xcite . this model adopts a transformer-based architecture composed of several layers of attention @xcite that are used to learn a deep bidirectional representation of language. bert-based models have demonstrated remarkable results in question answering when directly fine-tuned on the answer prediction task or additionally pre-trained using domain specific knowledge @xcite . a recent line of research attempts to enrich the input of bert with background knowledge in the form of explanations in order to boost generalisation and accuracy for challenging qa settings. here, the explanations are explicitly constructed through the adoption of language models (rajani et al., 2019) or information retrieval (ir) approaches @xcite . conversely, this paper explores mechanisms to implicitly encode explanatory knowledge in the neural representation to improve the capability of performing downstream inference. specifically, in this work, we adopt transformers as text neural encoders. leveraging external knowledge for scientific qa recently, many solutions have been proposed for science qa that leverage either external reference corpora @xcite @xcite or existing knowledge graphs @xcite @xcite @xcite . generally, previous works rely on information retrieval models or on structural embeddings for knowledge bases, while our work focuses on directly encoding explanatory knowledge, evaluating it in a downstream scientific qa setting. ## methodology scientific question answering has the distinctive property of requiring the articulation of multi-hop and explanatory reasoning. this can be contrasted with the lexical-retrieval style of factoid question answering. additionally, the explanatory chains required to arrive at the correct answer typically operate at an abstract level, through the combination of definitions and scientific laws @xcite . this characteristic makes the generalisation process more challenging, as the answer explanatory facts (worldtree) qa training-set (worldtree) triplify (subject, predicate, object) (1) triplification qa test-set explanatory knowledge acquisition cloze-style qa (2) pre-training (3) fine-tuning on worldtree zero-shot qa fine-tuning qa (4) question answering what is a common use of the mineral graphite? (a) to make glass (b) as a source of iron (c) to make pencil leads (d) as a household @xmath0, 'is used for', 'making pencil lead'> <'pencil lead', 'contains', 'mineral graphite'> 'mineral graphite is used for [mask]' '[mask] contains mineral graphite' θk θk+q target qa training-set cloze-style qa (4) fine-tuning on target dataset θf q t , c t l pred l pred q k , a k , l k q q , a q , l q q s , a s , l s prediction model needs to acquire the ability to perform abstraction from the specific context in the question. this paper hypothesises that it is possible to automatically transfer abstractive knowledge from explanatory facts into neural encoding representation for more accurate scientific qa, and for enabling zero-shot generalization. to this end, we propose n-xkt (neural encoding based on explanatory knowledge transfer) which encodes abstractive knowledge into neural representation to improve the effectiveness in both zero-shot qa task and fine-tuning based qa task. the general neural encoding mechanism is evaluated adopting the following training tasks: ## empirical evaluation we conduct our experiments on four widely used science qa datasets, worldtree v2.0 @xcite , arc easy and challenge @xcite , and openbook qa @xcite . the results tend to confirm our research hypothesis that explanatory knowledge encoding can improve generalization in downstream science question answering (qa) tasks. furthermore, we systematically analyze several factors which may have an impact on the final results, including the use of transformer-based models with a larger number of parameters (bert-large), testing the model on qa tasks using different types of explanatory background knowledge, and measuring training and test performance by further fine-tuning the model on other datasets. ## conclusion in this paper, we proposed a neural encoding mechanism for explanatory knowledge acquisition and transfer, n-xkt. we evaluated the impact of the encoding mechanism on downstream science qa. the proposed model delivers better generalisation and accuracy for qa tasks that require multi-hop and explanatory inference. the proposed encoding mechanism can be used to deliver zero-shot inference capabilities, providing comparable performance when compared to supervised models on qa. these results supports the hypothesis that pretraining tasks targeting abstract and explanatory knowledge acquisition can constitute and impor-tant direction to improve inference capabilities and generalization of state-of-the-art neural models.