Fine-tuning BERT with Focus Words for Explanation Regeneration (2020)

Abstract: Explanation generation introduced as the world tree corpus (Jansen et al., 2018) is an emerging NLP task involving multi-hop inference for explaining the correct answer in multiple-choice QA. It is a challenging task evidenced by low state-of-the-art performances(below 60% in F-score) demonstrated on the task. Of the state-of-the-art approaches, fine-tuned transformer-based (Vaswani et al., 2017) BERT models have shown great promise toward continued system performance improvements compared with approaches relying on surface-level cues alone that demonstrate performance saturation. In this work, we take a novel direction by addressing a particular linguistic characteristic of the data — we introduce a novel and lightweight focus feature in the transformer-based model and examine task improvements. Our evaluations reveal a significantly positive impact of this lightweight focus feature achieving the highest scores, second only to a significantly computationally intensive system.

## introduction multi-hop inference for explanation regeneration (mier) is an emerging task in nlp that concerns aggregating facts to justify the correct answer choice in multiple-choice question answering settings. the worldtree corpus @xcite that introduced this as a community shared task @xcite , was dedicated to finding systems that generate explanations for answers to elementary science questions based on the mier paradigm. the core task essentially entails two main steps: identification of relevant explanation facts from a given knowledge base, followed by ranking the this work is licensed under a creative commons attribution 4.0 international license. license details: http: //creativecommons.org/licenses/by/4.0/ . a tree is a kind of a plant which of the following helps the leaves break down after answer : decomposers they have fallen off the tree ? a leaf is a part of a tree a plant is a kind of a organism if a leaf falls off of a tree then that leaf is dead decomposition is when a decomposer breaks down dead organisms figure 1 : a elementary science question, its correct answer, and the ordered set of justification facts for the answer in the worldtree corpus @xcite depicted as a subgraph of lexical matches. in light of these caveats in the data, the task presents itself as a fairly complex inference task, where traditional methods for qa that are based on simple fact matching have proved inadequate @xcite . given the lexical match characteristic of the data, a slightly adapted application of tf-idf algorithm @xcite , unsurprisingly demonstrates high performance near that of state-of-theart neural models. the mier task defined in the worldtree corpus @xcite was introduced for the first time as a shared task at textgraph-13 @xcite . the state-of-the-art system @xcite employed a fine-tuned bert-based model in an extended computationally intensive architecture. generally, the performance of these fine-tuned transformer models depends on how related the data is to the original pretraining data and how best the input representation can be encoded. to this end, in this work, concentrating exclusively on enhancing the lexical match between a question, answer, and explanation, we encode a novel lightweight feature based on the psycholinguistic concept of focus words that has been defined by brysbaert et al. loosely, a focus word can be defined as a word which is not too tangible to be experienced directly by the five natural senses (i.e., smell, touch, sight, taste, and hearing), while as well not too abstract (e.g., acquirable) that the meaning may not be illustrated without using other words. from figure 1 , as an example, the focus words are break down, fall, decompose, organism, dead. inspired by @xcite , we demonstrate for the first time the application of focus words in the context of contemporary neural-based transformer models for the task of explanation generation. we observe that employing focus words in neural-based models enhances the lexical attention capability within transformer-based bert models and demonstrates an improvement on vanilla bert models. in fact, among all systems for the task, we obtain the highest scores, second only to the computationally intensive system by das et al. thus, our successful application of focus words in elementary science explanation generation demonstrates a poignant application of a vital psycholinguistic feature in the context of a contemporary problem in artificial intelligence. in our experiments, we examine two main research questions. the first assesses the optimal training experimental setting of the worldtree corpus (2018). specifically, rq1: how does the proportion of negative training examples impact finetuning model performance? the second directly assesses the impact of our focus word feature. rq2: what is the impact of the novel focus word feature on explanation generation in an optimal fine-tuned model? the rest of the paper is structured as follows. we define our problem in section 2, followed by a description of the related work in section 3. section 4 discusses our approach, with evaluation results presented in section 5. we conclude in section 6. ## problem definition given a question @xmath0, w 2 , .., w |q| }, its correct answer @xmath1, w 2 , ..., w |a| }, and a set of explanation facts e s.t. every @xmath2, w 2 , ..., w |e| } where w i are @xmath3. following the definition for the textgraphs-13 mier task @xcite , the aim is to obtain, for every question and its correct answer, an ordered list of a set of facts that are coherent in discourse from a knowledge base of facts. by definition, for a question-correct answer pair (q, a), there exists a set of ordered explanation facts r q,a ⊆ e called the relevant set. for each (q, a) pair, the task aims to generate an ordered list of all the explanation facts in the knowledge base as alluded to in the introduction, we induce novel focus word features from both the question and the answer, and the explanation facts. adapted from brysbaert et al., we deem as focus words @xmath4.0 and 4.2, i.e. one relegated as somewhere in between an abstract and concrete concept word which is relevant in elementary science since they often discuss phenomenon such as "evaporation," "dead," "break down," etc. the textgraph-13 mier shared task saw two flavors of approaches to the task. the traditional approach of using hand-crafted linguistic features in an svm ranker @xcite and with reranking rules to correct obvious prediction errors. and, on the other hand, the most recent bertbased rerankers over heuristically ranked data in a first stage. banerjee tested initial ranking using two different transformer models: bert @xcite and xlnet @xcite , and observe that including parts of gold explanations with question text when training for relevance as additional context offers performance improvement. their approach included reranking the top 15 ranked facts via cosine similarity. chia et al. explore an iterative tf-idf to recursively refine the results and achieve significant improvements on a baseline non-optimized tf-idf. in addition, they employ the results of this process in a bert-based re-ranker to rank the top 64 candidates. the top-ranked system by das et al. used fine-tuned bert both for the initial step and the reranking. where the first bert model is fine-tuned on the whole set of facts in the knowledge base, the second bert model is fine-tuned as a path ranking model. in this latter case, a bert model is trained with chains of valid multi-hop facts from the top 25 candidates. computing chains of multi-hop facts was a brute-force computationally exhaustive process which is not practically viable as noted by the authors. we also include results for a bert model trained purely on just the focus words of the question/answer pairs, and the explanations. this model obtains no signals at all from the data, an indication that focus words are best used as extra signals to the data as opposed to being utilized as standalone data by themselves. ## our approach our approach is illustrated in figure 2 and is described next. ## evaluation and results to develop and evaluate our approach, we use the textgraphs-13 mier shared task (jansen and ustalov, 2019) dataset and evaluation scripts, respectively. ## conclusions in this paper, we empirically determine that the number of negative examples in training has an impact on the fine-tuning process for the explanation regeneration task. we have presented a lightweight, nonetheless, effective solution to the problem of explanation regeneration in elementary science qa. staying on course with the current trend of investigating neural models, we implement a bert-based model with an additional linguistic focus words feature. thereby with our new feature we tap deeper into the nature of data in terms of its linguistic match characteristic. we obtained a considerable improvement in task performance. subsequently, since focus words have proven effective in our experiments, by their nature we hypothesize that attention-based models are a promising future direction for this task. in this work, we developed our system based on the mier textgraph-13 shared task @xcite definition for explanation generation, in the context of which we utilize only the correct answer for ranking explanation facts. toward an end-to-end model, as a first step, we plan to explore the training of an optimal model with all answer choices; following which, we plan to jointly model the question answering process together with explanation generation in a feedback loop such that both tasks mutually improve each other @xcite , except we will test our system for elementary science qa.