C hi S quare X at T ext G raphs 2020 Shared Task: Leveraging Pretrained Language Models for Explanation Regeneration (2020)

Abstract: In this work, we describe the system developed by a group of undergraduates from the Indian Institutes of Technology for the Shared Task at TextGraphs-14 on Multi-Hop Inference Explanation Regeneration (Jansen and Ustalov, 2020). The shared task required participants to develop methods to reconstruct gold explanations for elementary science questions from the WorldTreeCorpus (Xie et al., 2020). Although our research was not funded by any organization and all the models were trained on freely available tools like Google Colab, which restricted our computational capabilities, we have managed to achieve noteworthy results, placing ourselves in 4th place with a MAPscore of 0.49021in the evaluation leaderboard and 0.5062 MAPscore on the post-evaluation-phase leaderboard using RoBERTa. We incorporated some of the methods proposed in the previous edition of Textgraphs-13 (Chia et al., 2019), which proved to be very effective, improved upon them, and built a model on top of it using powerful state-of-the-art pre-trained language models like RoBERTa (Liu et al., 2019), BART (Lewis et al., 2020), SciB-ERT (Beltagy et al., 2019) among others. Further optimization of our work can be done with the availability of better computational resources.

## introduction the shared task is aimed at multi-hop inference for explanation regeneration. participants are required to develop new and improve existing methods to reconstruct gold explanations for the worldtree corpus @xcite of elementary science questions, their answers, and explanations. question: which of the following is an example of an organism taking in nutrients? (a) a dog burying a bone (b) a girl eating an apple (c) an insect crawling on a leaf (d) a boy planting tomatoes answer: (b) a girl eating an apple gold explanation facts: 1) a girl means a human girl: grounding 2) humans are living organisms: grounding 3) eating is when an organism takes in nutrients in the form of food: central 4) fruits are kinds of foods: grounding 5) an apple is a kind of fruit: grounding irrelevant explanation facts: 1) some flowers become fruits. 2) fruit contains seeds. 3) living things live in their habitat. 4) consumers eat other organisms ## dataset the dataset is the worldtree corpus v2.1 @xcite of explanation graphs and inference patterns supporting multi-hop inference (februrary 2020 snapshot). it is a newer version of the dataset used in the textgraphs-2019 @xcite . the comparison between the two datasets is shown in ## problem review the problem statement requires participants to build a system that, given a question and its answer choices, can identify the sentences that explain the answer given the question. this is a challenging task due to the presence of other irrelevant sentences in the corpora for the given question, which have equally significant lexical and semantic overlap as the correct ones @xcite . when a more classical graph theory approach using the semantic overlap of explanations and questions is tried, it leads to the problem of semantic drift @xcite . more classic graph methods were attempted in @xcite , where the challenge of semantic drift in multi-hop inference was analyzed, and the effectiveness of information extraction methods was demonstrated. also, approaching the question as a language generation task is not effective and the current state-of-the-art models @xcite are not capable of generating the exact explanations as required by this task. so this task can easily be transformed into a sentence ranking problem in which we need to rank the relevant facts over all other given facts present in the corpus. the evaluation metric used for the task is the widely used and robust mean average precision (map) metric. we have explained a few initial experiments that were undertaken in section 4.1, followed by the pre-processing methods we incorporated in section 4.2. we have then discussed our models in sections 4.3 through 4.6. we have finally shown all our results and discussions in section 5 followed by the conclusion and acknowledgments. ## results and discussion we present the scores in the table given below. we got our best performance from roberta. when we observe the results, we see that there is only a slight variation in the final scores of most pre-trained language models. we observe that the models overfit the given data. we could not perform a grid search to optimize all parameters due to computational constraints and had to manually search for the best hyperparameters due to which the performance of any given model may not be optimal. further, we have trained only roberta and bart for top k 500 explanations and not other models because they had a long training time or a higher ram requirement. ## conclusion we have given a system description of our team chisquarex which stood 4th place in the evaluation phase leaderboard with a map score of 0.4902. we have presented a system with optimized preprocessing of the dataset followed by an optimized tfidf information retrieval scheme to obtain initial ranks, and then further pre-trained language model based re-ranker to rank the final explanations. despite the computational constraints, just by leveraging google colab and other open-source tools, we have managed to fine-tune state-of-the-art pre-trained language models like roberta, bart and electra on the @xcite dataset and achieve a reasonable map score.