Towards Interpreting BERT for Reading Comprehension Based QA (2020)

Abstract: BERT and its variants have achieved state-of-the-art performance in various NLP tasks. Since then, various works have been proposed to analyze the linguistic information being captured in BERT. However, the current works do not provide an insight into how BERT is able to achieve near human-level performance on the task of Reading Comprehension based Question Answering. In this work, we attempt to interpret BERT for RCQA. Since BERT layers do not have predefined roles, we define a layer’s role or functionality using Integrated Gradients. Based on the defined roles, we perform a preliminary analysis across all layers. We observed that the initial layers focus on query-passage interaction, whereas later layers focus more on contextual understanding and enhancing the answer prediction. Specifically for quantifier questions (how much/how many), we notice that BERT focuses on confusing words (i.e., on other numerical quantities in the passage) in the later layers, but still manages to predict the answer correctly. The fine-tuning and analysis scripts will be publicly available at https://github.com/iitmnlp/BERT-Analysis-RCQA .

## introduction the past decade has witnessed a surge in the development of deep neural network models to solve nlp tasks. pretrained language models such as @xcite , bert @xcite , xlnet @xcite etc. have achieved state-of-the-art results on various nlp tasks. this success motivated various studies to understand how bert achieves human-level performance on these tasks. @xcite ; @xcite analyze syntactic and semantic roles played by different layers in such models. @xcite specifically analyze bert's attention * * now at google research, bangalore heads for syntactic and linguistic phenomena. most of these works focus on tasks such as sentiment classification, syntactic/semantic tags prediction, natural language inference, and so on. however, to the best of our knowledge, bert has not been thoroughly analyzed for complex tasks like rcqa. it is a challenging task because of 1) the large number of parameters and non-linearities in bert, and 2) the absence of pre-defined roles across layers in bert as compared to pre-bert models like bidaf @xcite or dcn @xcite . in this work, we take the first step to identify each layer's role using the attribution method of integrated gradients @xcite . we then try to map these roles to the following functions, deemed necessary in pre-bert models to reach the answer: (i) learn contextual representations for the passage and the question, individually, (ii) attend to information in the passage specific to the question and, (iii) predict the answer. we perform analysis on the squad @xcite and duorc @xcite datasets. we observe that the initial layers primarily focus on question words that are present in the passage. in the later layers, the focus on question words decreases, and more focus is on the supporting words that surround the answer and the predicted answer span. further, through a focused analysis of quantifier questions (questions that require a numerical entity as the answer), we observe that bert pays importance to many words similar to the answer (same type, such as numbers) in later layers. we find this intriguing since, even after marking confusing words spread across passage as important, bert's prediction accuracy is high. we also provide qualitative analysis to demonstrate the above trends. ## related work in the past few years, various large-scale datasets have been proposed for the rcqa task @xcite @xcite which have led to various deep neural-network (nn) based architectures such as @xcite ; @xcite . additionally, with complex pretraining, models such as @xcite ; @xcite ; @xcite are very close to human-level performance. due to the large number of parameters and nonlinearity of deep nn models, the answer to the question "how did the model arrive at the prediction?", is not known; hence, they are termed as blackbox models. motivated by this question, there have also been many works that analyze the interpretability of deep nn models on nlp tasks; many of them analyze models based on in-built attention mechanisms @xcite @xcite . further, various attribution methods such as @xcite al. (2015 they point out that bert is prone to be fooled by such attacks. unlike these earlier works, we focus on analyzing bert's layers specifically for rcqa to understand their qa-specific roles and their behavior on potentially confusing quantifier questions. ## experimental setup for our bert analysis, we use the bert-base model, which has 12 transformer blocks (layers), each with a multi-head self-attention and a feedforward neural network. we use the official code and pre-trained checkpoints foot_0 and fine-tune it for two epochs for the squad and duorc datasets to achieve f1 scores of 88.73 and 54.80 on their respective dev-splits. we use squad @xcite 1.1 with 90k/10k train/dev samples, each with a 100-300 words passage and the selfrc dataset in duorc @xcite with 60k/13k train/dev samples, each with a 500 (on average) words passage. for each passage, both datasets have a natural language query and answer span in the passage itself. ## layer-wise functionality as discussed earlier, we aim to understand each bert layer's functionality for the rcqa task; we want to identify the passage words that are of primary importance at each layer for the answer. intuitively, the initial layers should focus on question words, and the latter should zoom in on contextual words that point to the answer. to analyze the above, we use the attribution method integrated gradients @xcite on bert at a layerwise level. for a given passage p consisting of n words [w 1 , w 2 , . . . , w n ], query q, and model f with θ parameters, answer prediction is modeled as: ## conclusion in this work, we highlight that the lack of predefined roles for layers adds to the difficulty of interpreting highly complex bert-based models. we first define each layer's functionality using integrated gradients. we present results and analysis to show that bert is learning some form of passagequery interaction in its initial layers before arriving at the answer. we found the following observations interesting and with a potential to be probed further: (i) why do the question word representations move away from contextual and answer representation in later layers? (ii) if the focus on confusing words increases from the initial to later layers, how does bert still have a high accuracy? we hope that this work will help the research community interpret bert for other complex tasks and explore the above open-ended questions.