Reproducibility Report: Towards Interpreting BERT for Reading Comprehension Based QA
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
In the paper, the authors attempt to understand BERT’s exemplary performance for RCQA tasks by deﬁning each
Self-Attention Layer’s role using Integrated Gradients for SQuAD v1.1 and DuoRC SelfRC datasets.
After this, theyfollow through with experiments and analysis to infer how each layer works to predict the answer, based on the contextand question.
Scope of Reproducibility
Ramnath et al. suggest that the initial layers focus on query-passage interaction, while the later layers focus more oncontextual understanding and enhancing answer prediction.
In our reproducibility plan, we aim to validate this claimand other related claims by completely replicating the authors’ experiments to analyze BERT layers to understand their
RCQA-speciﬁc role and their behavior on potentially confusing Quantiﬁer Questions
Methodology
Since this paper’s ofﬁcial code is not available, we prepare our scripts and modules for processing the data andre-implement the approach as described in the paper.
We refer to the original research paper to cross-check our resultswith their reporting.
We use Google Colab’s free GPU for 35-40 hours for ﬁne-tuning the model and calculating the
Integrated Gradients.
The rest of the experiments can be performed on a CPU within 10-15 hours.
Results
Our reproduced results for all experiments support the central claim made in the paper.
All of our statistics and plotsagree with those in the original paper within a good margin.
We have also analyzed some results beyond the paper andﬁnd that the scope of the original paper is transferable and generalizable.
What was easy
Using HuggingFace Transformers and Datasets for the SQuAD v1.1 was easy as we could adapt the authors’ ideas toour code experiments and verify their central claim without much effort.
There are also libraries readily available for
Jensen-Shannon Divergence and t-SNE and could be used easily.
What was difﬁcult
Re-implementing the paper was more difﬁcult than we expected as there were ambiguities and conﬂicts in our approaches for Integrated Gradients calculation, as well as DuoRC preprocessing and postprocessing.
There were differences in ourmethods of implementation, and multiple iterations had to be performed to decide upon the case to be used, which took up a lot of computational power unnecessarily.
Communication with original authors
We had frequent interaction with the ﬁrst author via email for clariﬁcation and discussion.
Introduction
Previous works on interpreting BERT [1] have discussed its syntactic/semantic roles on simpler Natural Language taskslike sentiment classiﬁcation, syntactic/semantic tags prediction [2–4], etc.
The Reading Comprehension based Question
Answering (RCQA) task involves marking an answer span in a passage, given a question.
Pre-BERT systems [5–7]for question-answering tasks used pre-deﬁned layer-wise roles.
An analysis of BERT for complex tasks like RCQA ischallenging due to the lack of such layer roles and its large number of parameters.
Ramnath et al.
[8] attempt to deﬁnethese layer-roles for BERT using Integrated Gradients (IG) on the RCQA task [9] for SQuAD v1.1 [10] and DuoRC
SelfRC [11] datasets.
Following this, they perform analysis across all Self-Attention Layers to understand how themodel predicts the answers accurately.
In doing so, they provide a mechanism for interpreting the BERT layers andtheir roles, which can be extended to other complex tasks like machine translation, cloze-style question answering, etc.
As a part of the ML Reproducibility Challenge 2020, we replicate the experiments presented in the paper from scratchand analyze if their observations and claims hold true for our implementation.
Scope of Reproducibility
The authors address the lack of pre-deﬁned layer roles in BERT by deﬁning the roles of each layer using IG to get wordimportance scores across individual layers.
Then, they use analysis techniques such as Jensen-Shannon Divergence(JSD), t-Distributed Stochastic Neighbor Embedding (t-SNE) plots, Part-of-Speech (PoS) tagging, etc., to understandhow each layer contributes towards predicting the correct answer.
The central claim of the paper is that the initiallayers focus on query-passage interaction, while the later layers focus on contextual understanding and answerprediction .
Following are the claims that can be extracted from the paper and that we validate through our experiments:
The top-K important tokens show more divergence across layers than the rest of the tokens.
We explore this by
analyzing JSD Heatmaps in §4.1.2 and 4.2.2.
The importance scores of Contextual Words 1 and Answer Words increase from initial to ﬁnal layers, while the
importance scores of Query Words decrease.
This claim is veriﬁed in §4.1.3 using semantic statistics.
The initial layers ﬁnd Query Words 2 more important, while the ﬁnal layers focus on the Answer Words.
This
claim is veriﬁed using visualization in §4.1.4.
The initial layers represent similar words together.
As the layers progress, the Query Words and Answer Words
come closer to each other.
Eventually, all Query Words are separated from Answer and Contextual Spans 3 .
We plot t-SNE representation in §4.1.5 to validate this claim.
Numerical Words stay close to each other in representation throughout the layers.
The t-SNE plot in §4.1. checks this claim as well.
For Quantiﬁer Questions, the importance of Numerical Words increases from the initial to the ﬁnal layers,meaning that the confusing words are more important to the model towards the end.
§4.1.6 discusses therelevant scores and statistics.
BERT has a higher conﬁdence score on Quantiﬁer Questions with more than one numerical entity in passagevs.
Non-Quantiﬁer Questions.
§4.1.6 discusses the relevant scores.
Methodology
The authors use the ofﬁcial BERT ﬁne-tuning script for SQuAD 4 .
Since there is no ofﬁcial code for the paper yet,we implement the authors’ approaches from scratch in PyTorch.
We use HuggingFace’s (HF) Datasets [12] and
Transformers [13] for ﬁne-tuning BERT, and Captum for Integrated Gradients.
3.1 Fine-tuning and Integrated Gradients
Fine-tuning - We use HF’s BertForQuestionAnswering model and load the pre-trained checkpoint - bert-base-uncased .
The BERT model has 12 layers, 768 hidden units per token, 12 attention heads per layer, and 110M parameters.
It ispre-trained on lower-case English text gathered from Books Corpus and English Wikipedia.
Attributions - We use Captum’s implementation of Integrated Gradients.
IG is based on the ﬁne-tuned BERT model.
Note that our ( n + 1) th layer corresponds to the n th layer of the authors’ notation as we take the Embedding
Layer to be Layer 0.
We perform experiments on all Self-Attention Layers and the Embedding Layer .
3.2 Datasets3.2.1 SQuAD v1.1 (SQuAD)
We use the SQuAD v1.1 5 dataset available on HF’s Datasets 6 library for its simplicity.
The train/dev splits contain87599/10570 question-answer pairs for various passages, with start position and answer text specifying the answer.
Each example breaks into tokenized features 7 using a max-overlap stride of 128, and a max sequence length of 384.
Wedo not use a max query length of 64, unlike the ofﬁcial ﬁne-tuning script.
We use HF Transformers’ BertTokenizerFastwith the pre-trained checkpoint - bert-base-uncased .
In cases where the ground truth is absent in the context due tosplitting, we mark ‘[CLS]’ as the answer.
Our ﬁnal train/development sets have 88524/10784 features.
3.2.2 DuoRC SelfRC (DuoRC)
We use the original DuoRC SelfRC 8 train/dev splits with 60721/12961 question-answer pairs based on 4800/984different movie plots.
Additionally, a question can have multiple answers - which may or may not exist in the plot.
Nostart positions are provided for the answers.
Training BERT on a Question Answering task requires the start and endpositions of the answer.
Hence, we ﬁrst convert the DuoRC dataset to SQuAD format.
We ﬁnd the answers using exactmatching in the plot.
We consider the following four cases for train and dev sets:
No Answer exists - We keep the example with an empty answer.
There are 627/116 questions in train/dev.2.
Single Answer - We include single answers found in the plot using ﬁrst matching index.3.
Multiple Answers - We take the ﬁrst answer found and store ﬁrst index in the train set.
In dev set, we store all
the answers and corresponding ﬁrst indices.
Answer exists, but not found in plot - We drop such examples in train, but keep them with empty answers indev.
There are 26596/5768 such examples.
The exact details for processing DuoRC are not provided in the paper, but we assume that this process is similar tochoosing a sample for ﬁne-tuning and prediction.
The authors’ claims should hold true, irrespective of the samplechosen.
After this ﬁrst step of preprocessing, we get 34166/12961 examples in the train/dev sets, which are thenprocessed similarly as SQuAD to get 118676/44831 tokenized features.
3.3 Hyperparameters3.3.1 Fine-tuning
We use hyperparameters similar to the ofﬁcial BERT script 4 while ﬁne-tuning our model.
The train/eval batch sizes arechosen to be 6/8 after discussing with the authors.
AdamW [14] optimizer is used with a learning rate of 3 × 10 − 5 ,a weight decay of 0.01 and an epsilon of 1 × 10 − 6 .
The training is done for 2 epochs.
A polynomial learningrate scheduler with 10% of total training steps as warmup steps is used.
The other default hyperparameters in HF
Transformer’s
TrainingArguments are not changed.
We did not perform any hyperparameter search because the focusof the paper is not to improve the performance of BERT on the tasks, but to analyze its layers after training.
3.3.2 Integrated Gradients
Although the calculation of attributions using IG is performed after training, the choices made can signiﬁcantly affectthe importance distributions based on these attributions.
Hence, we provide a brief description of the same.
We calculate attributions on the softmax outputs of start and end logits from the BERT model.
The target positionschosen are those where start and end logits have the maximum value.
Based on our discussion with the authors, onlythose features which give the best answer for an example during the predictions are chosen.
Reimann Right numericalapproximation is used for calculating the integral value.
The number of steps chosen is 25, and an internal batch size of is used.
We perform IG on only 1000 examples from the dev sets due to computational restrictions.
3.4 Experimental Setup and Code
3.4.1 Fine-tuning BERT
We use HF Transformer’s Trainer to ﬁne-tune BERT on the tokenized features for training and validation with thehyperparameters mentioned in §3.3.1.
Post-training, the predictions for each feature are processed.
We choose thebest valid feature per example based on the best score (start + end logit) for the top 20 start and top 20 end logits.
We discard spans with length above 30 tokens.
We store the respective input token IDs, attention masks, predictedtext, ground start/end positions, predicted start/end positions for the best feature for all examples in a JSON ﬁle.
Weuse SQuAD v1.1 9 and SQuAD v2 10 evaluation scripts for SQuAD and DuoRC, respectively.
This is done as DuoRCpredictions/ground truth may contain no answers.
We consider exact match (EM) and F 1 scores from these evaluations.
3.4.2 Integrated Gradients
Taking the best features from predictions in the JSON ﬁle, we use respective input token IDs and attention masks tocalculate IG.
This eliminates the possibility of having multiple features per example, and improves our sample.
We create a method to ﬁnd the start and end logits given the layer index and the corresponding hidden states.
Wecalculate start and end attributions using Captum’s IntegratedGradients on 1000 randomly chosen examples.
We addstart and end attributions and take a Euclidean norm for each token in the sequence, which is then normalized to getan importance distribution for that sequence.
This is repeated for all 13 layers, including the Embedding Layer.
Thetoken-wise importance scores are stored for each layer and sample.
Note that this process is similar to the algorithmdescribed in the paper, except that they do not calculate attributions on Embedding Layer outputs.
We change the token-wise distributions to word-wise distributions by ignoring the special tokens - [CLS] , [SEP] , [PAD]- and adding importance scores for multiple tokens per word.
The concatenation of the tokens and the addition ofimportance scores is based on the fact that subsequent tokens for a word start with ##.
The offset mapping for eachtoken is used to get the exact word in the passage/question.
The combined scores are re-normalized to get a word-wiseimportance distribution.
Along with this, the word-wise categories - answer, question, context - are stored for eachsample based on the predicted answer spans.
3.4.3 Jensen-Shannon Divergence
For JSD heatmaps, we use pair-wise JSD of all 13 layers (Embedding Layer + Self-Attention Layers).
This gives us a13 × 13 heatmap for each example, which is then averaged over the 1000 examples we chose during IG calculation.
This helps us understand how the layer outputs are different from each other in terms of their attributions.
We use thesame library as the authors - dit [15] - to calculate JSD.
Index 0 in our heatmaps represents the Embedding Layer.
The authors create two heatmaps for each of the datasets - one with top-K token importance scores retained and the restzeroed out, and the other with top-K token importance scores zeroed out.
They chose K=2 for their experiments.
Wecreate similar heatmaps based on the token-wise importance scores generated in §3.4.2 using Seaborn [16] and vary the K values in - 2,5,10.
3.4.4 QA Functionality Tables
We calculate the percentage of predicted Answer Words, Query Words, and Contextual Words (within window size=5of
the Answer Spans) in the top-5 important words for the 1000 examples we chose for IG.
We represent the averagevalues in Tables 1 and 2.
The Query Words are selected using lower-case exact matching in the passage.
Only words inthe passage are considered for the statistical analysis.
3.4.5 t-SNE Representation
We plot t-SNE representations for tokens across multiple layers based on the Query Tokens, the predicted Answer
Tokens, and the Contextual Spans.
All the tokens from the sentence containing the predicted Answer Tokens (betweentwo periods (.)) are chosen as the Supporting/Contextual Spans.
[PAD] tokens are dropped, and only the context tokensare
3.4.6 Quantiﬁer Questions
We search Quantiﬁer Questions using ‘how man’ and ‘how much’.
We use ‘man’ instead of ‘many’ because oftypographical error in some questions.
There are 799 such examples in SQuAD and 310 in DuoRC dev-splits.
Weuse IG on our predicted features for these examples to get the importance distributions.
Then, using the word-wiseimportance scores, we ﬁnd out the percentage of numerical words which are tagged as “Cardinal"(CD) by NLTK’s PoS
Tagger [19].
Additionally, we also include phrases like “thousands", “hundreds" and “two thousand and three" usingthe word2number library.
We calculate the percentages of Numerical Words in top-5 words out of all Numerical Wordsin the passage and average the values.
For EM calculations on Quantiﬁer Questions, we use the respective evaluationscripts.
For calculation of the conﬁdence, we take the maximum of sum of softmax start and end scores.
Then, weaverage these values across the samples to get the average conﬁdence per category.
3.5 Computational Requirements
We use the free NVIDIA K80/T
GPU provided by Google Colab for training the BERT model and calculating
Integrated Gradients.
All other experiments are performed on an Intel i5-6200U quad-core CPU.
For each datasetﬁne-tuning the BERT model takes ∼ 5-7 hours on the GPU; prediction takes ∼ 10-20 GPU minutes; and processing ofpredictions takes ∼ 4-5 CPU hours; Integrated Gradients step takes ∼ 5-6 GPU hours per 1000 examples; JSD Heatmapgeneration takes ∼ 2-3 CPU hours for 1000 examples; while the tables are generated in negligible CPU time.
For
Integrated Gradients on Quantiﬁer Question on dev splits, SQuAD takes ∼ 4 GPU hours, and DuoRC takes ∼ 1.5 GPUhours.
The calculation of conﬁdence on Quantiﬁer/Non-Quantiﬁer Questions takes ∼ 10-20 GPU minutes per dataset.
Results
4.1 Results reproducing original paper
4.1.1 Fine-tuning
The authors achieved F 1 scores of 88.73 and 54.80 on SQuAD and DuoRC dev-splits.
Our ﬁne-tuned BERT modelachieves 88.51 and 50.73 on our tokenized dev-splits of SQuAD and DuoRC.
The performance can depend on theweight initialization of the classiﬁer layer, and differences in the data preprocessing (§3.2).
Additionally, training is astochastic process, hence the weights learned will vary.
Therefore, some variation in the performance is expected.
For SQuAD, we observe a minor change of 0.2 F 1 score.
Additionally, for DuoRC, our way of evaluation (§3.4.1) may bedifferent from the authors, which could be the reason behind the 4 point drop in F 1 score.
Jensen Heatmaps
[Figure]
We plot heatmaps for the JSD values, with top-2 scores in the token-wise distribution, retained and removed, for both
SQuAD and DuoRC (Figure 1).
We see a similar pattern on the heatmap as the paper - the top-2 retained JSD heatmapshave a higher range for both SQuAD (0.05-0.73), and DuoRC (0.10-0.70) across the Self-Attention layers.
The top-2removed JSD heatmaps have lower ranges for SQuAD (0.10-0.42) and DuoRC (0.10-0.32).
The ranges are slightlydifferent from the paper which can be attributed to sampling.
Regardless, this difference shows that the top-2 wordswhich the layers focus on are more different than the rest of the words across the
4.1.3 QA Functionality
We calculate the semantic statistics of the top-5 words for SQuAD and DuoRC as described in §3.4.4.
Table 1 for1000 examples of SQuAD follows the expected trend for Answer Words (37.58%(L1) - 42.94%(L12)), Contextual
Words (33.04%(L1) - 34.02%(L12)), as well as Query Words (22.20%(L1) - 10.42%(L12)).
The reasons behindslightly different percentages could be smaller sampling size and differences in counting Query and Contextual Words.
For DuoRC, Table 2 shows the average statistics for the 1000 examples.
While the Answer Words (11.70%(L1) -12.94%(L12)) and Query Words(19.20%(L1) - 8.68%(L12)) follow the expected trend, the percentage of Contextual
Words remains between 11-12% for the BERT Self-Attention Layers.
In the paper, Contextual Words do not follow anincreasing trend for DuoRC and vary between 15-33%.
For DuoRC, our percentage of Answer Words is low (11-13%)compared to the paper (33-44%).
In addition to the factors stated for SQuAD, DuoRC results can also deviate becauseof possible differences in how predictions are processed, as mentioned in §3.4.1.
Hence, claim 2 holds true.
[Table] [Table]
Claim 3 says that the focus is more on query-passage interaction in the initial layers, while in the ﬁnal layers moreimportance is given to the answer and contextual spans.
From the visualized example in Figure 2, we can see that
the
Query Words - ( percentage/% , increase/increased , agriculture ) - are given more importance in Embedding Layer (L0),
Layers 1,
[Table]
and 3.
While the Answer Words - 17 - and Contextual Words - % - receive more importance in the laterlayers.
This observation shows that the attributions shifts to Contextual and Answer Words in later layers.
Since thisexample is in agreement with the example shown in the original paper
[Table]
we consider claim 3 validated.
[Table]
increased by 17 % to $ 8 .
7 billion .
This represented 24 % of national agricultural production total gross value . ...
L
During 2003 – 04 , the gross value of Victorian agricultural production increased by 17 % to $ 8 .
7 billion .
This represented% of national agricultural production total gross value . ...
L
During 2003 – 04 , the gross value of Victorian agricultural production increased by 17 % to $ 8 .
7 billion .
This represented 24 % of national agricultural production total gross value . ...
L
During 2003 – 04 , the gross value of Victorian agricultural production increased by 17 % to $ 8 .
7 billion .
This represented% of national agricultural production total gross value . ...
L
During 2003 – 04 , the gross value of Victorian agricultural production increased by 17 % to $ 8 .
7 billion .
This represented 24 % of national agricultural production total gross value . ...
L
During 2003 – 04 , the gross value of Victorian agricultural production increased by 17 % to $ 8 .
7 billion .
This represented% of national agricultural production total gross value . ...
L
During 2003 – 04 , the gross value of Victorian agricultural production increased by 17 % to $ 8 .
7 billion .
This represented 24 % of national agricultural production total gross value . ...
[Figure] [Figure]
We plot t-SNE representations in Figure 3 which verify that Claims 4 and 5 hold true.
We observe that in the initiallayers similar words like california, francisco, santa, clara are close to each other.
In the later layers, the Answer Wordsand Query Words separate out.
The Layer 12 plot shows that the model has successfully recognised and separated
Answer and Query Words.
Also, BERT representations of confusing words are closer to each other in the later layers.
On careful observation, we see that Numerical Words like 50 , 2015 and 50th are close to the representation of the
Answer Word 2016 .
This means that BERT tends to focus on confusing words towards the end, as suggested by theauthors.
It is, thus, surprising that BERT is able to perform so well on the task, despite this behavior.
[Figure]
the game was played onfebruary
[Figure]
2016 , at levi
[Figure]
s stadium in the sanfrancisco bay area at santaclara , california
[Figure]
representation
[Figure] [Figure]
was the super bowl played on ?
[SEP] super bowl was an american football game to determine the champion of the national football league ( nfl ) for the 2015 season .
the american football conference ( afc ) champion denverbroncos defeated the national football conference ( nfc ) champion carolinapanthers 2410 to earn their third super bowl title .
the game was played on february , , at levi's stadiuminthe san francisco bayarea at santa clara , california .
as this was the 50th super bowl , the league emphasized the " golden anniversary " withvarious goldthemedinitiatives , as well as temporarily suspend##ing the tradition of naming each super bowl game with roman nu ##meral ##s ( under which the game wouldhavebeen knownas " super bowl l " ) ,sothat the logo could prominentlyfeature the arabic nu ##meral ##s 50 .
t-SNE representation for Question 10 and Layeranswer
[Table]
query words contextual words (
[Figure]
thegamewasplayed on february, 2016 , at levi 'sstadium inthesanfranciscobay areaat santaclara ,california .
as this wasthe
[Figure]
superbowl , the
[Figure]
the
[Figure]
" withvariousgold
[Figure]
themedinitiatives
[Figure]
temporarily
[Figure] [Figure]
the
[Figure]
nu ##
[Figure] [Figure]
( under which the gamewouldhavebeenknownas "
[Figure]
bowl l " ) , sothat the
[Figure]
could prominently feature the
[Figure]
t-SNE representation for Question 10 and Layer
[Figure]
the
[Figure]
the american footballconference
[Figure]
broncos defeated the
[Figure]
footballconference
[Figure]
the gamewasplayedonfebruary ,
[Figure]
at levi
[Figure]
stadium in thesanfranciscobayarea at
[Figure] [Figure]
was the
[Figure]
the leagueemphasized
[Figure]
t-
SNE
[Figure]
for Question 10 and
Layer
[Figure]
a SQuAD Example
4.1.6 Quantiﬁer Questions
For Quantiﬁer Questions, there are two claims - 6 and 7.
For the ﬁrst claim, we perform experiments mentioned in§3.4.6 to ﬁnd out the percentages of Numerical Words in top-5 words out of all such words in the passage.
We observethat this ratio increases as we go higher up in the layers (SQuAD - L1-6.83%, L11-9.44%, L12-9.93%; DuoRCL1-36.21%, L11-51.98%, L12-53.04%).
The ranges of numbers are different from the original paper, but the suggestedtrend is followed.
A reason for this deviation can be differences in counting of Numerical Words.
The author mentionedthat they only used cardinal (CD) Parts-of-Speech during our discussion.
But, a lot of Quantiﬁer Questions do not havecardinal (CD) words in the corresponding context.
The conﬁdence scores for Non-Quantiﬁer Questions (SQuAD79.04% ; DuoRC - 86.33%), are signiﬁcantly lower than for Quantiﬁer Questions with more than one Numerical Word(SQuAD - 85.24% ; DuoRC - 91.20%).
Our EM scores on Quantiﬁer Questions are 86.73% for SQuAD , and 54.65%for DuoRC.
This shows that even when BERT ﬁnds quantiﬁer words increasingly important towards the end layers, it isstill able to perform very well on Quantiﬁer Questions.
Our results validate claims 6 and 7.
4.2 Results beyond the paper
4.2.1 An alternative way of using Integrated Gradients
Before a thorough discussion with the authors, we performed Integrated Gradients in a different manner - we used theground truth positions as targets for attribution calculation, and the logit outputs (instead of softmax outputs).
Notethat this would make all the Layer 12 output attributions zero except for two tokens (which have the ground truth startand end positions).
When using softmax outputs, the token hidden states affect each other in Layer 12 because of thethe normalization term used in softmax, and hence all tokens get some attributions.
The categories for the words werealso chosen based on the ground truth answers.
Multiple features could be sampled from a single answer as the bestfeatures per example were not used.
Surprisingly, the results observed were similar to the paper and can be referred toin
Appendix B. This means that the claims 1 and 2 hold true for both the ways that we calculate IG.
4.2.2 JSD Heatmaps for multiple K values
We plot JSD heatmaps for different values of K (Appendix A).
We see that as we increase K, the range of the values onthe heatmaps reduce.
This means that layers tend to focus on similar words after the ﬁrst few values of K. For K=5, ourtop-5 retained heatmap has a range of 0.06-0.60 for SQuAD, and 0.06-0.59 for DuoRC.
When K is increased to 10, thetop-10 retained heatmap has a range of 0.06-0.56 for SQuAD, and 0.07-0.57 for DuoRC.
We expect these ranges toreduce further as we increase the value of K as the words/tokens will get progressively more similar.
Thus, limiting Kto 5 seems like a good decision on behalf of the authors.
Discussion
Due to lack of computational resources and time constraints, we were unable to perform IG on all dev samples, and thuschose 1000 random samples per dataset.
This can affect the results and statistics signiﬁcantly.
Additionally, the resultsfor DuoRC do not match very well with the authors due to several factors which have been mentioned throughout thereport.
At the same time, we also show that the authors’ claims hold true even for a fraction of the dev set for mostcases, which strengthens their claims.
Through our code, we also provide a system where researchers can extend thisanalysis to other datasets by just deﬁning a dataset class similar to ours, speciﬁcally a method which converts the datasetinto SQuAD format.
We also experimented with an alternative way of performing Integrated Gradients described in§4.2.1.
The results based on the same align with those of the authors and further strengthen their claim.
5.1 What was easy
The authors describe the IG algorithm in their paper, and also provide the link to the code they used to ﬁne-tune BERT.
This helped us to prepare the ﬁne-tuning code easily, and ﬁnd the correct hyperparameters accordingly.
Using HF
Datasets and Transformers reduced our workload signiﬁcantly.
Also, many popular articles and tutorials exist forﬁne-tuning BERT on SQuAD for both frameworks PyTorch and TensorFlow, which can be referred to for any helprequired with implementation.
The pair-wise JSD calculation across the layers was also simple with the help of dit[15].
The plotting of heatmaps, qualitative visualization, and t-SNE scatter plots was also easy because of very-welldocumented libraries - Seaborn and Matplotlib.
5.2 What was difﬁcult
Since the authors do not provide the original code at this time of writing, the conversion from DuoRC to SQuAD formatwas difﬁcult.
DuoRC contains examples which have no answers and multiple answers which may or may not exist inthe original span.
SQuAD, on the other hand has single answers in the training set with a start index provided.
This parttook a lot of time and computation unnecessarily.
IG has also not been described in the paper in great detail, despite themention of the algorithm.
One can use ground truth, max softmax logits, or the predicted positions for target positions.
Similarly, the output considered can be the logits or softmax output of the logits which would change the attributionvalues signiﬁcantly.
This was only clariﬁed through back-and-forth communication with the authors.
Finally, the exactdetails of how the numerical words are counted, what is done when a word is both contextual and a query words, etc.are also not mentioned in the paper, and we had to make our own choices after discussing with the authors.
5.3 Communication with original authors
We communicated with the author - Sahana Ramnath - very frequently for over two weeks.
We list all the signiﬁcantquestions we asked on our repository added as supplementary material.
Appendix AJSD Heatmaps with K=5,
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
(c)
[Figure]
Appendix BOld Integrated Gradients Results
This appendix reports the results we have discussed in §4.2.1:1.
JSD Heatmaps - A range of 0.01-0.85 (L1-L12) for SQuAD and 0.03-0.91 (L1-L12) for DuoRC in the casewhere top-2 scores were retained.
For the case where top-2 scores were removed, the ranges were 0.04-0.51(L1-L12) and 0.04-0.36 (L1-L12) for SQuAD and DuoRC.2.
QA Functionality - For SQuAD, we observed similar trends in the percentages of Answer Words(L130.50%, L11 - 38.18%, L12 - 27.84%).
The Query Words (24.58%(L1) - 4.62%(L12)) and Contextual Words(32.66%(L1) - 34.24%(L12)) also followed the trends.
For DuoRC, however, only the query words (18.66%(L1) - 3.70% (L12) followed a decreasing trend.
Contextual (7.36%(L1) - 6.66%(L12))and Answer Words(4.38%(L1) - 3.26%(L12)) in DuoRC remain more or less constant across the layers.
Refer Tables 3 and 4.
[Figure]
[Table] [Table]
Appendix CPoS Functionality
Based on the paper’s Appendix, we also calculate PoS ratios in top-5 words using NLTK’s PoS Tagger.
Our PoS Tables- Table 5 for SQuAD and Table 6 for DuoRC are shown.
We observe that all layers are majorly focused on entity basedwords with Nouns being 50%-60% of the top-5 words in SQuAD and 60%-70% in DuoRC.
However, in comparisonwith the authors, we observe slightly higher importance to stop words and slightly lower importance to adjectives in thetop-5 words.
The importance on punctuation marks is also higher.
Appendix DQualitative Visualization for DuoRC
We visualize an interesting example for DuoRC in Figure 7.
DuoRC examples contains long passages, which arebroken into several features when we use a maximum length of 384 tokens.
Also, since we allow examples to have noanswers, we store the ﬁrst feature (since there is no best feature) in the predictions in case of no answers.
In Figure 7,although the original example had no predicted answer, the BERT model is still giving importance to the actual answer“Chi-Chi" for the question provided.
This means that there was another feature which predicted “no answer" with ahigher score than this feature which predicted “Chi-Chi" .
However, there is a good chance that the other feature did nothave access to this portion of the context tokens and thus predicted “no answer" .
This implies that we need to re-thinkthe way we postprocess the predicted answers for each example, as some features might predict correct answers despitehaving a lower overall score.
We will explore this further in the future.
[Table]
[Table] [Table]
for classmate Chi - Chi ( Jamie Chung ) to that hes at the center of an intergalactic search for the ...
L ... average teen Goku ( Justin Chatwin ) , who breaks from his wholesale pining for classmate Chi - Chi ( Jamie Chung ) to that hes at the center of an intergalactic search for the ... L ... average teen Goku ( Justin Chatwin ) , who breaks from his wholesale pining for classmate Chi - Chi ( Jamie Chung ) to that hes at the center of an intergalactic search for the ...
L ... average teen Goku ( Justin Chatwin ) , who breaks from his wholesale pining for classmate Chi - Chi ( Jamie Chung ) to that hes at the center of an intergalactic search for the ... L ... average teen Goku ( Justin Chatwin ) , who breaks from his wholesale pining for classmate Chi - Chi ( Jamie Chung ) to that hes at the center of an intergalactic search for the ...
L ... average teen Goku ( Justin Chatwin ) , who breaks from his wholesale pining for classmate Chi - Chi ( Jamie Chung ) to that hes at the center of an intergalactic search for the ...
Appendix E
Recommendations to the authors
While we understand that the authors had to adhere to a certain page limit, the following information, if added, maybeas a supplementary material with the paper, could prove to be beneﬁcial for the reproducibility of the paper:
• The settings for Integrated Gradients :
Speciﬁcally, the choice of target positions, how are start and end
attributions combined, what kind of target outputs (softmax/logits) are chosen, and which features for each
example are chosen, can be mentioned in detail in order to make the results reproducible.
Additionally, whether
the Jensen-Shannon Divergence is calculated on words or tokens can be clearly speciﬁed.
Similarly, for the
rest of the analysis, whether it is performed on the token-wise importance scores or word-wise importance
scores can be clariﬁed.
• The pre-processing and post-processing details for DuoRC SelfRC dataset : SQuAD being a simplerdataset does not usually cause issues, but training a dataset like DuoRC which has combination of abstractive
and extractive question-answering tasks using a span-prediction model is relatively complex.
Which examplesin which stage are discarded and how is the answer chosen in each of the cases mentioned in 3.2.2 can bementioned.
During the post-processing, whether or not the question is allowed to have "no answer" as outputcan also be added.
• Quantiﬁer Questions : The information for ﬁnding the quantiﬁer questions can be added.
For example, wehad to search for ‘how man’ instead of ‘how many’ to get all such questions.
This detail was not mentionedin the paper.
Additionally, how are the Numerical Word percentages calculated can also be described.
Thisdetail is very important to reproduce the tables with high precision.
There can be multiple ways of countingNumerical Words depending on what kind of words are deﬁned as Numerical Words.
How the search isperformed, how is the data tagged, what kind of Part-of-Speech tagger is used, etc. could also be added.
Theequation used for conﬁdence calculation could also be shown because it can be calculated in several ways.• Categorizing the words for t-SNE representation :
There may be words which belong to both query wordsand contextual words .
The authors can describe how this conﬂict is resolved.
Additionally, how are contexualwords chosen, and whether ground truth or predicted answer tokens are used for categorization can also bementioned.
• Alternatives to t-SNE : We note that t-SNE representation varies signiﬁcantly with random initialization,and there could be better, relatively stable choices for dimensionality reduction.
Techniques like PrincipalComponent Analysis (PCA), Uniform Manifold Approximation and Projection (UMAP) can also be used asalternatives.
This will make the visualizations more reproducible.
Alternatively, a short description of why
t-SNE was chosen can also be added.
Appendix F
Ethical Considerations
Since Machine Learning systems will compound and propagate the personal biases that are incorporated when humansprepare datasets, it is important to mention the possible causes and biases that will be present in our training datasets
— SQuAD and
DuoRC here - so that skewness is minimized and fairness is maximized
while
interpreting/looking at
results
of our model.
We also mention that the impact of fairness in Reading Comprehension tasks, in general, is not assevere as in other tasks that directly impact business/important decisions like job recommendations [20], face-baseddatasets [21] etc.
At most, it will lead to a reader’s demographic preference for questions from the corpus
• The SQuAD dataset consists of questions posed by crowdworkers on a set of Wikipedia articles, the demographic distribution and composition, qualiﬁcation/education, their environment of the crowdworkers willaffect the kind of questions they pose on the Wikipedia articles.
The people who have authored those articleswould have written it from their own perspective.
The relationship between the author-reader perspectives willaffect the kind of questions-answers that are prepared.
• The DuoRC dataset consists of QA pairs from two different versions of 7680 plots from IMDb and Wikipedia.
The racial, gender bias incorporated in those movies, in addition to those of the reviewers, plot authors,etc.would be incorporated in the QA pairs.
Additionally, any sarcasm, jokes present in the movie plots will begiven equal importance as any other dialogue/plot, which may result in unfair or biased results during theprediction.
However, the fairness of our model is of a lower-priority in this analysis since there will not be any disparate impact ofour ﬁndings on any minority/historically disadvantaged group.
We do not use sensitive attributes in our study.
Ourfocus, in line with that of the authors of the paper, is more on addressing answerability, accuracy, interpretability of
BERT models.
This means that the approach presented will work efﬁciently with either biased/unbiased data.
Since the BERT model is itself pre-trained on a Wikipedia-based corpus, it is very much possible that unethicalstatements/bias are ingrained in its parameters in some form.
Indeed, there has been work on ﬁnding and mitigatingsocial/intersectional/gender biases in contextualized word representations present in BERT [22–24].
Having mentionedthat, we believe that ﬁne-tuning on a large enough dataset with unbiased examples could mitigate the issue to someextent.
Reiterating, the approach presented in the original paper and discussed here is purely mathematical in nature,and will work equally well on any dataset, biased or unbiased, given enough number of examples.