T owards Transparent and Explainable Attention Models, ML
Reproducibility
Anonymous submission
Reproducibility Summary
Scope of Reproducibility
The original paper proposes new LSTM variants that are supposed to improve the transparency and interpretability of the attention mechanism.
They claim that modi(cid:2)ed LSTMs do not hurt the performance (Claim 1) and the attention distributions in the modi(cid:2)ed model variants provide a faithful explanation of the model’s predictions (Claim 2).
Additionally, we provide our extensions.
We verify whether the mean conicity of the attention values in the Vanilla
Transformer is similar to the conicity of the VanillaLSTM (Claim 4), whether the performance of a modi(cid:2)ed Transformer does not signi(cid:2)cantly hurt performance across different datasets (Claim 5) and if the orthogonalization procedure has homogeneous effects across 4 languages on the sentiment classi(cid:2)cation task (Claim 6).
Methodology
We have used the original author’s code and the experiments were run on the Lisa cluster with GTX1080Ti GPU.
For experiments that extended the original paper’s results we have re-implemented part of the existing pipeline and added respective job scripts that were run on the cluster.
Results
All the claims stated by the original paper were con(cid:2)rmed.
However,not all our experiments overlap with corresponding experiments in the original paper.
When reproducing the experiment about POS tags for QQP dataset [ 1 ] we did not obtain large cumulative attention to punctuation marks and our results of the rationale experiment are not conclusive.
Additionally, we found the Vanilla Transformer model had 3 times smaller conicity values than the VanillaLSTMs.
Constraining the conicity in the transformer’s self-attention mechanism did not signi(cid:2)cantly hurt performance.
What was easy
It was easy to run the code published by the authors and in our opinion, overall the paper is very good from a reproducibility point of view.
What was dif(cid:2)cult
Some of the datasets (MIMIC ICD9 Anemia and MIMIC ICD9 Diabetes [ 2 ]) were not easily available due to licensing issues.
There were couple of very large datasets (Yelp [ 3 , 4 ]) and Amazon [ 3 , 5 ]) that took too much time and computational resources to run (more than 24 hours on GTX1080Ti).
It’s also important to note that the package runs either on Linux or MacOS with Python 3.7.
For better reproducibility, we would recommend the authors to include a table with dataset sizes or their training times.
Communication with original authors
We did not have any contact with the original authors.
Preprint.
Under review.
Introduction
Attention-based mechanisms [ 6 ] have recently gained popularity due to the boost in performance and robustness that they offer across problems in the Natural Language Processing domain.
Another convenient feature of those models is that the attention distribution can be visually analyzed to (cid:3)ag words and phrases that trigger the model’s decision-making.
Although this approach has been widely adopted by the research community [ 7 ] as a way to improve the model’s explainability, some scientists argue that this method provides neither faithful nor plausible insights [8].
Authors of [ 9 ] suggest that this might be due to the lack of variability in the hidden state representations.
To overcome this problem, they introduce diversity-driven training and the orthogonalization for LSTMs to increase the variability of the hidden states.
In this work, we attempt to reproduce the results that were carried out in [ 9 ] and extend it with multilingual datasets [ 10 ].
Additionally, we apply their idea to Transformer models [ 11 ] and ensure a reproducible environment by containerizing the software 1 with Docker [ 12 ].
We have managed to reproduce most of their results, however,some datasets were not easily available due to the licensing issue, or were too large to process.
Scope of reproducibility
The original paper introduces an extension to the LSTM models, which makes the model more transparent.
The goal is to make the hidden layers correspond to the explainability, so the insights into the model are not only plausible but also faithful.
The main claims that we were able to reproduce from the original paper are as follows: (cid:149) Claim 1:
The performance of modi(cid:2)ed LSTM variants does not signi(cid:2)cantly (1-3% accuracy change) hurt performance across different tasks and datasets.
(cid:149) Claim 2:
The attention distributions in the modi(cid:2)ed model variants provide a faithful explanation for the model’s predictions which is exhibited by low-fraction of hidden state deletions leading to a different outcome (Figure 2) and by large output difference when the attention weights are randomly permuted (Figure 3).
Additionally, we introduce other claims from the extended work on Transformers and multilingual data: (cid:149) Claim 3:
The mean conicity of the attention values in the VanillaTransformer is similar to the conicity of the
VanillaLSTM.
(cid:149) Claim 4:
The performance of a modi(cid:2)ed Transformer (an extension) does not signi(cid:2)cantly (1-3%) hurt performance across different datasets.
(cid:149) Claim 5:
The orthogonalization procedure has homogeneous effects across 4 languages on the sentiment classi(cid:2)cation task.
Methodology
We used the publicly available code provided by the author 2 .
We could run experiments without major issues by following instructions in the Readme (cid:2)le 3 .
3.1 Model descriptions
In [ 9 ], there were two proposed LSTM variants - Orthogonal LSTM and Diversity LSTM which were compared to the vanilla LSTM as a baseline.
3.1.1 Vanilla LSTM
Depending on task, we use a single or a pair of input, for double version we encode
P = f w p1 ; :::; w pm g ; Q = f w q1 ; :::; w qn g by passing their embedding through LSTM encoder.
h pt =
LSTM
P (cid:0) e ( w pt ) ; h pt (cid:0) 1 (cid:1) 8 t 2 [1 ; m ] h q t =
LSTM
Q (cid:0) e ( w qt ) ; h qt (cid:0) 1 (cid:1) 8 t 2 [1 ; n ] Our code is availableat: https://anonymous.4open.science/r/1bbc9de7-18c6-4c39-a67e-a4596874ba6b/
Authors’ repository is available at https://github.com/akashkm99/Interpretable-Attention
The only minor step missing from the manual is the need to download NLTK POS taggers in the environment by running nltk.download(’averaged_perceptron_
With e ( w ) as embedding for the word w .
We attend to the intermediate representations of
P; H p = f h p1 ; :::; h pm g
R mxd using the last hidden state h qn 2 R d as the query, using the attention mechanisms.
~(cid:11) t = v T tanh (
W 1 h pt + W 2 h qn + b ) 8 t 2 [1 ; m ] (cid:11) t
= softmax (~ (cid:11) t ) c (cid:11) =
P mt = (cid:11) t
h p t
For tasks with a single input sequence, we use a single LSTM to encode the sequence, followed by an attention mechanism (without query) and a (cid:2)nal output projection layer.
3.1.2 Diversity LSTM
A further extension to the vanilla LTSM model with an additional objective to minimize the conicity, together with maximizing the log-likelihood of the training data, while training the model.
L ( (cid:18) )
= (cid:0) p model ( y j P ; Q ; (cid:18) )
+ (cid:21)conicity (cid:0) H P (cid:1)
Where y is a ground truth class,
P and
Q are the input sentences,
H p = h p1 ; :::; h pm 2 R m (cid:2) d contains all the hidden states of the LSTM, (cid:18) is a collection of the model parameters and p model ( : ) represents the model output probability.
(cid:21) is a hyperparameter that controls the weight given to diversity in hidden states during training.
3.1.3 Orthogonal LSTM
A model similar to vanilla LSTM, except that we replace vanilla LSTM with orthogonal LSTM architecture.
To ensure the low conicity, we orthogonalize the hidden states.
f t = (cid:27) ( W f x t + U f h t (cid:0) 1 + b f ) i t = (cid:27) ( W i x t
+ U
i h t (cid:0) 1 + b i )
o t = (cid:27) ( W o x t + U
o h t (cid:0)
+ b o ) ^c
t = tanh (
W c x t + U c h t (cid:0) 1 + b c ) c t = f t (cid:12) c t (cid:0)
+ i t (cid:12) ^c t ^h t =
o t (cid:12) tanh ( c t ) h t =
P t (cid:0) 1 i
= h
i h t = ^h t (cid:0) ^h
Tt h t h
Tt h t h t where
W f ; W i ; W o ; W c 2 R d 2 (cid:2) d 1 ; U f ; U i ; U o ; U c 2 R d 2 (cid:2) d 1 ; b f ; b i ; b o ; b c 2 R d 2 ; d 1 and d 2 are the input and hidden dimentions respectively.
3.1.4 Extension: Transformer
We further investigate the impact of orthogonalization of attended states on another attention-based model architecture: the transformer [ 13 ].
In this paper we focus on classi(cid:2)cation tasks, containing one input sequence.
This section treats the transformer block used in later experiments.
We (cid:2)rst describe the self-attention layer.
The main ingredient thereof, the dot product attention, takes as input a set of queries
Q 2 R T (cid:2) k , keys
K 2 R T (cid:2) k and values
V 2 R T (cid:2) k
where
T is the sequence length, and k is the hidden dimension of the queries, keys, and values.
Given a k dimensional representation of the input tokens and positions
X = f x 1 ; :::x
T g
R T (cid:2) k , we de(cid:2)ne the queries, keys and values as linear transformations of the i -th input vector where the transformation matrices are learnable, k (cid:2) k parameters.
To enable attending to different aspects of the sequence we use a multi-head attention mechanism, where the number of heads is denoted by h .
Multi-head attention creates h separate attention mechanisms, concatenates them, and learns the interaction between them through a linear layer, parametrized by
W o 2 R h (cid:1) k (cid:2) t that brings the dimension from
T (cid:2) h (cid:1) k back to
T (cid:2) k .
This can be summarized by the following equations:
Multihead ( Q; K; V ) =
Concat ( head 1 ; :::; head h ) W O ; head i =
Attention ( Q i ; K i ; V i )
Attention ( Q; K; V ) =
softmax (cid:18) QK
T p k (cid:19)
V; Q =
XW q ; k =
XW k ; V =
XW v
Figure 1: Visualization of a transformer block [11].
Now we relate back to the orthognalization procedure of the paper by Mohankumar et al.
In the self-attention layer described above, for each attention-head, each x i will have a corresponding attention output, y i which is a weighted average over the attention values, v i .
This can be described by the equation y i =
P
Tj ( softmax (cid:16) QK
T p k (cid:17) ) ij v j , where
V = f v 1 ; :::v
T g
R T (cid:2) k .
Following the main conjecture of [ 9 ], we aim to minimize conicity of representations being attended to, in this case the attention values.
In particular, we minimize the average conicity of the values coming from different attention-heads.
Finally, we describe the transformer block, visualized in Figure 1.
It consists of (1) self-attention layer, (2) layer normalization over the embedding dimension, (3) a feed-forward layer applied independently to each vector, and (4) another layer normalization.
Residual connections are added between the input and (1) as well as between (2) and (3).
Finally, the max-pooling operation is applied to the output of the transformer block along the time dimension followed by a linear layer that brings down the dimension to the number of classes.
Weinvestigate two variants of the transformer: a baseline,
VanillaTransformer to minimize the negative log-likelihood of the training data.
A second variant referred to as
Diversity Transformer additionally minimizes the mean conicity of the attention values across attention heads.
L ( (cid:18) )
= (cid:0) p model ( y j X; (cid:18) )
+ (cid:21)
h
h X i conicity ( V i ) ; where y is the ground truth class,
X is the input sequence,
V i = f v i1 ; :::; v iT g
R T (cid:2) k contains the attention values of the i -th head, (cid:18) is a collection of the model parameters and p model ( : ) represents the model output probability.
(cid:21) is a hyperparameter that controls the weight given to diversity in attention values during training.
3.2 Datasets 3.2.1 Original datasets
The original paper used 12 datasets which are well described by [ 9 ] and [ 14 ].
Therefore we refrain from repeating their description in this work, but at the same time strongly advise the reader to familiarize himself with the relevant sections before going further.
We managed to reproduce 7 out of these 12 datasets.
Two datasets (
Anemia and
Diabetes ) are part of a larger medical database which is only available after going through an online course and sending an application - due to time constraints we didn’t try to obtain them.
Another dataset that we didn’t use is the
CNN dataset because the download link expired and as con(cid:2)rmed by the author is no longer available.
Finally, we didn’t obtain results for the
Yelp [ 3 , 4 ] and
Amazon [ 3 , 5 ] datasets because they were too computationally too expensive to train.
Regarding the datasets that we were able to reproduce, all of them were automatically downloaded using a script as described in the repository apart from
Twitter
ADR dataset which we obtained from authors of [14] via email.
All reproduced text classi(cid:2)cation datasets fall in the category of binary classi(cid:2)cation and all of them are almost balanced (less than 1% difference in class distribution).
In Table 1 we include a number of examples for training, development, and testing sets of all reproduced datasets.
split/dataset 20News
IMDB
Tweets
SST
Babi
Babi Babi
SNLI
QQP train       549, 327, devtest
Table 1: Number of examples per split for all reproduced datasets.
3.2.2 Cross-language sentiment classi(cid:2)cation
In this work, we have extended the experiments by including the sentiment classi(cid:2)cation task on the cross-lingual dataset CLS 4 .
This dataset is the most similar to the Amazon dataset that was used in the original paper [ 9 ].
The dataset has been already preprocessed to balance the labels by original authors [ 10 ].
The dataset consists of 12.
data samples for each of the 4 languages: English, German, French, and Japanese.
The dataset is split into train, test, and dev datasets that consist of 6000, 3000, and 3000 samples respectively.
Originally, the dataset was divided into subcategories of reviews (book, movies, and DVD), however, for the sentiment classi(cid:2)cation we use all categories together.
We provide a script available at our code repository that automatically downloads, preprocesses, and tokenizes the data.
The preprocessing is carried out as in the original paper’s implementation [ 10 ].
Firstly, we split words on delimiters, map digits into < num> token, remove punctuation, and convert to lower case letters.
Additionally, for
English data we normalize contraction (e.g. "don’t" to "do not").
Finally, we convert corresponding sentiment labels (i.e. "positive", and "negative") to integer representations.
Results and Discussion
In the following section, we compare our reproduced results to the results shown in the original paper.
We brie(cid:3)y discuss whether claims of the original paper are supported by our experiments.
4.1 Results reproducing original paper
Table 2 shows that Claim 1 does hold in our results.
Our results are similar to the original paper, apart from BAbIand 3 for which the accuracy differs signi(cid:2)cantly.
However,in general, the performance is comparable for all model variants and the conicity is signi(cid:2)cantly lower for Diversity and Orthogonal LSTM.
[Table]
In Figure 2 we check what is the fraction of removed attention weights required to change the prediction.
In theory, a low fraction removed before decision (cid:3)ip con(cid:2)rms the faithfulness of the model.
We observe that our and the original results are almost identical which con(cid:2)rms Claim 2. Interestingly, the Tweets dataset requires a larger fraction of hidden 4 Dataset is available at https://zenodo.org/record/3251672#.YAll0-hKibg states to be deleted than sentiment classi(cid:2)cation datasets.
That is perfectly reasonable as classifying whether a tweet contains adverse drug reaction is more nuanced than sentiment analysis and requires attention to a larger fraction of the input sentence.
SNLI presents similar behavior to QQP as both tasks concern relationships between a pair of input sentences.
[Figure]
permutation sentence pair tasks the attention weights are more homogenous therefore the permutation has a
[Figure]
Dataset
VanillaLSTM
Diversity LSTM
Orthogonal LSTM
RationaleAttention
Rationale Length
RationaleAttention
Rationale Length
RationaleAttention
Rationale Length
Our results SST 0.
0.183 IMDB 0.
0.356 20News 0.
0.201 Tweets 0.
0.326 Original results
SST 0.
0.175 IMDB 0.
0.169 20News 0.
0.173 Tweets 0.
it seems that this observation which occurred in 2 out of 12 datasets is more of an artifact of these two datasets, rather than a general problem.
That is not to say that the vanilla model has plausible explanations, but rather that this particular experiment cannot properly verify this claim.
[Figure]
Our results indicate the vanilla and Diversity LSTMs.
Finally, we consider rationale which is a minimal set of tokens required to make the correct prediction. The rationale length is the fraction of tokens included in a rationale out of all sentence tokens and rationale attention is cumulative attention of the words that are included in the rationale out of all words in the input sentence. High rationale attention and low rationale length mean that the model focuses on a couple of important words. Therefore rationale attention is positively correlated with rationale length - the more words are in the rationale, the more cumulative attention is attributed.
We see in Table 3 that the cumulative rationale attention of vanilla LSTM in our results is similar or sometimes higher than Diversity LSTM and much higher than vanilla LSTM in the original results.
This speaks in favor of denying Claim 2.
However,the length is also much higher so it might be just a matter of the hyperparameter determining rationale length which was used for generating the rationale.
That’s why this experiment neither con(cid:2)rms nor denies this claim and would require further research to draw a de(cid:2)nitive conclusion.
4.2 Multilingual Dataset
Table 5 shows the results for 4 different languages and 3 LSTM models averaged over 5 runs.
The results show a similar trend as in the original paper [ 9 ].
The best performance is achieved for vanilla LSTM models, however,the conicity values are a few times bigger than for the Diversity and Orthogonal variants.
There is a 3-5% decrease in accuracy for
Diversity models and 1-3% for Orthogonal models when compared to the vanilla LSTM.
The conicity values are very similar between Diversity and Orthogonal LSTMs which might suggest that the latter one achieves a better performance while maintaining a good level of explainability. Also, it is worth noting that original authors achieved much higher
Dataset
Vanilla Transformer
Diversity Transformer
Accuracy
Conicity
Accuracy
Conicity
Binary Classiﬁcation
CLS English 78.09 (cid:6) 1 : :(cid:6) 0 : :(cid:6) 1 : :(cid:6) 0 :
CLS German 77.49 (cid:6) 2 : :(cid:6) 0 : :(cid:6) 0 : :(cid:6) 0 :
CLS French 80.20 (cid:6) 1 : :(cid:6) 0 : :(cid:6) 1 : :(cid:6) 0 :
CLS Japanese :(cid:6) 5 : :(cid:6) 0 :75.53 (cid:6) 1 : :(cid:6) 0 :
SST 71.25 (cid:6) 1 : :(cid:6) 0 : :(cid:6) 2 : :(cid:6) 0 :
IMDB :(cid:6) 1 : :(cid:6) 0 :81.48 (cid:6) 0 : :(cid:6) 0 :20News_sports :(cid:6) 1 : :(cid:6) 0 :78.71 (cid:6) 2 : :(cid:6) 0 :
Tweets 89.89 (cid:6) 1 : :(cid:6) 0 : :(cid:6) 0 :
[Table]
[Table]
accuracy scores on the Amazon dataset (the best accuracy is 93.73%).
This is because we don’t use any pre-trained word embeddings during the preprocessing steps due to inconsistency between languages.
4.3 Transformers
Table 4 portrays the accuracies and mean conicity of the attention values for various datasets.
The purpose of this experiment is to verify if the transformer’s self-attention mechanism also tends to attending to highly similar token representations which in the original paper [9] caused the attention weights to carry little meaning.
We ﬁnd that, at least for classiﬁcation tasks, the conicity of the attention values in the Vanilla Transformer is signiﬁcantly smaller (mean 0.24) compared to the conicity of the hidden states (mean 0.58) in the Vanilla LSTM’s.
This leads to the rejection of
Claim 3, as these values differ signiﬁcantly.
This result suggests that the problem of the attention weights failing to offer a plausible explanation of the prediction might be less severe in the Vanilla Transformers as compared to the
Vanilla LSTMs.
However, subsequent experiments would have to be conducted to conﬁrm this statement.
Similarly to the ﬁndings from Section 4.1, we ﬁnd that the performance of diversity-driven training does not signiﬁcantly differ from the performance of vanilla transformer (mean difference 1.15), which conﬁrms Claim 4.
Conclusions
The majority of the experiments from the paper were reproducible.
In the reproduced results, we have also noticed that modiﬁed LSTM variants do not signiﬁcantly degrade the performance.
Additionally, our results seem to support the claim that the LSTM variants provide more faithful explanations for the model’s predictions.
Going a step further, the experiments were extended by adding multilingual datasets and repeating part of the experiments on the Transformer model.
Interestingly, a similar performance drop across LSTM variants could be observed for multilingual datasets.
The vanilla Transformer model had much smaller conicity values than the vanilla LSTMs.
However, the diversity-driven training does not impact the performance of Transformer variants.
Overall, we are quite pleased with obtained results and the quality of original paper’s implementation.