Reproducibility study - Does enforcing diversity in hidden states of LSTM-Attention models improve transparency?
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
It has been shown (Jain and Wallace, 2019) that the weights in attention mechanisms do not necessarily offer afaithful explanation of the model’s predictions
In the paper ‘Towards
Transparent and Explainable Attention Models’
(Mohankumar et al., 2020) the authors propose two methods to enhance faithfulness and plausibility of the explanations provided by an LSTM model combined with a basic attention mechanism
Scope of Reproducibility For this reproducibility study, we focus on the main claims made in this paper:
• The attention weights in standard LSTM attention models do not provide faithful and plausible explanationsfor its predictions.
This is potentially because the conicity of the LSTM hidden vectors is high.
• Two methods can be applied to reduce conicity: Orthogonalization and Diversity Driven Training.
When
applying these methods, the resulting attention weights offer more faithful and plausible explanations of themodel’s predictions, without sacriﬁcing model performance.
Methodology
The paper includes a link to a repository with the code used to generate its results.
All our experiments with this code are conducted on GPU nodes of the Lisa Cluster at SURFsara 1 .
We follow four investigative routes: (i) Replication: we rerun experiments on datasets from the paper in order to replicate the results, andadd the results that are missing in the paper; (ii) Code review: we scrutinize the code to validate its correctness;(iii) Evaluation methodology: we extend the set of evaluation metrics used in the paper with the LIME method, in anattempt to resolve inconclusive results; (iv) Generalization to other architectures: we test whether the authors’ claimsapply to variations of the base model (more complex forms of attention and a BiLSTM encoder)
Results We conﬁrm that the Orthogonal and Diversity LSTM achieve similar accuracies as the Vanilla LSTM, whilelowering conicity.
However, we cannot reproduce the results of several of the experiments in the paper that underlietheir claim of better transparency.
In addition, a close inspection of the code base reveals some potentially problematicinconsistencies.
Despite this, under certain conditions, we do conﬁrm that the Orthogonal and Diversity LSTM canbe useful methods to increase transparency.
How to formulate these conditions more generally remains unclear anddeserves further research.
The single input sequence tasks appear to beneﬁt most from the methods.
For these tasks, theattention mechanism does not play a critical role for achieving performance.
What was easy / difﬁcult The codebase of the authors is accessible and can be run easily, with good facilities toprepare datasets and deﬁne conﬁgurations.
The Orthogonalization and Diversity Training methods are well explained inthe paper and mostly cleanly implemented.
The larger datasets (Amazon and CNN) are difﬁcult to run due to memoryrequirements and compute times.
The codebase can be hard to navigate, a consequence of the choice to accommodate alarge variation of models and datasets in one framework.
Communication with original authors We reached out to the authors on a fundamental but unexplained choice inthe model architecture but unfortunately did not hear back before the deadline of our assignment.
Introduction
The popularity of attention models has sparked many studies on the interpretability of the attention distributions, withoften conﬂicting claims (Jain and Wallace, 2019;
Wiegreffe and Pinter, 2019; Serrano and Smith, 2019).
Mohankumaret al.
(2020) argue that the reason why attention weights do not always provide a faithful explanation of the model’spredictions is that the learned hidden states of the LSTM based encoder are very similar across time steps, which isexpressed by high conicity of these vectors.
As a result, random permutation of the attention weights leads to a similarﬁnal context vector, which implies the weights do not provide a faithful explanation.
The authors propose two methodsthat force the hidden states of the LSTM to be more diverse.
Orthogonal LSTM ensures low conicity by orthogonalizingthe hidden state at time t with respect to the mean of the previous hidden states.
In Diversity LSTM the model is trainedto jointly maximize the log-likelihood of the training data and to minimize the conicity of the hidden states.
Scope of reproducibility
In this reproducibility study we focus on the authors’ main claim that the Diversity LSTM and Orthogonal LSTMlead to more faithful and plausible explanations, while maintaining accuracy of the predictions.
The authors supporttheir claim by evaluating a series of metrics (Mohankumar et al., 2020) that are assumed to be indicative of levels offaithfulness and plausibility.
We follow four investigative routes:
• Replication: The main part of our study is focused on reproducing the results on the metrics in Mohankumaret al. (2020), and to validate whether we can conﬁrm their observations and conclusions. Furthermore, as the
original paper only presents the results of a selection of models and datasets
we complement the results where possible.
Most notably, we add results on the Orthogonal LSTM that were not in the original paper.
Models, code and datasets are described in Section 3. Our replication results are presented in Section 4; • Code review: As the authors’ code 2 is publicly available, we use their code for the reproduction.
In Section we investigate whether the implementation is consistent with the description of the algorithms in the paper;• Evaluation methodology: In Section 6 we report on our attempt to resolve inconclusive results we found onthe attribution methods by extending the set of evaluation metrics used in the paper with the LIME method 3 ;• Generalization to other architectures:
In Section 7 we test whether the authors’ claims apply to variations ofthe base model (more complex forms of attention and a BiLSTM encoder).
We conclude this paper in Section 8 with a discussion on the conditions under which the proposed methods are mostlikely to be effective, and a reﬂection on our replication study.
Methodology
Code The code accompanying the paper is an extension based on the code ﬁrst developed by Jain and Wallace (2019) 4 .
The entry point of the code is clear and well documented and allows a user to deﬁne speciﬁc jobs using command linearguments for hyperparameters.
Preprocessing routines for the most datasets are included 5 .
Datasets We reran the experiments on 11 of the 14 datasets used in the paper.
The nature and size of the datasetscovers a wide range, from relatively simple binary sentiment classiﬁcation tasks with single input sequence (abbreviated:
SS) (e.g. SST with average input sentence length of 20 words), to complex question answering tasks with dual inputsequences (abbreviated: DS) 6 (e.g. CNN with average document size of 760 words and an average of 26 answercategories).
Some illustrations of data points can be found in Appendix D.
The code repository includes links to thedatasets, as well as the pre-processing routines used by the authors.
We excluded the Amenia and Diabetes datasetsbecause they were not accessible.
The Amazon dataset caused memory issues when running the experiments.
Despitethese issues we were able to get the accuracies and conicity values for this dataset.
Model descriptions
The baseline model (Vanilla LSTM) used in the paper is shown in Figure 1.
For DS tasks, itconsists of two uni-directional LSTM encoders that act on a P-path (for document input phrases) and a Q-path (for question input phrases).
When applied on SS tasks in the paper, only the P-path is used.
An attention decoder is appliedto the hidden states of the P-path LSTM to form the context vector c α on which the model calculates its output.
Thelast hidden state of the Q-path is used as the query term for DS tasks. [Table]
The Diversity and Orthogonal LSTM that Mohankumar et al.
(2020) propose are variants of the baseline model.
The Orthogonal LSTM applies an orthogonalization procedure to the LSTM hidden state vectors during training: thehidden state in timestep t is set to the component that is orthogonal to the mean of previous hidden states.
Thisenforces low conicity of the hidden state vectors h pt .
The Diversity LSTM uses a standard LSTM cell with no explicitorthogonalization, but minimizes conicity jointly with the standard loss.
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table] [Figure]
Hyperparameters Given the wide variety of tasks and datasets, there is an elaborate set of model- and optimizationhyperparameters.
Not all parameter values are indicated in the original paper, some were retrieved by inspecting thecode (an overview is presented in Appendix A).
For all parameters, we used the defaults provided in the original code.
We do not engage in further hyperparameter optimization to stay close to the original paper’s approach.
Note that weare interested in transparency and explainability of the models, not their optimal performance.
Experimental setup and computational requirements We strictly follow the code environment as dictated by therequirements ﬁle that accompanies the code.
All models are run on Nvdia GTX1080Ti GPUs (11Gb VRAM).
Train and evaluation times varied between datasets and model variations, from ca.
5 minutes (SST dataset) to more than40 hours (CNN dataset).
We ran multiple seeds only on a selection of critical datasets to verify that differences weobserved w.r.t.
the results in the original paper were signiﬁcant.
Due to resource constraints, all other comparisonsare based on single seeding, as was done in the original paper.
This means that our observations are indicative, notconclusive.
Replication of the paper’s results4.1 Core replication results
Our reproduction study reveals numerous differences in results reported by Mohankumar et al.
(2020), for all datasetswhere we ran the experiments.
Despite the differences, we support the observation that Diversity and Orthogonal LSTM reach similar accuracies as Vanilla LSTM, and lower conicity values, with the same exception reported in thepaper (CNN).
However, we ﬁnd the claim that Diversity LSTM leads to more transparent attention distributions is notconsistently supported.
For Orthogonal LSTM, some results were omitted in the original paper, and we ﬁnd conﬂictingresults about the effect on faithfulness and plausibility.
We present an overview of the comparisons by metric, and theimpact our ﬁndings have on the main claims of the authors.
Accuracy and conicity Of all accuracy and conicity values reported by Mohankumar et al.
(2020), we are ableto reproduce 86% within a 3%-point margin.
Models and datasets that produced the most notable differences arehighlighted in Table 1.
Despite the different values, the observation that Diversity and Orthogonal LSTM reach similaraccuracies as Vanilla LSTM still holds, except for the CNN dataset.
Also, we can conﬁrm that conicity values are muchlower in Diversity and Orthogonal LSTM, except for CNN in the Diversity LSTM.
The largest difference in accuracywe observe for bAbI3, but the output ﬁles reveal that the model was not done training after the default 200 epochs.
[Table]
Importance of hidden representation Mohankumar et al.
(2020) analyse the importance of hidden representationsusing intermediate representation erasure (Serrano and Smith, 2019) and also by examining the effect of permuting theattentions weights (Jain and Wallace, 2019).
A visual comparison of the box plots about representation erasure in the paper with box plots in our reruns showssimilar results in 25 of the 30 boxes.
Despite the fact that our rerun shows lower medians for the box plots for the
LSTM in IMDB and 20News dataset, the observation still holds that Diversity LSTM and Orthogonal LSTM reach aquicker decision ﬂip for SS tasks.
We concur with the authors’ observations on the paraphrase detection (QQP) and
Q&A task (bAbI1).
In our rerun we see that the quick decision ﬂip that is shown in bAbI1 also occurs in bAbI2 andbAbI3.
Mohankumar et al.
(2020) do not report on SNLI and CNN, where our rerun shows no improvement of the
Diversity LSTM and Orthogonal LSTM models over Vanilla LSTM.
The impact of permuting attention weights is difﬁcult to compare with our results as Mohankumar et al.
(2020) onlyreport a graphical representation (violin plots) of median output difference.
After visual comparison we judge that theoverall results are similar for IMDB, 20News and Yelp.
We also evaluate the median output difference for datasets notreported by Mohankumar et al.
We observe that the results for SST and Tweets show a similar ‘shift to the right’as reported for other binary classiﬁcation tasks.
For DS tasks we observe that Vanilla LSTM already has relatively highmedian output difference, and the Diversity LSTM and Orthogonal LSTM provide less improvement.
We conclude that in our experiments, the Diversity and Orthogonal LSTM do result in quicker decision ﬂips and higheroutput difference for SS tasks, but not consistently for the other tasks.
Comparison with rationales Our rerun of rationale length and rationale attention shows very different results asreported by Mohankumar et al.
(2020), see Table 2.
Although we can conﬁrm that Diversity LSTM results in shorterrationales, we cannot support the claim that Diversity LSTM provides much higher attention to the rationale than
Vanilla LSTM.
In our rerun this only holds for 20News.
The data for the Orthogonal LSTM, which were not reported by Mohankumar et al.
(2020), show much shorter rationalelength, consistent with the paper’s claim.
However, impact on the share of attention on the rationale is mixed: it ishigher for Yelp and 20 News, similar for IMDB and Tweets, but lower for SST.
For DS tasks, the rationale comparison is not implemented by the authors, we suspect because of the high computationalcosts involved for calculating rationales in tasks with multiple output categories.
Comparison with attribution methods The rerun of the correlation metrics shows numerous differences in both
Pearson correlation and JS Divergence.
After studying Pearson correlation, we support the authors’ claim that comparedwith Vanilla LSTM, Diversity LSTM produces attention weights that better correlate with gradients and integratedgradients, although in our results the relative increase of correlation with gradients is smaller: 13% instead of the 65%reported by Mohankumar et al.
However, we do not see the claimed reduction in JS Divergence.
In fact, for
[Table] all datasets the Diversity LSTM produces similar or even higher JS Divergence values than Vanilla LSTM [Table] except JS
Divergence with Integrated Gradients for 20News, see Table 3. The Orthogonal LSTM, for which no correlation data isreported in the paper, is in line with the Diversity LSTM in this respect
[Table]
[Table] importance that is attributed to various POS tags shows similar importanceand ranking for the SST, 20News and Tweets datasets. For Yelp and QQP we get different outcomes. Most notably,with vanilla LSTM model for Yelp we see no attention given to punctuations (PUNC), for which Mohankumar et al.(2020) reports highest attention.
For QQP, Mohankumar et al. (2020) reports 23% on PUNC, while we ﬁnd only 9%. Our results indicate the improvements shown in POS tags are less clear than reported by Mohankumar et al. Human evaluation We could not reproduce the human evaluation within the four-week time frame of our research.
Mohankumar et al.
(2020) reports convincing results, and we also believe human interpretation should play a key rolein judging whether their methods improve transparency.
We include some examples in Appendix D for this purpose.
4.2 Conclusion regarding reproducibility
Our ﬁndings are summarized in Table 4.
We conclude that it is not immediately clear that Diversity LSTM
and
Orthogonal LSTM provide better transparency for all the studied datasets.
The Orthogonal LSTM clearly leads to lower conicity than Vanilla LSTM, but Mohankumar et al.
showlittle evidence with other metrics that indicate higher faithfulness: of the 14 datasets, only 6 boxplots and 4 violin charts are included.
The results observed in our rerun are mixed.
For example, Orthogonal LSTMworks well for 20News, but for SNLI there is hardly any effect on the box plot, and also correlation/JSD with(integrated) gradients is worse.
• For Diversity LSTM, Mohankumar et al.
(2020) show convincing evidence with substantial data.
We observesimilar trends in conicity, and the impact of diversity training is clear in the box plots and violin charts for the
binary classiﬁcation tasks.
However, for the tasks that require two input sequences like SNLI, bAbI2, CNN ourrerun shows that Diversity LSTM does not contribute much to faithfulness and can lead to lower correlation with (integrated) gradients and higher JS Divergence.
Code Review
As part of the reproduction study, we familiarized ourselves with the code to understand how the model and theexperiments had been implemented.
We also scrutinized the code to check whether we could ﬁnd a cause for thedifferences we found in the reported metrics.
The code’s class architecture can accommodate a wide range of tasks, [Table] datasets and model conﬁgurations.
While convenient, this also makes the codebase complex and susceptible to errors.
The code review revealed several debatable choices, of which the main ones are described below.
Orthogonalization of Q-path in dual input sequence tasks For DS tasks, we expect the orthogonalization procedureto only be activated in the P-path (the path of the input document) of the model, as this is the path on which the attentionmechanism applies its weights α t .
However, in the code, orthogonalization is also applied to the Q-path (the path of thequestion phrase in the Q&A tasks, or the second input phrase in SNLI and QQP).
In our view, this introduces a potentially problematic effect.
The attention mechanism uses only the last hidden statevector h qt as the query term.
This representation for the last word in the sequence will only retain the vector componentorthogonal to the mean of the previous word representations, as a result of orthogonalization.
We argue that the directionof h qt in the hidden space will represent the exclusive ‘change of meaning’ that the last word adds to the sequence.
Thisis not a problem in the bAbI tasks, where the prompt word in the question phrase is always the last word (e.g., ‘Whereis Jane’).
But for longer questions where the prompt words appear earlier in the question, this may impede the attentionmechanism from ﬁnding the right prompt words.
[Table]
In order to test this sensitivity, we conduct an experimentfor the simpler SS tasks.
We apply orthogonalization duringtraining and compare model performance when i) attentionweights are left unconstrained vs. ii) all attention weights areset to zero, except for the last hidden state.
The result is shownin Table 5.
What is striking is the performance remains onpar (marked in green) for Vanilla LSTM when only attendingto the last hidden state, indicating the model performs wellwithout the attention mechanism.
However, we observe aperformance drop of 10%-34% (absolute) when attention isconstrained for the Orthogonal LSTM (marked in red).
Indeed,it appears part of the information required for inference is lost.
How this effect impacts the results requires further study.
Itmay explain the accuracy drop from 63% (Vanilla LSTM) to58%/54%(Diversity/Orthogonal LSTM) for CNN as reportedin Table 2 by Mohankumar et al.
We have contactedthe authors to verify their intentions, but did not receive a response prior to submission of this reproduction study.
Disparate calculation of ﬁnal prediction For DS tasks, in the code the ﬁnal prediction layer is implemented as ˆ y =softmax ( W r (tanh( W
p c α + b p + W q h qn + b q ))
+ b r ) .
This deviates from the prediction function ˆ y = softmax ( W 0 c α ) described in Section 2.1 by Mohankumar et al.
However, this does not affect the core architecture, namely LSTM and attention, so we did not modify the code or conduct further experiments.
The models use pre-trained embeddings except for the bAbI datasets.
Words outside ofthe pre-trained embeddings’ vocabulary are initialized with zero-vectors.
All embeddings are ﬁne-tuned (i.e. trainable),independently for the P- and Q-paths for DS tasks.
This is not mentioned in the original paper and this choice isquestionable as it leads to an excessive number of trainable parameters (e.g., >40M for the CNN dataset, see Appendix
A) and training time, while it is unlikely to be critical for the tasks.
Deﬁnition of dev set for bAbI datasets While pre-processing bAbI datasets, 15% of the train set is randomly selectedto be used as dev set, resulting in much higher similarity between these two splits compared to the test set.
As a result,the trained model is overﬁt on the train set, and we observe a large gap between dev and test accuracy.
Extension of the evaluation methods
As discussed in Section 4.1, our rerun of Pearson’s correlation and JS Divergence between attention weights andgradients/integrated gradients points towards a less convincing conclusion.
We therefore also used the LIME framework(Ribeiro et al., 2016) as a third metric for comparing how transparent the attention weights are as explanations, as wellas how much improvements are brought about by the Diversity and Orthogonal LSTM. [Table]
We use LIME to generate a score for the predicted class on eachword-position in the sentence, which can then be comparedwith the attention weights.
For calculating JS divergence wealso rescaled the lime score so that the scores range from 0 to 1,and sums to 1 per sentence (i.e. similar to attention scores).
Theresults are shown in Table 6, where we experimented with onlya representative selection of datasets due to time and resourceconstraints.
Similar to our comparison of attention weights with gradientbased methods, Table 6 indicates Diversity and Orthogonal
LSTM fail to produce explanations consistent with LIME.
It isalso not clear which statistical measure is best for comparingwhether two explanation methods agree with each other.
Inseveral instances (e.g. 20News and Tweets), we observe an increase in Pearson’s correlation and an increase in JS
Divergence at the same time when going from Vanilla LSTM to Orthogonal/Diversity LSTM models.
Generalization to other model architectures
Despite the differences we found between our observations and the observations reported by (Mohankumar et al., 2020),we still see the potential value of the methods they propose.
This is because we did observe sparser attention weightswhen using Diversity and Orthogonal LSTM, and because of the strong preference expressed for the Diversity LSTM inthe human evaluations conducted by Mohankumar et al.
We therefore investigate how well these methods work in alternative settings.
So far the Orthogonalization and Diversity
Training methods are only tested on one-layer uni-directional LSTM models with attention.
However, in many recentstudies, BiLSTM-based attention models or Transformer models are used (Zhou and Wu, 2018;
Lee et al., 2019; Aziz Sharfuddin et al., 2018).
Similarly, more complex attention mechanisms such as self-attention and multi-head attention(Vaswani et al., 2017) gained popularity due to their superior performance.
For this reason, we investigate whether theproposed methods can be extended to more complex models and whether the authors’ two main claims still apply.
Other attention mechanisms The application of more advanced attention mechanisms (such as multi-head attention)poses a challenge because they produce more than one attention weight per word.
It is not straightforward to generateexplanations and word importance based on these weights.
As a consequence, several of the evaluation metrics used bythe authors cannot be applied in their current form.
This would require making non-trivial design choices on how tocombine multiple distributions of the attention weights.
Further research is required to investigate this and whetherexisting methods such as Attention Flow and Attention Rollout (Abnar and Zuidema, 2020) can provide a resolution.
BiLSTM Experiments We replace the uni-directional LSTM in the model (Figure 1) with a bi-directional LSTM.
We choose the BiLSTM architecture, and not a Transformer based architecture, as the latter requires dealing with themore advanced attention mechanisms discussed above.
In order to maintain the decoder’s complexity (the attention mechanism), we preserve the output dimension of the LSTM.
This requires halving the dimension of the hidden states, which also ensures that the number of trainableweights of the BiLSTM is comparable to that of the unidirectional LSTM.
For the Diversity BiLSTM, the same diversityweights are used as in Mohankumar et al.
The conicity term present in the loss function of the Diversity BiLSTM is calculated based on the concatenated forward and backward hidden representations.
Orthogonalization forthe Ortho BiLSTM is applied before concatenation of the forward and backward hidden states.258 Results show that the application of the two methods proposed by Mohankumar et al.
(2020) do not result inperformance loss and do lower conicity.
However, on other metrics and across datasets, the picture is mixed like we sawin our reproducibility results for the unidirectional LSTM, indicating the methods do not unconditionally improveexplanations.
We will not discuss these results in detail, but conclude that it is indeed possible to extend the proposedmethods to BiLSTM attention models.
Full results are included in Appendix C for completeness.
Discussion
Our reproduction shows that enforcing low conicity between the hidden states of an LSTM encoder does not guaranteeimproved transparency in the studied datasets, at least not on the metrics used by Mohankumar et al.
We ﬁndthe authors’ claim about improved transparency not generally applicable and under certain conditions their methodseven hurt accuracy.
Still, the Diversity LSTM and Orthogonal LSTM do lead to improved metrics on some datasets,and the human evaluation Mohankumar et al.
(2020) conducted shows strong preference for the Diversity LSTM over
Vanilla LSTM.
This raises the question under what conditions these methods should be applied.
[Table]
Conditions underlying effectiveness One pattern that seems toemerge is that the beneﬁts of orthogonalizing or diversity training aremost apparent for the relatively simpler SS tasks.
The potential to improve faithfulness of the weights might be high in those cases as it nota given that attention weights carry any meaning for these task.
For some tasks, the LSTM does not strictly need the attention mechanismto perform well, as is shown in Table 7 when the attention mechanismis constrained to be either uniform or attending to the last word only.
In contrast, the more difﬁcult DS tasks do require the attention mechanism in order to reach higher accuracies.
This pattern is similar to thatdescribed by Wiegreffe and Pinter (2019).
We suspect that there is a relation between a) how crucial the attentionmechanism is for performance in a given task, b) how much improvement
Orthogonal/Diversity LSTM can offer w.r.t.
plausibility of the attentionweights for explaining the model’s outputs.
This relationship, and theconditions under which orthogonalization and diversity training offer thebest results, deserves additional investigation.
Reﬂection on our replication study A key insight we have gained is that even with access to the original code, exactreproduction of the results is not guaranteed.
We have not been able to ﬁnd the cause of several differences in results.
The available time and hardware limited our possibilities to repeat these experiments with multiple seeds to ﬁnd anestimate of the variance of outcomes.
Another insight we gained is that the metrics concerning faithfulness and plausibility can be hard to interpret, as it isdeeply entangled with the nature of the dataset as well as the model implementation.
To enable scalable development oftransparent AI models, reliable quantitative metrics are needed that can accurately approximate real humans’ judgement.
We believe further development of transparency metrics is an important area for further research to help build moretransparent models.
Acknowledgement
We’d like to thank [name to be added] for his guidance and insightful discussions.
Appendix A
Details of models and datasets
[Table] [Table]
[Figure]
[Table]
[Figure] [Table] [Figure] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
Color coding:
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Rationale attentionrationale_summary_test.txt - - - - - - - - - -
[Table]
Rationale length rationale_summary_test.txt - - - - - - - - - - - diversity lstm Rationale attentionrationale_summary_test.txt - - - - - - - - - - - Rationale length rationale_summary_test.txt - - - - - - - - - -
[Table] [Table]
[Figure] [Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table]
[Figure] [Figure]
[Figure]
[Table] [Table]
[Figure]
Appendix D: Selected data examples and illustration of model behaviorSST examples
[Table]
Yelp examples
[Table]
bad
[Table]
SNLI examples
[Table]
QQP examples
[Equation] [Equation]
[Table]
[Equation]
[Figure]
[Table] [Table]
Sandra travelled to
[Table]
office .
Daniel
[Table]
to
[Table]
Attentions :
[Table] [Table]
[Table] [Table]
[Table]
Mary travelled to the bathroom .
Sandra took the football there . 0.
0.03 Daniel picked up the apple there .
Sandra went back to the kitchen <EOS> 0.
[Table]
Where
[Table] [Table]
CNN examples
[Table]
, trained with default (tanh) attention (test acc=.595, conicity=.395)
P path : <SOS> ( @entity ) one @entity citizen was killed and another injured in what Attentions : 0.
0.00 police are calling a suspected terror attack wednesday night near @entity .
@entity8 0.
0.00 0.07 spokesman @entity said ayear - old @entity motorist from @entity struck two 0.
0.00 people standing at a bus stop in the @entity section of the city .
one victim , 0.
0.00 identified by police as @entity18 , , died at the hospital .
ayear - 0.
0.00 old woman remains in serious condition , according to @entity .
the driver has been 0.00 0.
0.00 arrested and is under investigation by the @entity .
" from the investigation and first 0.
0.00 findings
, there is a strong suspicion that we 're talking about a terror attack , 0.
0.00 " @entity said .
amid the ongoing investigation , a magistrate court has issued a 0.00 0.
0.00 gag order on details of the incident .
<EOS> 0.
Q path : <SOS> the suspect is a 37 - year - old @placeholder from @entity11 , @entity1 police say <EOS>
Answer: @entity10 Predicted: @entity
CNN, Ortho-LSTM, trained with default (tanh) attention (test acc=.536, conicity=.099)
P path : <SOS> ( @entity ) one @entity citizen was killed and another injured in what Attentions : 0.
0.00 police are calling a suspected terror attack wednesday night near @entity .
@entity8 0.
0.03 0.00 spokesman @entity said ayear - old @entity motorist from @entity struck two 0.
0.00 people standing at a bus stop in the @entity section of the city .
one victim , 0.
0.00 identified by police as @entity18 , , died at the hospital .
ayear - 0.
0.01 old woman remains in serious condition , according to @entity .
the driver has been 0.00 0.
0.00 arrested and is under investigation by the @entity24 .
" from the investigation
[Figure]
findings , there is a strong suspicion that
[Figure]
on details of the
[Figure]
: <
[Figure]
>
[Figure]
suspect is
[Figure] [Figure]