T OWARDS V ERIFIED R OBUSTNESS UNDER T EXT D ELETION I NTERVENTIONS
A BSTRACT
Neural networks are widely used in Natural Language Processing, yet despite their empirical successes, their behaviour is brittle: they are both over-sensitive to small input changes, and under-sensitive to deletions of large fractions of input text.
This paper aims to tackle under-sensitivity in the context of natural language inference by ensuring that models do not become more conﬁdent in their predictions as arbitrary subsets of words from the input text are deleted.
We develop a novel technique for formal veriﬁcation of this speciﬁcation for models based on the popular decomposable attention mechanism by employing the efﬁcient yet effective interval bound propagation (IBP) approach.
Using this method we can efﬁciently prove, given a model, whether a particular sample is free from the under-sensitivity problem.
We compare different training methods to address under-sensitivity, and compare metrics to measure it.
In our experiments on the
SNLI and MNLI datasets, we observe that IBP training leads to a signiﬁcantly improved veriﬁed accuracy.
On the SNLI test set, we can verify 18.4% of samples, a substantial improvement over only 2.8% using standard training.
I NTRODUCTION
Natural language processing (NLP) widely relies on neural networks, a model class known to be vulnerable to adversarial input perturbations (Szegedy et al., 2013; Kurakin et al., 2016).
Adversarial samples typically expose over-sensitivity to semantically invariant text transformations (Belinkov &
Bisk, 2017; Ettinger et al., 2017), e.g. character ﬂips (Ebrahimi et al., 2018) or paraphrases (Ribeiro et al., 2018b; Iyyer et al., 2018).
Feng et al.
(2018) exposed another type of problematic behaviour: deleting large parts of input text can cause a model’s conﬁdence to increase; Figure 1 shows an example.
That is, reduced sets of input words can sufﬁce to trigger more conﬁdent predictions.
Such under-sensitivity is problematic: neural models can ‘solve’ NLP tasks without task-relevant textual comprehension skills, but instead ﬁt spurious cues in the data that sufﬁce to form correct predictions.
Models might then achieve strong nominal test accuracy on data of the same (biased) distribution as the training set, by exploiting predictive shortcuts that are not representative of the given NLP task at hand.
Consequently, they fail drastically when evaluated on samples without these spurious cues (Jia & Liang, 2017; Poliak et al., 2018; Gururangan et al., 2018; Niven & Kao, 2019).
A major issue with identifying reduced inputs is the combinatorially large space of arbitrary text deletions; this can only be searched exhaustively for short sequences.
Prior work has considered heuristics like beam search (Feng et al., 2018) or bandits (Ribeiro et al., 2018a), but these are generally not guaranteed to ﬁnd the worst-case reductions.
In this work, we address the under-sensitivity issue by designing and formally verifying the undersensitivity speciﬁcation that a model should not become more conﬁdent as arbitrary subsets of input words are deleted.
1 Under-sensitivity behaviour is not reﬂected in nominal accuracy, but one can ∗ internship DeepMind. This speciﬁcation is discussed in Section 3. Although a conservative choice, we ﬁnd it is rarely satisﬁed.
[Figure]
instead use this speciﬁcation to measure and evaluate the extent with which samples exhibit undersensitivity.
Instead of better, yet still imperfect search heuristics, we describe how interval bound propagation (IBP)
(Gowal et al., 2018; Mirman et al., 2018) – a formal model veriﬁcation method – can be used to efﬁciently cover the full reduction space, and verify the under-sensitivity speciﬁcation.
IBP can be applied at test time to arbitrary model inputs to verify whether or not they are undersensitive; but it can also be used to derive a new auxiliary training objective that leads to models veriﬁably adhering to this speciﬁcation, and which we ﬁnd generalises to held-out test data.
While under-sensitivity has been demonstrated for several NLP tasks (Feng et al., 2018), we chose to study the use case of natural language inference (NLI) (Dagan et al., 2006; Bowman et al., 2015) in particular as a representative task: sequences are comparatively short, datasets large, and the label complexity is small.
We investigate the veriﬁcation of the popular decomposable attention model (DAM) 2 (Parikh et al., 2016) in detail.
This architecture covers many of the neural layer types of contemporary models, and we focus on a detailed description for how IBP can be leveraged to efﬁciently verify its behaviour.
We then experimentally compare various training methods addressing under-sensitivity: i) standard training ii) data augmentation iii) adversarial training iv) IBP-veriﬁed training and v) entropy regularisation, and evaluate their effectiveness against nominal (test) accuracy, adversarial accuracy, IBP-veriﬁed accuracy and a veriﬁcation oracle.
To summarise, the main contributions of this paper are (1) Formalisation of the problem of verifying an under-sensitivity speciﬁcation, (2) Veriﬁcation of the Decomposable Attention Model using Interval Bound Propagation, and (3) Empirical analysis of the efﬁcacy of (i) different evaluation methods for verifying robustness; and (ii) different training methods for developing veriﬁably robust models.
R ELATED W ORK
Natural Language Inference.
Natural Language Inference (Dagan et al., 2006) is the task of predicting whether a natural language premise entails a natural language hypothesis.
The availibility of large-scale datasets (Bowman et al., 2015; Williams et al., 2018) has spurred a profusion of neural architecture development for this task, e.g. (Rockt¨aschel et al., 2016; Parikh et al., 2016;
Chen et al., 2017), among many others.
Adversarial Vulnerability in NLP.
There is a growing body of research into NLP adversarial examples, each using a slightly different choice of semantically invariant text transformations, or a task-speciﬁc attack.
A ﬁrst class of attack considers word- and character-level perturbation attacks (Ebrahimi et al., 2018;
Alzantot et al., 2018)
while another type of attack exploits back-translation systems to either mine rules (Ribeiro et al., 2018b) or train syntactically controlled paraphrasing models (Iyyer et al., 2018).
Li et al.
(2017) use syntactic and lexical transformations, whereas Belinkov & Bisk (2017) investigate synthetic and natural noise in Machine Translation.
Jia & Liang (2017) and Mudrakarta et al.
(2018) introduce task-speciﬁc adversarial attacks for Reading Comprehension/QA, Zhao et al.
(2018) for Machine Translation and NLI, and Thorne & Vlachos (2019) for Fact Checking.
In NLI in particular, Minervini & Riedel (2018) penalise adversarially chosen [Figure] logical inconsistencies in NLI predictions, Kang et al.
(2018) use background-knowledge guided adversaries, and Glockner et al.
(2018) utilise lexical entailment relationships.
Ribeiro et al.
(2016) and Ribeiro et al.
(2018a) describe analysis tools that have uncovered model over-sensitivity and under-sensitivity, respectively.
Feng et al.
(2018) focus on under-sensitivity, showing that models can become more conﬁdent as large fractions of input text are deleted, whereas
Niu & Bansal (2018) address under-sensitivity in a dialogue setting.
Jacobsen et al.
(2019) demonstrated a link between excessive prediction invariance and model vulnerability in computer vision.
Formal Veriﬁcation.
Formal veriﬁcation provides a provable guarantee that models are consistent with a formally deﬁned speciﬁcation (a mathematical relationship between the inputs and outputs of the model).
Examples of speciﬁcations include robustness to bounded adversarial perturbations, monotonicity of the output with respect to a subset of the inputs, and consistency with physical laws (Qin et al., 2019).
Literature can be categorised into complete methods that use Mixed-Integer Programming (MIP) (Bunel et al., 2017; Cheng et al., 2017) or Satisﬁability Modulo Theory (SMT)
(Katz et al., 2017), and incomplete methods that solve a convex relaxation of the veriﬁcation problem (Weng et al., 2018; Wong & Kolter, 2018;
Wang et al., 2018; Raghunathan et al., 2018b).
Complete methods perform exhaustive enumeration to ﬁnd a counter-example to the speciﬁcation or rule out the existence of counter-examples (thus proving that the speciﬁcation is true).
Hence, complete methods are expensive and difﬁcult to scale.
Incomplete methods are conservative (i.e. they cannot always prove that a speciﬁcation is true even when it is), but are more scalable and can be used inside the training loop for training models to be consistent and veriﬁable (Raghunathan et al., 2018a; Wong & Kolter, 2018; Dvijotham et al., 2018a; Gowal et al., 2018; Dvijotham et al., 2018b).
While Barr & Klavans (2001) address the issue of veriﬁcation in NLP, most of the recent work has focused on (cid:96) ∞ norm-bounded perturbations for image classiﬁcation.
This paper complements work on incomplete veriﬁcation methods by extending IBP to NLI where inputs are inherently discrete (to the contrary of images which are continuous).
In the NLP context in particular, Huang et al.
and Jia et al.
(2019) have very recently veriﬁed CNN, and LSTM models with speciﬁcations against over-sensitivity adversaries under synonym replacement.
Wang et al.
(2019) study veriﬁcation of output length speciﬁcations in machine translation models, showing that the outputs of machine translation and image captioning systems can be provably bounded when the inputs are perturbed within a given set.
In contrast, this work examines under-sensitivity behaviour: excessive model prediction invariance under arbitrary word combination deletions.
We highlight that the veriﬁcation of neural networks is an extremely challenging task, and that scaling complete and incomplete methods to large models is an open problem.
F ORMULATING A S PECIFICATION AGAINST U NDER - SENSITIVITY
Neural networks are expressive models that can ﬁt large datasets and achieve strong nominal test accuracy.
At the same time however, they can ﬁt data in a way that violates our idea of how they should ﬁt it, from an input attribution perspective.
Figure 2 visualises the extent of the problem in NLI: it is, for example, for 20% of the SNLI test set possible to delete 78% or more of premise words while the prediction conﬁdence increases or remains the same.
We will next formally describe a speciﬁcation that checks a model’s under-sensitivity, i.e. whether any such reduction exists.
The speciﬁcation addresses model output probabilities when parts of the input text are deleted.
To this end, we ﬁrst introduce the notion of a perturbation space X in ( x nom ) of an original nominal input x nom .
This space contains all possible reductions, i.e. inputs where arbitrarily chosen tokens of the original nominal input x nom are deleted.
Note that this space grows exponentially in the length of the input.
We would like to verify whether or not there exists any reduced input x − with higher probability for the (nominal) prediction than x nom has.
More formally, this can be stated as a speciﬁcation:
[Equation]
where ˆ y is the (nominal) model prediction.
Determining precisely how prediction probabilities should change when input words are deleted is contentious and prone to inconsistencies: removing stop words for example may lead to little relevant change, while crucial information carrying words (e.g., ‘ not ’) can signiﬁcantly alter the meaning of the sentence in a task.
It is important to be cautious and not too restrictive in the speciﬁcation design, and certain that whatever is speciﬁed is desirable.
A speciﬁcation to at least not increase prediction probabilities under arbitrary input deletion is a conservative choice.
3 Other speciﬁcations are worth consideration, such as monotonically decreasing certainty as more input is deleted.
We will however see that even our very conservative choice of an under-sensitivity speciﬁcation is hard to positively verify for most inputs in the DAM model.
There are different approaches to establish if the Speciﬁcation (1) is satisﬁed.
With unlimited computational capacity, the property could exhaustively be evaluated for all x − ∈ X in ( x nom ) .
Statistically sampling from the reduction space can give an indication of under-sensitivity, but has a very limited coverage rate.
Search heuristics can try to identify violations (and be used for ‘adversarial’ training), but there is no guarantee that a stronger search procedure cannot ﬁnd more or worse violations.
IBP veriﬁcation on the other hand offers a formal guarantee across the whole space by establishing outer bounds for X in ( x nom ) and resulting bounds on output probabilities.
B ACKGROUND
We next give a brief introduction to the Decomposable Attention Model (DAM) which we will later verify.
The DAM architecture comprises commonly used neural NLP components, such as word embeddings, attention, and feed-forward networks.
Subsequently we introduce Interval Bound
Propagation and then bring these together to verify the behaviour of the DAM, i.e. efﬁciently assert whether an input satisﬁes Speciﬁcation (1).
Decomposable Attention.
The NLI task takes two word sequences as input – a premise and a hypothesis – and outputs a discrete entailment label prediction in { entailment , neutral , contradiction } .
The DAM architecture (Parikh et al., 2016) assumes the input word sequences to be embedded (e.g. as d -dimensional word vectors), i.e. operates on two sequences of input vectors:
A =
[ a 1 ; . .
a I ] ∈ R d × I , and B = [ b 1 ; . . .
; b J ] ∈ R d × J , where [ . ; . ] denotes concatenation, and I and J are sequence lengths.
Word vectors are individually transformed with a vector-valued function F ( . ) , and pairs thereof are then compared:
[Equation]
Note that we follow the notation of Parikh et al.
and that e ij is not related to a basis vector.
In the general model formulation F can be a linear transformation or MLP; this does not affect the derivations made here.
Adopting matrix notation across position pairs ( i, j ) , Equation (2) can instead be rewritten in matrix form as E = F ( A ) (cid:62) F ( B ) ∈
R I × J , which is used to compute two attention masks – one over each sequence – by normalising across i or across j :
[Equation]
These two attention masks serve as coefﬁcients in a convex combination over the original word vectors, aggregating each of the two sequences:
[Equation]
That is, A and B hold attention-aggregated word vectors from A and B at positions j = 1 , . .
J and i = 1 , .
, I , respectively.
These are joined with the original word representations, mixed using a position-wise feed-forward network G : R 2 d → R d (cid:48) and ﬁnally summed into a single vector representation for each sequence:
[Equation]
As a last step, a logit vector with entries for each class is computed as H ([ v 1 , v 2 ]) , where H :
R 2 d (cid:48)
→ R C is again a feed-forward network, and C is the number of output classes.
Interval Bound Propagation.
IBP is an incomplete but efﬁcient veriﬁcation method that can be used to verify input-output relationships.
It tracks how a part of the input space (in our case: the perturbation space X in ( x nom ) ) propagates forward through the network.
IBP starts with an axisaligned bounding box surrounding X in ( x nom ) , and uses interval arithmetic to obtain an axis-aligned bounding box for the output set.
Formally, let us assume that the neural network is deﬁned by a sequence of transformations h k for each of its K layers.
That is, for z 0 ∈ X in ( x nom )
[Equation]
The output z K ∈ R C has C logits corresponding to C classes.
IBP bounds the activation z k of each layer by an axis-aligned bounding box (i.e., z k ≤ z k ≤ z k ) using interval arithmetic.
We have for each coordinate z k,i of z k :
[Equation]
where e i is the standard i th basis vector.
Finally, at the last layer, an upper bound on the worst-case violation of the speciﬁcation can be evaluated quickly from the logit lower and upper bounds
z K and z K respectively, as the bounds translate directly into bounds for the softmax probabilities.
IBP can be performed in parallel while running the nominal forward pass.
However in general the output bounds are loose, which is exacerbated with increasing network depth.
Consequently IBP over-approximates the true extent of the image of X in ( x nom ) in output space, and can result in false negatives.
It is thus in practice important to keep the bounds as tight as possible.
IBP can be used at test time for veriﬁcation, but also for training, minimising loss terms derived from the logit bounds.
IBP has been used on MLPs and convolutional networks with monotonic activations (Gowal et al., 2018; Huang et al., 2019).
One technical contribution of this work is to apply it to a model with an attention component (Section 5).
V ERIFYING U NDER -S ENSITIVITY G UARANTEES FOR THE DAM M ODEL
To address under-sensitivity, we aim to verify Speciﬁcation (1) for the DAM model.
If the upper probability bound z K of the entire perturbation space X in ( x nom ) is smaller than the probability
P (ˆ y | x nom ) for the predicted class, then the speciﬁcation is veriﬁed.
That is ∀ z 0 ∈ X in ( x nom ) :
[Equation]
Using this inequality, we can assert whether Speciﬁcation (1) is veriﬁably satisﬁed for any given x nom , i.e. whether there exist any reduced samples with higher probability than x nom .
Overview.
We will ﬁrst describe the model behaviour when removing a single word at ﬁxed position, and then extend this to deleting single words at any position, and ﬁnally generalise this to arbitrary multi-token deletions.
One key difference to IBP bounds of other architectural components, such as CNNs or feed-forward layers, is the need for bounds on the attention normalisation, which has to take into account per-token upper and lower bounds.
We will exploit the fact that each vector of B is a convex combination of the J vectors that constitute B (and similarly for A ).
Hence, component-wise bounds on B can be obtained efﬁciently by maximising over those J vectors.
A and B are then inputs to a regular feed-forward network ( G followed by H ), for which IBP can be used.
Deleting Single Word: Particular Position.
We ﬁrst describe how model variables behave when an individual token at a ﬁxed position r is removed from one of the sequences.
Without loss of generality, we delete words from the second sequence, noting that the model has a symmetric architecture and that the same can be derived for the other input sequence.
We denote all resulting quantities with a bar (as in ¯ B ).
That is, when removing a single token at position r :
[Equation]
whereas ¯ A =
A .
Since F ( . ) is applied per-position in the sequence, the effect of word deletion remains isolated at this point; the matrix product ¯ E = F ( ¯ A ) (cid:62) F ( ¯ B ) ∈ R
I × ( J − 1) has identical entries as before, but the r -th column disappears.
Renormalising Attention Masks.
Likewise the attention mask ¯ P ( A ) has identical entries compared to P ( A ) , but the r -th column removed.
That is, for i = 1 , . .
, I and j = 1 , . . . , J s.t.
j (cid:54) =
r : [Equation]
[Equation]
The attention mask ¯ P ( B ) on the other hand has renormalised entries.
The values retain their relative order, yet the entries are larger because the r -th normalisation summand is removed.
For j (cid:54) =
r :
[Equation]
[Equation]
Hence we can compute ¯ P ( B ) ij in closed form as
[Equation]
To summarise the above: attention weights P ( B ) ij remain largely unchanged when deleting token r , but are rescaled to take into account missing normalisation mass.
In the next step, the model computes convex combinations ¯ A and ¯ B .
Concretely, ¯ A
= ¯ A · ¯ P ( A ) ∈
R d × ( J − 1) has unchanged elements compared to before (as A remains unchanged), but the r -th column is removed.
For ¯ B = ¯ B · (¯ P ( B ) ) (cid:62)
∈
R
d × I the dimensionality remains unchanged, but ¯ B has fewer elements and (¯ P ( B ) ) (cid:62) is renormalised accordingly.
Note that all this can still be computed in closed form using Equation (13), i.e. without need for IBP thus far, and these quantities can further be fed through the remaining network layers G and H to obtain concrete probabilities.
Deleting Single Word: Arbitrary Position.
We have reached the point where ¯ A , ¯ B , ¯ A and ¯ B are derived in closed form, for ﬁxed position r .
These can be computed exactly without approximation, for deleted words at any position r in the sequence.
Extending this to arbitrary single-word deletions, we take the elementwise minimum / maximum across all possible single word deletions, e.g.
[Equation]
which establishes upper and lower bounds for each element, and analogously for the other matrices
In the DAM architecture, these matrices are next fed into dense feed-forward layers
G (Equations (5) and (6)) and H , each with two layers.
5 We use IBP to propagate bounds through these layers, feeding in bounds on ¯ A , ¯ B , ¯ A and ¯ B as described above.
As a result, after propagating these bounds through G and H , we obtain bounds on output logits (and consequently on probabilities) for deletions of a single token at any position.
One further simpliﬁcation is possible: we compute ¯ v 2 directly from v 2 by subtracting the r -th summand for ﬁxed r (see Equation (6)).
Generalising this to arbitrary positions r , we can bound the subtracted vector with max r =1 ,...,J { G ([ ¯ A r ; ¯ b r ]) } and min r =1 ,...,J { G ([ ¯ A r ; ¯ b r ]) } , and thus directly obtain bounds for ¯ v 2 .
Deleting Several Words.
We have described the behaviour of intermediate representations (and bounds for them) under deletions of arbitrary individual words; the case of removing several words is similar.
The values of remaining individual word vectors a i and b j naturally remain unchanged.
The previously established bounds for single word deletions can be partly re-used to establish bounds for arbitrary multi-word deletions, see appendix A for more detail.
The resulting bounds for ¯ v 1 and ¯ v 2 are then input to a regular feed-forward network, for which IBP can be used.
E XPERIMENTS
We now evaluate to which extent the DAM model veriﬁably satisﬁes the Speciﬁcation (1) against under-sensitivity, and we will furthermore compare different training approaches.
Experiments are conducted on two large-scale NLI datasets:
SNLI (Bowman et al., 2015) and multiNLI (Williams et al., 2018), henceforth MNLI.
Whereas Feng et al.
(2018) addressed deletions of hypothesis words in SNLI, we establish the phenomenon also for MNLI, and for premise reductions.
In our experiments we use premise reductions, noting that under-sensitivity is also present for hypotheses (see
Fig. 2).
For SNLI we use standard dataset splits, tuning hyperparameters on the development set and reporting results for the test set.
For MNLI we split off 2000 samples from the development set for validation purposes and use the remaining samples as test set.
We use the same types of feed-forward components, layer size, dropout, and word embedding hyperparameters described by
Parikh et al.
Evaluation Metrics We evaluate with respect to the following metrics:
Accuracy: Standard test accuracy.
Veriﬁed Accuracy: This metric measures whether both i) the prediction is correct ii) it can be veriﬁed that no reduction with higher probability exists, using IBP veriﬁcation.
Beam Search Heuristic: This metric uses beam search to ﬁnd speciﬁcation violations in the perturbation space, following the protocol of Feng et al.
Search begins from the full sequence, gradually deleting words while keeping a beam of width 10.
This metric then measures whether both i) the search heuristic found no counterexample, and ii) the prediction is correct.
Note that this heuristic does not cover the full perturbation space,
i
e. does not sufﬁce to rule out counterexamples to the speciﬁcation.
This metric provides an upper bound for veriﬁed accuracy.
Training Methods We will compare the following training methods:
Standard Training: This provides a baseline for under-sensitivity behaviour under standard log-likelihood training.
Data Augmentation: A ﬁrst and comparatively simple way to address under-sensitivity is by adding training samples with random word subsets deleted, and penalising the model with a loss proportional to the speciﬁcation violation.
Adversarial Training: Here we use a more systematic approach than random word deletions: we search within the perturbation space for inputs with large differences between [Table] Heuristic [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Entropy Regularisation [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
Entropy Regularisation
[Table]
[Table]
nominal prediction probability and reduced probability, i.e. the strongest speciﬁcation violations.
We compare both i) random adversarial search that samples 512 randomly reduced perturbations and picks the strongest violation, and ii) beam search with width 10, following the protocol of Feng et al.
Both for data augmentation and adversarial training, altered samples are recomputed throughout training.
Entropy Regularisation:
Feng et al.
(2018) observed that entropy regularisation on prediction probabilities can partially mitigate the severity of under-sensitivity.
IBP-Training: Here we use IBP veriﬁcation as described in Section 5, which provides upper bounds on the prediction probability of arbitrarily reduced inputs (Eq.
We penalise the model with an auxiliary hinge loss on the difference between the upper probability bound for the gold label y and the nominal probability P ( y | x nom ) .
Note that the upper bound serves as a proxy for the adversarial objective, as it over-approximates the probabilities of arbitrary reduced samples, covering the full reduction space comprehensively.
Training Details
The training methods described above make use of an additive contribution to the training loss besides standard log-likelihood.
We tune the scale of the respective contribution in [0.01, 0.1, 1.0, 10.0, 100.0].
All experiments used a learning rate of 0.001, Adam optimiser, and batch size 128.
We perform early stopping with respect to veriﬁed accuracy, for a maximum of 3M training steps.
For veriﬁed training, we found it useful to continuously phase in the volume of the perturbation space to its maximum size, similar to Gowal et al.
Concretely, we compute the per-dimension center of upper and lower bound, and start linearly increasing its volume until it reaches the full perturbation space volume.
Similarly we phase in the perturbation radius, i.e. the maximum number of words deleted from 1 to the maximum sequence length of 48.
We tune phasein intervals in [10 0 , 10 3 , 10 4 , 10 5 , 10 6 ] training steps.
We also experimented with over-inﬂating the perturbation volume to larger than its real size at training time, as well as randomly sampling a maximum perturbation radius during training, neither of which improved veriﬁability results.
6. R ESULTS AND A NALYSIS
Evaluating the Effectiveness of IBP for Veriﬁcation.
Tables 1a and 1b show the main results.
For both datasets, a non-negligible portion of data points can be veriﬁed using IBP.
The gap between (standard) accuracy however is striking: only a small fraction of correctly predicted inputs is actually veriﬁably not under-sensitive.
Note that IBP accuracy is naturally bounded above by the beam search heuristic, which does however not cover the full reduction space, and overestimates veriﬁcation [Table] [Table]
rates. IBP veriﬁcation becomes particularly effective when adding the IBP-veriﬁability objective during training, verifying 18.36% and 17.44% of samples on SNLI and MNLI.
Veriﬁability does however come at a cost: test accuracy is generally decreased when tuning for veriﬁability, compared to Parikh et al.
This highlights a shortcoming of test accuracy as a metric: it does not reﬂect the under-sensitivity problem.
Once under-sensitivity is taken into account by dedicated training objectives or tuning for veriﬁcation rates, nominal accuracy suffers.
Computational Efﬁciency of IBP Veriﬁcation.
Table 2 gives a breakdown of the computational cost incurred for veriﬁcation, both empirically, and the theoretical worst-case number of forward passes required per sample.
IBP veriﬁcation comes with small computational overhead compared to a standard forward pass, which is incurred by propagating upper and lower interval bounds through the network once.
A full oracle is computationally infeasible, instead we used an exhaustive search oracle, but only up to a maximum budget of 200K forward passes per sample.
Even when stopping as soon as a single reduced sample is found that violates the speciﬁcation, the incurred time is orders of magnitude larger than veriﬁcation via IBP.
Comparing Training Methods.
We next discuss the differences between training methods, and how they reﬂect in veriﬁed model behaviour.
In absolute terms, standard training does not adhere to the under-sensitivity speciﬁcation well, neither on SNLI nor MNLI.
Data augmentation and random adversarial training lead to slightly different results on the two datasets, albeit without major improvements.
These methods have a strong random component in their choice of deletions, and this tends to lead to lower veriﬁcation rates on MNLI, where premises are on average 6.2 tokens longer, and the reduction space is correspondingly larger.
Beam search adversarial training leads to improved veriﬁcation rates on SNLI, yet not for MNLI, and it is noteworthy that when also trained with beam search adversarial samples, beam search evaluation improves substantially.
Entropy regularisation improves veriﬁed accuracy over standard training; this is in line with previous observations that it mitigates under-sensitivity behaviour, made by Feng et al.
Finally, the dedicated
IBP-Training objective substantially raises veriﬁcation rates compared to all other approaches.
In an ablation (Table 3) we evaluate performance on short sequences (up to 12 tokens) in the SNLI test set: here an exhaustive search over all possible reductions is feasible.
Still the absolute veriﬁcation rates are low in absolute terms, but we observe that shorter sequences are comparatively easier to verify, and that the incomplete IBP veriﬁcation can approach the veriﬁcation levels of the complete oracle (see rows 1,2,5, and 6).
For adversarial training (rows 3 and 4), however, oracle veriﬁcation rates are much closer to the Beam Search Heuristic.
This suggests that i) for short sequences the smaller perturbation space can be covered better by beam search, and ii) adversarial training can lead to high veriﬁability on short sequences, but it ﬁts a model in a way that results in loose IBP bounds.
D ISCUSSION
Veriﬁcation of a speciﬁcation offers a stronger form of robustness than robustness to adversarial samples.
Adversarial accuracy, as e.g. derived from beam search, might conceptually be easier to compute, yet has no guarantees to ﬁnd all or the strongest violations.
In fact, evaluating against weak adversaries under-estimates the extent of a problem (Uesato et al., 2018) and may lead to a false sense of conﬁdence.
IBP veriﬁcation can provide guarantees on the nonexistence of reduced inputs, but it is incomplete and can have false negatives.
Observations of comparatively low veriﬁcation or adversarial accuracy rates – as in this work – are not new, and have been found to be a general problem of datasets with high sample complexity (Schmidt et al., 2018).
We emphasise that under-sensitivity is a very challenging problem to address; even the relatively conservative speciﬁcation of non-increasing probability under deletion cannot be fulﬁlled for the majority of test samples under the baselines tested.
We see the veriﬁcation of the attention-based DAM model as a stepping stone towards the veriﬁcation of larger and more performant attention-based architectures, such as BERT.
Following the derivations here, token deletion bounds could similarly be propagated through BERT’s self-attention layer.
Towards this end, however, we see two main hurdles: i) BERT’s network depth, resulting in gradually looser IBP bounds ii) BERT’s word piece tokenisation, which requires special consideration in conjunction with token-level perturbations.
C ONCLUSION
We have investigated under-sensitivity to input text deletions in NLI and recast the problem as one of formally verifying a speciﬁcation on model behaviour.
We have described how Interval Bound
Propagation can be used in order to verify the popular Decomposable Attention Model, and have then compared several training methods in their ability to address and be veriﬁed against undersensitivity.
We observed that only a relatively small fraction of data points can be positively veriﬁed, but that IBP-training in particular is capable of improving veriﬁed accuracy.
A A PPENDIX : M ULTI -W ORD D ELETION B OUNDS
In this section we will elaborate on how bounds for ¯ v
and ¯ v 2 are computed in the case of arbitrary multi-word deletions.
Bounds on ¯ v 1 : Recall from Equation 5 that
[Equation]
We can compute bounds on ¯ v 1 under arbitrary multi-word deletions by bounding both ¯ a i and ¯ B i , and then propagating these bounds using IBP.
The values of a i remain unchanged under deletions in the second input sequence ( ¯ a i = a i ), and we will thus focus on deriving upper and lower bounds for ¯ B i .
To this end, recall that individual columns in B are computed as convex combination of vectors in
B .
That is, the i th column is computed as B i =
(cid:80) j b j P ( B ) i,j , where P ( B ) i,j is the entry of P ( B ) at position ( i, j ) .
The (elementwise) minimum and maximum values that ¯ B i can assume, are given by the (elementwise) minimum and maximum of single vectors
b j : b min = min j { b j } , and b max = max j { b j } .
The values of ¯ B i are hence bounded elementwisely from above by the values in b max : ¯ B i = (cid:88)
[Equation]
16) where D is an arbitrary set of indices of deleted tokens.
Note that whichever token set is deleted, the (renormalised) attention weights ¯ P ( ¯ B ) i,j always sum to 1.
The same follows analogously for b min as elementwise lower bound on ¯ B i .
Bounds on ¯ v 2 : Recall from Equation 6 that
[Equation]
where g j = G ([ A j ; b j ]) for notational convenience.
The function G is a dense feed-forward neural network with softplus nonlinearity; consequently all values in g j are strictly positive.
Since each g j has positive values, their sum will monotonically decrease if summands are removed, and monotonically increase as summands are added.
We consider two extreme cases of deleting word combinations: i) removing all but one word ii) removing precisely one word.
These will be used to bound ¯ v 2 for any other number of deleted words which lie in between these extremes.
For the case that all but one words are removed (at position r ), ¯ v 2 = g r , and the smallest values this expression can assume (elementwise) is min r =1 ,...,J { g r } .
This is thus a lower bound on ¯ v 2 for sequences with only one word, and, due to the monotonicity of ¯ v 2 in its number of summands, also for ¯ v 2 under any combination of deleted words.
For the case of deleting only a single word at position r , one single summand is subtracted from v 2 : ¯ v 2 = v 2 − g r , and this expression is bounded from above by v 2 − min r =1 ,...,J { g r } .
Again, due to the monotonicity of ¯ v 2 when deleting more symbols, this upper bound for single-word deletions is consequently also an upper bound for any combination of more deleted words.
To summarise, ¯ v 2 is bounded as follows:
[Equation]