S PARSE AND S TRUCTURED V ISUAL A TTENTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Visual attention mechanisms have been widely used in image captioning models.
In this paper, to better link the image structure with the generated text, we replace the traditional softmax attention mechanism by two alternative sparsity-promoting transformations: sparsemax and Total-Variation Sparse Attention (TV MAX ).
With sparsemax, we obtain sparse attention weights, selecting relevant features.
In order to promote sparsity and encourage fusing of the related adjacent spatial locations, we propose TV MAX .
By selecting relevant groups of features, the TVMAX transformation improves interpretability.
We present results in the Microsoft
COCO and Flickr30k datasets, obtaining gains in comparison to softmax.
TVMAX outperforms the other compared attention mechanisms in terms of humanrated caption quality and attention relevance.
I NTRODUCTION
The goal of image captioning is to generate a ﬂuent textual caption that describes a given image (Farhadi et al., 2010; Kulkarni et al., 2011; Vinyals et al., 2015; Xu et al., 2015).
Image captioning is a multimodal task: it combines text generation with the detection and identiﬁcation of objects in the image, along with their relations.
While neural encoder-decoder models have achieved impressive performance in many text generation tasks (Bahdanau et al., 2015; Vaswani et al., 2017; Chorowski et al., 2015; Chopra et al., 2016), it is appealing to design image captioning models where structural bias can be injected to improve their adequacy (preservation of the image’s information), therefore strengthening the link between their language and vision components.
State-of-the-art approaches for image captioning (Liu et al., 2018a;b; Anderson et al., 2018;
Lu et al., 2018) are based on encoder-decoders with visual attention.
These models pay attention either to the features generated by convolutional neural networks (CNNs) pretrained on image recognition datasets, or to detected bounding boxes.
In this paper, we focus on the former category: visual attention over features generated by a CNN.
Without explicit object detection, it is up to the attention mechanism to identify relevant image regions, in an unsupervised manner.
A key component of attention mechanisms is the transformation that maps scores into probabilities, with softmax being the standard choice (Bahdanau et al., 2015).
However, softmax is strictly dense , i.e., it devotes some attention probability mass to every region of the image.
Not only is this wasteful, it also leads to “lack of focus”: for complex images with many objects, this may lead to vague captions with substantial repetitions.
Figure 1 presents an example in which this is visible: in the caption generated using softmax (top), the model attends to the whole image at every time step, leading to a repetition of “bowl of fruit.”
This undesirable behaviour is eliminated by using our alternative solutions: sparsemax (middle) and the newly proposed TV MAX (bottom).
In this work, we introduce novel visual attention mechanisms by endowing them with a new capability: that of selecting only the relevant features of the image .
To this end, we ﬁrst propose replacing softmax with sparsemax (Martins & Astudillo, 2016).
While sparsemax has been previously used in NLP for attention mechanisms over words , it has never been applied to computer vision to attend over image regions .
With sparsemax, the attention weights obtained are sparse, leading to the selection (non-zero attention) of only a few relevant features.
Second, to further encourage the weights of related adjacent spatial locations to be the same (e.g., parts of an object), we introduce a new attention mechanism: Total-Variation Sparse Attention (which we dub TV MAX ), inspired by prior work in structured sparsity (Tibshirani et al., 2005; Bach et al., 2012).
With TV MAX , sparsity is allied to the ability of selecting compact regions.
According to our human evaluation experiments, this leads to better interpretability , since the model’s behaviour is better understood by looking at the selected image regions when a particular word is generated.
It also leads to a better selection of the relevant features, and consequently to the improvement of the generated captions.
This paper introduces three main contributions:
• We propose a novel visual attention mechanism using sparse attention, based on sparsemax (Martins & Astudillo, 2016), that improves the quality of the generated captions and increases interpretability.
• We introduce a new attention mechanism, TV MAX , that encourages sparse attention over contiguous 2D regions, giving the model the capability of selecting compact objects.
We show that TVmax can be evaluated by composing a proximal operator with a sparsemax projection, and we provide a closed-form expression for its Jacobian.
This leads to an efﬁcient implementation of its forward and backward pass.
• We perform an empirical and qualitative comparison of the various attention mechanisms considered.
We also carry out a human evaluation experiment, taking into account the generated captions as well as the perceived relevance of the selected regions.
S ELECTIVE V ISUAL A TTENTION
Attention mechanisms have the ability to select the relevant features, in this case spatial locations.
This requires a mapping from importance scores to a distribution, z ∈ R k (cid:55)→ p ∈ (cid:52) k , where (cid:52) k := (cid:110)
p ∈
R k | (cid:80) ki
=1 p i = 1 , p (cid:62) 0 (cid:111) denotes the simplex (the set of all probability distributions over k values).
The standard choice for this mapping is softmax, deﬁned as:
[Equation]
However, as softmax is strictly positive, its output is dense.
Thus, the model must pay some attention to the whole image and, consequently, assign lower attention weights to the relevant regions.
This motivates our proposed selective visual attention mechanisms, which, by being sparse, are able to better isolate the relevant image regions.
2. S PARSEMAX
To achieve selective capabilities, we propose the use of sparsemax (Martins & Astudillo, 2016), a sparse mapping consisting in the Euclidean projection of z onto the probability simplex:
[Equation]
which allows to obtain sparse outputs with a small increase in complexity.
Output sparsity is an attractive property for attention mechanisms, since some features do not provide relevant information for the current prediction.
In the image captioning case, using sparsemax allows focusing only on the spatial locations of the image that are relevant to the word being generated, assigning zero attention weight to all other regions.
2. S PARSE AND S TRUCTURED V ISUAL A TTENTION
To generate descriptive captions, the model should identify the objects present in the image.
Thus, when generating object-related words, the attention mechanism should assign high weights to the regions of the image containing the object.
However, sparsemax is unstructured and index-invariant, leading it to select discontinuous regions.
To overcome this, we propose a new visual attention mechanism, TV MAX .
TV MAX is a non-trivial generalization of fusedmax (Niculae & Blondel, 2017), a transformation based on fused lasso, to the 2D case.
To this end, we ﬁrst extend fusedmax even more generally, to arbitrary graphs.
2.2. G ENERALIZED F USED L ASSO
Let w ∈ R k , and let I = { 1 , . . .
, k } .
Consider a graph over I deﬁned by its edges E ⊆
I × I, where an edge between i and j means we want to encourage w i to be close to w j .
For simplicity we use i ∼ j as shorthand for ( i, j ) ∈ E .
The generalized fused lasso penalty (Tibshirani et al., 2005) is deﬁned as:
[Equation]
Minimizing Ω E encourages “fused” solutions, i.e., it encourages w i = w j for i ∼ j .
In particular, its proximal operator 1 can be seen as a fused signal approximator , seeking a vector w that approximates z well (in terms of Euclidean distance) and that is encouraged to be fused:
[Equation]
Computing the value of prox λ Ω E is non-trivial in general (Xin et al., 2016), but for certain edge conﬁgurations, described below, efﬁcient algorithms exist.
• If E forms a chain, i.e. i ∼ j ⇐⇒ i =
j − 1 , the problem is called 1D total variation and can be solved in O ( k ) time using the taut string algorithm (Davies & Kovac, 2001; Barbero & Sra, 2014).
We use the quasilinear algorithm of Condat (2013), which is very fast in practice.
• If the indices are aligned on a 2D grid, as in an image, and i ∼ j holds iff.
j is to the right or immediately below i , the problem is called 2D total variation .
Unlike the 1D case, exact algorithms are not available.
However, for an input of size a × b , it is possible to split the penalty into a column-wise and b row-wise 1D problems.
We may then apply a number of iterative methods, for instance proximal Dykstra (Barbero & Sra, 2014).
2.2. TV MAX
TV MAX combines 2D total variation (TV2D) regularization with sparsemax.
This way it promotes sparsity and encourages the attention weights of adjacent spatial locations to be the same, selecting contiguous regions of the image.
TV MAX is deﬁned as follows:
Deﬁnition 1 (TV MAX ) .
Let z ∈ R k , such that z ’s indices can be decomposed into rows and columns.
The TV MAX transformation is deﬁned as
[Equation]
where λ is an hyper-parameter controlling the amount of fusion ( λ = 0 recovers sparsemax) and Ω TV 2 D is a 2D total variation penalty.
Note that Eq. 5 differs from Eq. 4 in which the variable p is further constrained to lie in the probability simplex.
We show next how the forward and backward passes can be efﬁciently computed.
2.2. G ENERALIZED FUSED SPARSE ATTENTION
To construct generalized fused sparse attention , we follow Niculae & Blondel (2017) and deﬁne
[Equation] [Equation]
This can be seen as a constrained fused lasso approximator, because the solution p must be a probability distribution vector.
While the optimization function is very similar to Eq.
4, the additional constraint that p ∈ (cid:52) increases complexity.
Fortunately, the following result holds:
Proposition 1 (Computing generalized fusedmax) .
[Equation]
The proof is given in Appendix A.2.
Proposition 1 also provides a shortcut for deriving the Jacobian of generalized fusedmax via the chain rule : denoting by J F the Jacobian of prox λ Ω E , we have
[Equation]
As we already know how to compute J sparsemax (Appendix A.1), we may concentrate our effort on deriving the simpler J F (Eq. 9).
Proposition 2 (Group-wise characterization of prox λ Ω E ) .
Let w (cid:63) :=
prox λ Ω E , and denote by G i the set of indices fused to w i in the solution, G i may be deﬁned recursively:
i ∈
G i for all i , and 2.
j ∈ G i if there exists m ∈ G i such that m ∼ j and w (cid:63)m = w (cid:63)j .
Deﬁne s ij = sign ( w (cid:63)i − w (cid:63)j ) .
Then, the solution has the expression
[Equation]
Proposition 2 shows how to easily compute a generalized Jacobian of gfusedmax : since small perturbations in z never change the groups G i nor the signs of across-group differences s ij , differentiating
Eq.
8 yields
[Equation]
This generalizes Lemma 1 of Niculae & Blondel (2017) to generalized fused lasso, with a simpler proof, given in Appendix A.3.
2.2. C OMPUTATION
As we show in Proposition 1, computing TV MAX ’s forward pass can be done by chaining efﬁcient algorithms for TV2D and sparsemax.
From Eq.
7 we have that TV MAX ’s Jacobian can be computed as J TV MAX =
J sp ( prox λ Ω TV 2 D ( z ))
J tv ( z ) , where J sp is the sparsemax’s Jacobian and J tv is the Jacobian of the Total Variation proximal operator.
As derived in Proposition 2, ( J tv )
i,j = 1 / n ij if i and j are fused in a group with n ij elements, and 0 otherwise.
The backward pass intuitively involves ”spreading” the credit assigned to one image location evenly across all locations fused with it.
This can be implemented by Algorithm 1 in O ( k + N g log k ) where
N g is the number of groups of fused positions. In the worst case, when there are no positions fused, the complexity is O ( k + k log k ).
This algorithm is inspired by ﬂood ﬁlling algorithms (Burtsev & Kuzmin, 1993).
1 Input: p = TV MAX ( z ) , d p ∈ R k . 2 Output: d z = J (cid:62) TV MAX (
d p ) ∈ R k # chain rule 3 Initialize: N ← ∅ # neighbours stack
V ← ∅ # visited positions
G ← ∅ # current group s
# intermediate value used for J TV MAX ’s computation 7 d w d p
# Eqs.
14 and 15 of § A.
# check if all positions have been visited pick stack while do if
# check if element [Table] fused [Table]
neighbour
[Table] # sum [Table] d
[Table]
[Table]
[Table] [Table]  I MAGE C APTIONING M ODEL To compare the proposed attention mechanisms, we use a straight-forward simple encoder-decoder model with visual attention, inspired by Liu et al.
(2018a).
The model is sketched in Figure 2.
Given an image, we use a residual CNN pretrained on ImageNet (He et al., 2016; Russakovsky et al., 2014) to get a feature map with spatial dimension of size 8 × 8 and channel dimension of size 2048, that go through a ﬁne-tuned feedforward layer yielding g = 512 feature maps.
The visual feature matrix V =
[ v 1 , v 2 , . .
, v k ] , with v i ∈ R g and k = 64 = 8 × 8 , contains the image information used to generate the corresponding caption.
Following Liu et al.
(2018a), we use input and output attention to select the relevant features for the current generation.
To generate the word at position t , the input attention , α t , is computed using the LSTM’s previous hidden state, h t − 1 ∈ R d .
First, a similarity score z t,i , i ∈ { 1 , . . .
, k } , is computed between h t − 1 and the i th image cell via a feedforward transformation (Bahdanau et al., 2015), as z t,i = w (cid:62) tanh ( aﬃne ([ v i ; h t − 1 ])) , for all k image cells.
Then, α t is obtained by normalizing the k -dimensional vector of scores z t with softmax, α t = softmax ( z t ) .
Using these attention weights, a vector representation of the image to be used as input of the LSTM, is obtained, s t =
V α t .
The output attention (cid:101) α t , is computed in the same way as above, but applied to the current LSTM hidden state h t , instead of h t − 1 , and normalized with the different proposed transformations.
This produces output visual features (cid:101) s t
= V (cid:101) α t , which are passed through a feedforward layer to yield the image representation r t = tanh ( aﬃne ( (cid:101) s t )) .
Finally, the predictive probability of the next word is:
[Equation]
[Equation]
softmax
[Equation] [Figure]
E XPERIMENTS
Settings.
The input images are resized to 256 × 256 before going through the residual CNN and the feature maps obtained have a size of 8 × 8 .
We use an LSTM hidden size of d = 512 and a word embedding size of 256 , for all models.
The models were trained for 50 epochs using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0 . 0001 and a decay of 0 . 8 and 0 .
999 for the ﬁrst and second momentum, respectively.
After the 10 th epoch, the learning rate starts decaying with a decay factor of 0 . 99 .
For TV MAX , we set λ = 0 .
Datasets and Metrics.
We report our results on the Microsoft COCO (MSCOCO) and Flickr30k datasets.
MSCOCO is composed of 113,287 images of common objects in context while Flickr30k consists in 31,000 pictures of people involved in everyday activities and events.
Each image is annotated with 5 captions.
We use the split proposed by Karpathy & Fei-Fei (2015), which stipulates equal validation and test sizes of 5,000 images (MSCOCO) and 1,000 (Flickr30k).
The metrics we report are SPICE (Anderson et al., 2016), CIDEr (Vedantam et al., 2015), longest common subsequence ROUGE, (denoted ROUGE L ; Lin, 2004), 1– to 4–gram BLEU (denoted BLEU 4 ; Papineni et al., 2002), and METEOR (Banerjee & Lavie, 2005).
To investigate whether selective attention alleviates repetition, we also measure the n-gram repetition metric REP (Malaviya et al., 2018).
[Table]
Automated metrics.
As can be seen in table 1, overall sparsemax and TV MAX attention mechanisms achieve better results when compared with softmax, indicating that the use of selective attention leads to better captions.
This improvement does not come at a high computational cost: at inference time, models using TV MAX and sparsemax are only 1.3x and 1.1x slower than softmax.
Moreover, for TV MAX , automatic metrics results are slightly worse than sparsemax but still superior to softmax on MSCOCO and similar on Flickr30k.
We show next that this is compensated with fewer repetitions and higher scores in the human evaluation of the captions and attention relevance.
[Table]
Human rating.
The caption evaluation consisted in attributing a score from 1 to 5 to the caption of each model while the attention evaluation consisted in scoring the relevancy of the attended areas, from 1 to 5, when generating the non stop words of the captions.
A full description of the human assessment can be found in Appendix B.
Despite performing slightly worse than sparsemax under automated metrics, TV MAX outperforms sparsemax and softmax in the caption human evaluation and the attention relevance human evaluation, reported in Table 2.
The superior score on attention relevance shows that TV MAX is better at selecting the relevant features and its output is more interpretable.
Additionally, the better caption evaluation results demonstrate that the ability to select compact regions induces the generation of better captions.
We next explore possible explanations for the TV MAX superior results.
Repetition.
Figure 1 illustrates that softmax attention is prone to spuriously repeating references to the same object.
Selective attention mechanisms like sparsemax and especially TV MAX reduce repetition, as measured by the REP metric reported in Table 1.
This expected success can be attributed to the sparsity of the attention weights distribution and to the ability to select compact regions exclusively and can be one of the causes of the human evaluation results.
This happens even though TV MAX generates longer sentences than sparsemax and softmax (9.5 against 9.0 words on average) and shows the beneﬁt of promoting structured and sparse attention simultaneously.
To corroborate our intuition that sparsity leads to less repetition, we measured the Jensen-Shannon divergence (JS) between the attention distributions for each step of the generation of the captions correspondent to the images of the MSCOCO test set.
The mean JS values are 0.12, 0.29, and 0.34 for softmax, sparsemax, and TVmax, respectively.
This shows that sparsity leads to less similar attention distributions along the generation of the captions and, consequently, to less repetitions.
Object detection.
Using the MSCOCO object detection ground truth, we compared the percentage of objects present in the image that are referred to in the captions, using each attention mechanism.
With TV MAX 28.2% of the reference objects are referred, against 27.5% and 27.4% for sparsemax and softmax, repectively.
This shows that promoting high attention to groups of spatial locations of the image leads to a more precise identiﬁcation of the objects.
Sparsity.
The average image area that receives zero attention is 34 % for sparsemax and 25 % for
TV MAX .
To illustrate where the models attend to, we display the output attention in Figures 1 and 3.
As expected, softmax weights are spread widely across the image, ending up missing the relevant regions.
In contrast, sparsemax and TV MAX weights are zero for the non-relevant spatial locations.
Qualitative comparison.
As the image of Figure 1 contains various similar objects, the softmax model (top) generates a incoherent, repetition-laden caption.
In contrast, the sparsemax (middle) and TV MAX (bottom) models better identify the relevant parts of the image, generating coherent and descriptive captions.
Moreover, the groups obtained with TV MAX are clearly visible and more aligned to object boundaries, offering better interpretability, as revealed by human attention assessment.
In Figure 3 it can also be noticed that with TV MAX (bottom) the model correctly identiﬁed “a group of people” instead of “a soccer player” as with sparsemax (middle) and softmax (top).
This indicates its superior ability to correctly deﬁne the relevant groups of features and that this ability leads to improved captions.
R ELATED W
ORK
Image captioning.
In the last years, neural models with visual attention mechanisms have been receiving increased interest.
Several researchers have been studying diverse attention mechanisms in order to reﬁne visual information for image captioning.
Xu et al.
(2015) proposed the use of hard attention, which only attends to one region at each step.
However, to generate descriptive captions the model should, often, focus on more than one region.
In addition, hard attention is non-differentiable, requiring imitation learning or Monte Carlo policy gradient approximations.
Anderson et al.
(2018) proposed bottom-up attention, using an object detection model designed to identify bounding boxes of objects, and top-down attention, selecting the relevant bounding-boxes.
Wang et al.
(2019) proposed an hierarchical attention network composed by a patch detector, object detector, and concept detector.
Using object detection models is less demanding on the attention mechanism, since it only has to select the boxes the model should attend to.
However, such models are limited by the bounding boxes position’s accuracy.
Gao et al.
(2019) introduced a deliberate attention network to reﬁne the attended visual features.
Yet, the attention distribution remained dense.
Sparse attention.
In several tasks only a few features are relevant for the current prediction.
This can be attained when using sparse attention.
Various prior works have proposed sparse attention mechanisms with promising results, (Xu et al., 2015; Martins & Astudillo, 2016; Malaviya et al., 2018; Peters et al., 2019).
Niculae & Blondel (2017) proposed 1D fusedmax, which incorporates the fused lasso, so that adjacent words are encouraged to have the same attention weight.
In this work, the authors were able to improve interpretability without sacriﬁcing performance, obtaining superior results on textual entailment and summarization.
We derive a generalized fused attention mechanism, extending 1D fusedmax.
C ONCLUSIONS AND F UTURE W ORK
We propose using sparse and structured visual attention, in order to improve the process of selecting the features relevant to the caption generation.
For that, we used sparsemax and introduced TV MAX .
Results on the image captioning task, show that the attention mechanism is able to select better features when using sparsemax or TV MAX .
Furthermore, in the human assessment and attention analysis we see that the improved selection of the relevant features as well as the ability to group spatial features lead to the generation of better captions, while improving the model’s interpretability.
In future work, TV MAX attention can be applied to other multimodal problems such as visual question answering.
It can also be applied in other tasks for which we have prior knowledge of the data’s stucture, for instance graphs or trees.
A F ORWARD AND BACKWARD PASS OF 2D FUSEDMAX ATTENTION . A. P RELIMINARIES
The proximal operator of a function f : R d → R ∪ {∞} is deﬁned as
[Equation]
11) and it is guaranteed to have a unique solution, thanks to the strong convexity of the Euclidean distance.
The indicator function of a set C ⊂ R
d is the function
[Equation] [Equation]
The projection onto a convex set C ⊂ R d is deﬁned as
[Equation]
showing that the proximal operator can be seen as a generalization of projection.
The sparsemax attention mapping (Martins & Astudillo, 2016) is the projection onto the simplex, sparsemax
[Equation]
A necessary component for using sparsemax for attention is its Jacobian, the matrix of its partial derivatives ( J sparsemax )
i,j = ∂ sparsemax ( z )
[Equation]
∂z j . Martins & Astudillo (2016) derive its expression
[Equation]
(15) where s j = 1 if sparsemax ( z ) j > 0 and s j = 0 otherwise.
A. P ROOF OF P ROPOSITION
Proof.
This result is a slight extension of Proposition 2 in Niculae & Blondel (2017), and also follows from Corrolary 4 of Yu (2013), by taking f = ι (cid:52) , and noting that ι (cid:52) is symmetric: if p ∈ (cid:52) , then any vector p (cid:48) obtained by permuting p is also in (cid:52) , because its values remain nonnegative and sum to 1.
A. P ROOF OF P ROPOSITION
Let w (cid:63) :=
prox λ Ω E , and denote by G i the set of indices fused to w i in the solution.
Deﬁne s ij = sign ( w (cid:63)i − w (cid:63)j ) .
Proof.
The subgradient optimality conditions of Eq.
4 are: (Friedman et al., 2007)
[Equation]
where t ij = sign ( w (cid:63)i − w (cid:63)j ) if w (cid:63)i (cid:54) = w (cid:63)j , otherwise t ij is a free variable in [ − 1 , 1] .
We focus on a single group G
= G i , dropping the index i for brevity.
Within a fused group, the solution is constant, i.e., w (cid:63)j = w for j ∈ G .
We separate the sums in Eq.
according to whether k ∈ G or not, and move the “constant” terms to the right hand side, yielding the system
[Equation]
Summing up the Eq.
17 over all j ∈ G , we observe that for any k ∈ G , the term λt jk appears twice with opposite signs.
Thus,
[Equation]
Dividing by | G | gives exactly Eq. 8.
This reasoning applies to any group G i .
B H UMAN EVALUATION DESCRIPTION
To perform the human evaluation ﬁrstly 100 images were randomly selected from the test set of the MSCOCO dataset (using the split proposed by Karpathy & Fei-Fei (2015)).
For each of the selected images, the human evaluators selected a score from 1 to 5 for the captions generated by the models using softmax attention, sparsemax attention, and TV MAX attention.
They were also asked to evaluate whether the models attend to the relevant regions of the image when generating a certain word.
For that they observed the attention plots corresponding to the non stop words of the caption of each of the models.
While in Figures 1 and 3, 4, and 5 we emphasized sparsity with a hard white mask, for the human evaluation the sparse regions of the attention plots were simply fully transparent, to avoid biasing the evaluators.
The possible scores were also between 1 and 5.
The 100 images were judged by 6 persons both for the captions evaluation and attention evaluation.
The order of the captions and attention plots was randomly chosen for each image.
With these scores, we computed the mean of the captions evaluation scores and the mean of the attention relevance evaluation scores.
The results are reported in Table 2.
C A DDITIONAL A TTENTION V ISUALIZATION