I MPROVED A DVERSARIAL I MAGE C APTIONING
A BSTRACT
In this paper we study image captioning as a conditional GAN training, proposing both a context-aware LSTM captioner and co-attentive discriminator, which enforces semantic alignment between images and captions.
We investigate the viability of two discrete GAN training methods: Self-critical Sequence Training (SCST) and Gumbel Straight-Through (ST) and demonstrate that SCST shows more stable gradient behavior and improved results over Gumbel ST
I NTRODUCTION
Signiﬁcant progress has been made on the task of generating image descriptions using neural image captioning.
Early systems were traditionally trained using cross-entropy (CE) loss minimization (Karpathy & Li, 2015; Xu et al., 2015).
Later, reinforcement learning techniques (Ranzato et al., 2015; Liu et al., 2017) based on policy gradient methods were introduced to directly optimize metrics such as CIDEr or SPICE Anderson et al.
Along a similar idea, Rennie et al.
(2017) introduced Self-critical Sequence Training (SCST), a light-weight variant of REINFORCE, which produced state of the art image captioning results using CIDEr as an optimization metric.
To address the problem of sentence diversity and naturalness, image captioning has been explored in the framework of GANs.
However, due to the discrete nature of text generation, GAN training remains challenging and has been generally tackled either with reinforcement learning techniques (Hjelm et al., 2017; Rajeswar et al., 2017; Dai et al., 2017) or by using Gumbel softmax relaxation (Jang et al., 2016), as in (Shetty et al., 2017; Kusner & Hern´andez-Lobato, 2016).
Despite impressive advances, image captioning is far from being a solved task.
It remains a challenge to satisfactorily bridge the semantic gap between image and captions to produce diverse, creative, and “human-like” captions.
Although applying GANs to image captioning for promoting human-like captions is a very promising direction, the discrete nature of the text generation process makes it challenging to train such systems.
The recent work of Caccia et al.
(2018) showed that the task of text generation for current discrete GAN models is difﬁcult, often producing unsatisfactory results, and requires therefore new approaches and methods.
In this paper, we propose a novel GAN-based framework for image captioning that enables better language composition, more accurate compositional alignment of image and text, and light-weight efﬁcient training of discrete sequence GAN based on SCST.
A DVERSARIAL C APTION G ENERATION
In this Section we present our novel captioner and discriminator models.
We employ SCST for discrete GAN optimization and compare it to the approach based on the Gumbel relaxation trick.
Context Aware Captioner G θ .
For caption generation we use an LSTM with visual attention (Xu et al., 2015; Rennie et al., 2017) together with a visual sentinel Lu et al.
(2017) to give the LSTM a choice to attend to visual or textual cues.
While Lu et al.
(2017) feeds at each step t only an average image feature, we feed a mixture of image and visual sentinel features from t − 1 to make the LSTM aware of the last attentional context (called Context Aware attention), as seen in Fig.
This simple modiﬁcation gives signiﬁcant gains, as the captioner is now aware of the visual information used in the past.
As reported in Tab. 1, a captioner with an adaptive visual sentinel Lu et al.
(2017) gives 99.7 CIDEr versus 103.3 for our Context Aware Attention on COCO validation set.
[Table]
[Table]
w ∗ t −ˆ c t −
WE
Linear
LSTM h t −h t
Att
I
MLP w ∗ t x t y t s t ˆ c t p ( w t | h t , ˆ c t )
Co-attention Pooling Discriminator D η .
Previous works jointly embed the modalities at the similarity computation level, referred to as Joint-Emb (e.g., Dai et al.
Instead, we propose to jointly embed image and caption in earlier stages using a co-attention model Lu et al.
and compute similarity on the attentive pooled representation.
We call it a Co-attention discriminator, see Fig.
In Section 3 we compare D η with Joint-Emb of (Dai et al., 2017; Shetty et al., 2017), where E I is the average spatial pooling of CNN features and E S the last state of LSTM.
[Figure]
i H i
P α
i
I i
Linear Linear
E S
E I
E TI E S
Sigmoid fake/real
[Equation]
[Figure]
Training D η .
Our discriminator D η is not only trained to distinguish real captions from fake (generated), but also to detect when images are coupled with random unrelated real sentences, thus forcing it to check sentence composition and semantic relationship between image and caption.
We solve the following optimization problem: max η L D ( η ) , where the loss L D ( η ) is E
[Equation]
where w is the real sentence, w s is sampled from generator G θ (fake caption), and w (cid:48) is real but random caption.
Training G θ .
The generator is optimized to solve max θ L G ( θ ) , where L G ( θ ) =
E I log D η
( I, G θ ( I )) .
The main difﬁculty is the discrete, non-differentiables nature of the problem.
We propose to solve this issue by adopting SCST Rennie et al.
and compare it to the
Gumbel relaxation approach of Jang et al.
Training G θ using SCST .
SCST is a REINFORCE variant that uses the reward under the decoding algorithm as baseline.
In this work, the decoding algorithm is a “greedy max”, selecting at each step the most probable word from arg max p θ ( .
| h t ) .
For a given image, a single sample w s of the generator is used to estimate the full sequence reward, L IG ( θ ) =
log( D ( I, w s )) where w s ∼ p θ ( .
| I ) .
Using SCST, the gradient is estimated as follows:
[Equation]
where ˆ w is obtained using greedy max (see Fig. 3).
Note that the baseline does not change the expectation of the gradient but reduces the variance of the estimate.
Also, observe that the GAN training can be regularized with any NLP metric r NLP (such as CIDEr) to enforce closeness of the generated captions to the provided ground truth on the n -gram level; the gradient then becomes:
[Equation]
There are two main advantages of SCST over other policy gradient methods used in the sequential
GAN context: 1) The reward in SCST can be global at the sentence level and the training still succeeds.
In other policy gradient methods, e.g., Dai et al.
; Liu et al.
(2017), the reward needs to be deﬁned at each word generation with the full sentence sampling, so that the discriminator needs to be evaluated T times (sentence length).
In (Dai et al., 2017; Liu et al., 2017; Hjelm et al., 2017), many Monte-Carlo rollouts are needed to reduce variance of gradients, requiring many forward-passes through the generator.
In contrast, due to a strong baseline, only a single sample estimate is enough in SCST.
Training G θ using the Gumbel Trick.
An alternative way to deal with the discreteness of the generator is by using Gumbel re-parameterization Jang et al.
Deﬁne the soft samples y jt , for t = 1 , . . .
T (sentence length) and j = 1 , . . .
K (vocabulary size) such that: y jt =
Softmax
[Equation]
+ [Equation] ) (cid:1) , where g j are samples from Gumbel distribution, τ is a temperature parameter.
We experiment with Gumbel Soft and Gumbel Straight-Through (Gumbel ST) approach, recently used in (Shetty et al., 2017; Kusner & Hern ´ andez-Lobato, 2016).
For Gumbel soft , we use the soft samples y t as LSTM input w st +1 at the next time step and in D η , i.e., ∇ θ L IG ( θ ) =
∇ θ log( D η
( I, y 1: T )) .
For Gumbel ST , we deﬁne one-hot encodings
O t =
OneHot(arg max j y jt )
and approximate the gradients
∂ O jt /∂y j (cid:48) t = δ jj (cid:48) .
To sample from
G θ we use the hard O t as LSTM input w st +1 at the next time step and in D η , hence the gradient becomes ∇ θ L IG ( θ ) =
∇ θ log( D η
( I, O 1: T )) Observe that this loss can be additionally regularized with Feature Matching (FM):
[Equation]
where ( w ∗ 1: T ) is the ground truth caption corresponding to image I , and E I and E S are co-attention image and sentence embeddings (as deﬁned earlier).
Feature matching enables us to incorporate more granular information from discriminator representations of the ground truth caption, similar to how SCST reward can be regularized with CIDEr.
E XPERIMENTS
Experimental Setup.
We evaluate our proposed method and the baselines on COCO dataset Lin et al.
Each image is encoded by a resnet-101 He et al.
(2016), followed by a spatial adaptive max-pooling to ensure a ﬁxed size of 14 × 14 × 2048.
An attention mask is produced over the 14 ×spatial locations, resulting in a spatially averaged 2048-dimension representation.
LSTM hidden state, image, word, and attention embedding dimensions are ﬁxed to 512 for all models.
Before the
GAN training, all the models are ﬁrst pretrained with the cross entropy (CE) loss.
Experimental Results.
Tab.
2 presents results on COCO dataset for context-aware captioner, two discriminator architectures (ours Co-att, and baseline Joint-Emb) and all training algorithms (SCST,
Gumbel ST, and Gumbel Soft).
For reference, we also include results for CE (trained only with cross entropy) and CIDEr-RL (pretrained with CE, followed by SCST to optimize CIDEr), as well as results from non-attentional models.
As expected, CIDEr-RL greatly improves the language metrics as compared to CE model (101.6 to 116.1 CIDEr), leading to a signiﬁcant drop in the vocabulary coverage (from 9.2% to 5.1%).
On the other hand, the underperformance of GANs over CIDEr-RL in terms of CIDEr is also expected since GAN’s objective is to make the sentences more descriptive and human-like, deviating from the vanilla ground truth captions.
The results also show the advantage of our Co-att architecture as compared to the Joint-Emb one, showing the importance of the early joint embedding of the image/caption pair for better similarity computation. Regularizing GANs with CIDEr additionally
[Table]
[Table]
improves the language metrics but sacriﬁces sentence diversity by reducing vocabulary coverage.
Finally, note that the non-attentional models are behind in all metrics, except for vocabulary coverage.
Interestingly, Co-att discriminators still provide better semantic scores than Joint-Emb despite non-attentional generators.
D i s c
r
i m i n a t
o r S
c
o r e
[Figure] methods.
SCST vs.
Gumbel Our experiments also showed that SCST is a more stable approach for training discrete GAN nodels, achieving better results as compared to Gumbel relaxation approaches.
[Figure]
To demonstrate that our experiments fairly compared both approaches, in Fig.
4 we show training of different Gumbel methods, where we plot the discriminator scores across gradient updates.
As can be seen, at the end of training the generated sentences are scored around 0.5, random near 0.1 and real sentences above 0.7, indicating a properly trained discriminator and a healthy execution of all the Gumbel methods.
Fig.
5 also compares gradient behaviors during training for SCST and Gumbel, showing that SCST gradients have smaller average norm and variance across minibatches, conﬁrming our conclusion.
C ONCLUSION
In summary, we demonstrated that SCST training for discrete GAN is a promissing new approach that outperforms the Gumbel relaxation in terms of training stability and the overall performance.
Moreover, we showed that our context-aware attention gives larger gains as compared to the adaptive sentinel or the traditional visual attention.
Finally, our co-attention model for discriminator compares favorably against the joint embedding architecture.