L EARNING C OMPACT R EWARD FOR I MAGE C APTIONING
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial learning has shown its advances in generating natural and diverse descriptions in image captioning.
However, the learned reward of existing adversarial methods is vague and ill-deﬁned due to the reward ambiguity problem.
In this paper, we propose a reﬁned Adversarial Inverse Reinforcement Learning (rAIRL) method to handle the reward ambiguity problem by disentangling reward for each word in a sentence, as well as achieve stable adversarial training by reﬁning the loss function to shift the stationary point towards Nash equilibrium.
In addition, we introduce a conditional term in the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions.
Our experiments on MS
COCO show that our method can learn compact reward for image captioning.
I NTRODUCTION
Image captioning is a task of generating descriptions of a given image in natural language.
In a general encoder-decoder structure (Vinyals et al., 2015), image features are encoded in a CNN and decoded into a caption in a word by word manner.
Based on the loss function, standard approaches to the problem could be divided into three categories: MLE (Maximum Likelihood Estimation), RL (Reinforcement Learning) and GAN (Generative Adversarial Network).
Early proposed methods were based on MLE function and made improvements by designing speciﬁc model structure (Xu et al., 2015).
MLE adopts the cross-entropy loss and learns a one-hot distribution for each word in the sentence.
By maximizing the probability of the ground truth word whilst suppressing other reasonable vocabularies, the probability distribution learned by MLE tends to be sparse and the generated captions have limited diversity (Dai et al., 2017).
On the other hand, RL has advantages in boosting the model performance by optimizing the handcrafted metrics (Rennie et al., 2017; Liu et al., 2017; Chen et al., 2019).
However, due to the reward hacking problem,
RL maximizes the reward in an unintended way and fails to produce human-like descriptions (Li et al., 2019a).
Considering naturalness and diversity of the generated captions, GAN has raised attention in image captioning for its capability of producing descriptions that are indistinguishable from human-written ones (Dai et al., 2017;
Shetty et al., 2017; Chen et al., 2019; Dognin et al., 2019).
In image captioning, the generator of GAN learns true data distribution by maximizing the reward function learned from a discriminator, and the discriminator distinguishes the generated sample from the true data.
The adversarial training converges to an equilibrium point (i.e., Nash equilibrium) at which both the generator and discriminator cannot improve (Goodfellow et al., 2014).
As shown in
Figure 1, the learned distribution of GAN is closer to the ground truth distribution than that of other methods (i.e., MLE and RL) on different splits.
However, previous work of adversarial networks in image captioning gives one reward function D for a complete sentence consisting of n words.
This strategy causes the reward ambiguity problem (Ng et al., 1999) since which word(s) causes the reward to increase or decrease is not accounted for, and thus there are many optimal policies that determine the sentence can explain one reward.
An example is that each image in MS COCO has ﬁve ground truth captions.
Although these captions may vary in formats, each caption has the same reward value in GAN.
From the perspective on the system level, learning sentence-level reward from different image-caption pairs is analogous to learning reward of a trajectory from different system dynamics, which makes the discriminator unable to distinguish the true reward functions from those shaped by the environment dynamics (Fu et al., 2018).
[Figure]
Facing the challenge, we adopt AIRL (Fu et al., 2018) to solve the reward ambiguity problem by disentangling reward for each action (i.e., word in a sentence) from different image-caption pairs and learning a compact reward function.
compact means words with similar semantics, such as children and kids , correspond to close reward values.
Driven by the compact reward function of the discriminator, the generator learns the optimal policy and thus produces qualitative descriptions.
However, there are still two major problems to address: 1) AIRL is difﬁcult to converge to Nash equilibrium using policy gradient (See Section 4.2 for details); 2) AIRL is designed without mode control, and thus the outputs have limited diversity, which is a commonly encountered issue called mode collapse (Mirza & Osindero, 2014).
In this paper, we propose a reﬁned AIRL method to learn a compact reward function for each word, as well as achieve stable adversarial training by reﬁning the loss function to shift the stationary point towards Nash equilibrium.
The reﬁned method makes it possible to reach the equilibrium point for a non-concave model function of the generator.
In addition, a conditional term is introduced in the loss function to mitigate mode collapse and to increase the diversity of the generated descriptions.
Both the caption evaluator (i.e., discriminator) (Cui et al., 2018; Sharif et al., 2018) and the generator are cast into this uniﬁed framework, where the discriminator evaluates captions using a learned compact reward function, and the generator produces qualitative image descriptions.
We demonstrate the effectiveness of our method in the experiments.
R ELATED W ORK
Image Captioning.
The development of image captioning can be summarized into two directions: model structure design (Lu et al., 2017; Yao et al., 2018) and loss function construction (Rennie et al., 2017;
Ren et al., 2017).
In the methods based on model structure design, attention mechanism and the fusion of visual and semantic information are the key focus.
Lu et al.
(2017; 2018) proposed a sentinel gate to learn adaptive attention between visual content and non-visual text.
Yao et al.
(2018) explored the role of visual relationship in image captioning.
On the other hand, methods based on loss function construction focus on optimization of the loss function.
Rennie et al.
(2017) optimized on non-differentiable evaluation metric using policy gradient, and improved scores of these metrics on various models.
Ren et al.
(2017) designed an embedding reward under actor-critic reinforcement learning.
Similarly, we address the construction of loss functions, and thus our algorithm can be built on existing model structures.
See Appendix F for a short discussion about different loss functions.
Adversarial Methods for Image Captioning.
Adversarial methods are known for producing plausible samples by training the generator and the discriminator in an adversarial manner (Goodfellow et al., 2014).
In image captioning, the discriminator is formed as a binary classiﬁer that distinguishes the generated sentence from the ground truth, while the generator produce captions that can fool the discriminator.
Conditional GAN was proposed in (Dai et al., 2017) to improve the naturalness and diversity of generated captions.
CNN and RNN based discriminators were introduced in (Chen et al., 2019).
However, existing methods estimate a reward function for the complete sentence consisting of n words, where multiple optimal policies that determine the sentences can correspond to one reward
(Ng et al., 1999).
Thus the learned reward is ambiguous and ill-deﬁned.
We solve this problem by recovering a compact reward function for each word in the sentence under a reﬁned AIRL framework.
Although AIRL has been utilized to solve problems in other ﬁelds (Wang et al., 2018; Li et al., 2019b; Shi et al., 2018), we are the ﬁrst to make algorithmic improvements to AIRL such that Nash equilibrium can be reached even for a non-concave model function of the generator, and that diversity of the outputs can be increased.
A DVERSARIAL I NVERSE R EINFORCEMENT L EARNING
Due to the high variance estimate of a full sentence and the reward ambiguity problem, instead of learning reward for a complete sentence, we could learn reward distribution p θ ( w t , s t ) for each word-state pair ( w t , s t ) so that the true reward can be recovered at optimality (Fu et al., 2018).
In the following, we introduce how AIRL disentangles reward for each word-state pair ( w t , s t ) .
AIRL is an adversarial reward learning algorithm based on Maximum-Entropy-IRL.
Finn et al.
ﬁrst proved that Maximum-Entropy-IRL is mathematically equivalent to GAN under a special form of the discriminator:
[Equation]
where p θ ( w t , s t ) is the actual probability distribution estimated by the discriminator.
π ( w t , s t ) is the policy distribution produced by the generator, which is also called vocabulary distribution under the context of image captioning.
The goal is to estimate p θ ( w t , s t ) that approximates the true data distribution p true ( w t , s t ) , as well as to learn an optimal vocabulary distribution π that maximizes the reward.
Subsequently, considering reward ambiguity problem, Fu et al.
(2018) further extended the theory to AIRL by adding a reward shaping term h ϕ into f θ ( w t , s t ) , and thus disentangled reward from different system dynamics (i.e., image-caption pairs):
[Equation]
where g θ denotes the reward approximator that recovers the true reward up to a constant, and h ϕ is the reward shaping term that preserves the optimal π .
γ is a constant in range (0 , 1] .
In the context of divergence minimization, the adversarial process can be represented as a min-max game (Mescheder & Geiger, 2017):
[Equation]
where p true is the true data distribution and π is the vocabulary distribution learned by the generator.
( w true t , s true t ) is the word-state pair of the true data.
Despite of the capability of AIRL in disentangling reward for each word, it is difﬁcult for the above
AIRL algorithm to converge to Nash equilibrium and to produce diverse outputs through adversarial training (See Section 4.2 for details).
These issues can result in a non-optimal solution and lack of diversity of the generated descriptions.
Aiming to learn the optimal compact reward as well as diverse captions, we reﬁne the loss function to shift the stationary point towards Nash equilibrium and to mitigate mode collapse in the two-player game.
L EARNING C OMPACT R EWARD FOR I MAGE C APTIONING
To address the problems discussed above, we reﬁne the loss function to: 1) ﬁnd a compact reward function that is optimal; 2) increase diversity of the generated captions.
In particular, a constant term is used to solve 1) by shifting the stationary point to Nash equilibrium, and a conditional term is introduced to solve 2) by utilizing mode control techniques.
Our algorithm is detailed in Algorithm 1, where n is the sentence length and N denotes number of iterations.
In the following notations, θ and ϕ are the parameters of the discriminator, and ψ represents the parameter of the generator.
w t and s t denote the t th word and its corresponding hidden state vector, respectively.
For better clarity, policy π ψ is hereinafter referred to as vocabulary distribution.
4. D ISCRIMINATOR
The objective of the discriminator is to distinguish the true caption from the generated one.
At time t , the discriminator maximizes the divergence in Eq.
(4) by
[Equation]
where p true is the distribution of the true caption, and π ψ is the vocabulary distribution estimated by the generator.
D θ,ϕ is computed as in Eq.
(1) and Eq.
(2), where the discriminator learns the reward function f θ,ϕ for D θ,ϕ and the generator estimates the vocabulary distribution π ψ for D θ,ϕ , respectively.
4. G ENERATOR
In the following, D θ,ϕ is represented as below (Fu et al., 2018) using Eq.
(1) and Eq.
[Equation]
Given word w t that is sampled from the vocabulary distribution π ψ , the generator maximizes
D θ,ϕ ( w t , s t ) by
[Equation]
Using REINFORCE algorithm (Sutton & Barto, 1998)
[Equation]
the gradient ∇ ψ L t becomes:
[Equation]
[Equation]
When the generator converges (i
[Equation]
e.
[Equation]
∇ ψ L t = 0 )
[Equation]
there exists two stationary points: ∇ ψ π ψ = 0 and log( π ψ )
= f θ,ϕ ( w t , s t ) − 1 .
If Nash equilibrium can be reached at optimality, D θ,ϕ should converge to 0 .
5 when ∇ ψ L t = 0 .
Thus it’s only possible for the ﬁrst point to reach Nash equilibrium since
D θ,ϕ =
[Equation]
ϕ
[Equation]
s
[Equation]
= sigmoid
[Equation]
1) (cid:54) = 0 .
5 at the second point.
However, even for the ﬁrst point, Nash equilibrium exists only for a concave π ψ , requiring Hessian of the gradient vector ﬁled being positive deﬁnite (Mescheder & Geiger, 2017).
To relax this constraint, a constant term is added into the expectation in Eq.
[Equation]
(10) to expand the feasible region of reaching Nash equilibrium by shifting the second stationary point to
D θ,ϕ ( w t ) = sigmoid (0) = 0 .
The constant term makes it possible to achieve Nash equilibrium even for a non-concave π ψ , which also makes it easier for the adversarial model to converge .
It is noted that the constant term can also be regarded as baseline in REINFORCE, except it is utilized to centralize the stationary point instead of reducing variance of the estimation.
In practice, mode collapse occurs when the generator produces a single or limited modes, which exhibits as little diversity in image captioning.
To mitigate mode collapse (Mirza & Osindero, 2014) and increase the diversity of the generated captions, we add ground truth data into the generator as a conditional term :
[Equation]
where π true ψ is the approximated true caption distribution in the generator, and E w true t
∼ π true ψ [ · ] is the conditional term .
See Appendix G for the deduction details.
The coefﬁcient of log( π true ψ ) is symmetrical to the coefﬁcient of log( π ψ ) and is updated adaptively in the training process.
The conditional term helps in strengthening the generator in the adversarial training.
When D true > D gen , the gradient of the true data becomes larger than that of the generated one ( ∇ π true ψ L t > ∇ π ψ L t ), and thus the generator further increases the probability of the true word ( π true ψ ).
Otherwise (i.e., D true < D gen ), the generator prefers sampling its self-generated words to fool the discriminator.
By switching between the true words and the generated ones, the generator maintains informative gradient during the adversarial training (Peng et al., 2019).
Note that adding the conditional term does not change the model’s convergence to Nash equilibrium since π ψ = π true ψ at the second stationary point.
E XPERIMENTS
In the experiments, we validate the effectiveness of the proposed algorithm by answering three questions: 1) Is the caption evaluator (i.e., discriminator) capable of learning compact reward? 2)
Driven by the learned reward, is the caption generator effective to produce qualitative captions? 3)
How does our algorithm perform when built on or compared with existing methods?
To answer 1), we ﬁrst tested the compactness of the learned reward by observing performance of the collected top- k captions.
Then we explored the correlation between the learned reward and the human evaluation results.
To answer 2), we evaluated the quality of the generated caption on its content, diversity and grammar.
To answer 3), we built our algorithm on existing learning methods and compared their performance.
We also conducted ablation experiments to demonstrate the importance of each component of our algorithm.
5. I MPLEMENTATION D ETAILS
We conducted experiments on the well-known benchmark dataset MS COCO (Chen et al., 2015).
The dataset has 123,287 labeled images and each image has at least 5 human annotated captions as reference.
To assess the robustness of our algorithm, we use two splits of the COCO dataset: standard split (Karpathy & Fei-Fei, 2015) which is created by randomly picking test images, and robust split (Lu et al., 2018) which is organized to maximize difference of the co-occurrence distribution between the training and test set.
The robust split is recently proposed and is more challenging due to its distribution difference between the training and test set.
The standard split has 113287 / 5000 /train/val/test images and the robust split has 110234 / 3915 / 9138 train/val/test images.
We implement our algorithm using Adam optimizer (Kingma & Ba, 2014) with ﬁxed learning rate 10 − 5 .
Our vocabulary size is ﬁxed to 10,000 including a special start sign < BOS > and an end sign < EOS > .
In the generator, the number of hidden nodes of every layer is 512 .
For simplicity, the discriminator has the same model structure as the generator except having one additional layer for estimating h ϕ .
For fair comparison, all the methods in MLE, RL, GAN, AIRL and rAIRL were produced using the same image features and model structure in (Anderson et al., 2018).
Speciﬁcally,
RL is the self-critical algorithm in (Rennie et al., 2017).
GAN is the adversarial algorithm in (Dai &
Lin, 2017) that learns sentence-level reward.
AIRL is the standard adversarial inverse reinforcement learning method in (Fu et al., 2018), and rAIRL is the proposed method.
Note that our scores of MLE are lower on the standard split but higher on the robust split than (Anderson et al., 2018) because
1) we used ﬁxed number of the bounding box (i.e., 36 ) for simplicity; 2) the hyperparameters were tuned to adapt to both splits and thus are not exactly the same with (Anderson et al., 2018).
5. P ERFORMANCE OF THE R ECOVERED R EWARD
[Table]
Compactness.
Compactness means that the reward values should be close for similar words and different for distinct words.
For example, kid can also be referred to as little boy or little girl , and thus their reward values should be close to each other in the discriminator.
To see the correlation between the reward differences and semantic differences, we replace a given word w t in the generated caption with a similar word w similar and a distinct word w distinct , respectively.
Speciﬁcally, in a generated caption, the ﬁrst word that belongs to the COCO 80 class 1 (Lu et al., 2018) is replaced.
A sentence is discarded if no word can be replaced.
The words within the same class are considered to be similar (such as bike and bicycle ), and the words that belong to difference classes are distinct (such as man and bike ).
For RL, since it maximizes a handcrafted reward (SPICE (Anderson et al., 2016) in our experiment) instead of learning a reward function, the reward difference is the variation of SPICE before and after replacement.
For reward-learning methods, the reward difference is the variation of the learned reward.
The semantic difference is the Euclidean distance between the Glove embedding vectors of two words (Pennington et al., 2014).
The results are reported in Table 1.
Higher correlation indicates better compactness.
RL serves as a baseline in that the handcrafted reward SPICE compares n -gram overlapping without considering the semantic difference.
The reward differences of rAIRL correlate the best with the semantic differences for both similar words and distinct words, proving the compactness of the learned reward.
It’s also noted that due to the reward ambiguity problem, the reward differences of GAN poorly correlate with the semantic differences, especially for similar words.
[Table]
Correlation with human evaluation.
As a caption evaluator, the discriminator learns g θ that recovers the true reward up to a constant at optimality (Fu et al., 2018).
We explore the correlation between the recovered reward and the human evaluation scores from the COMPOSITE dataset (Aditya et al., 2017), where the Amazon Mechanical Turk (AMT) workers evaluate two aspects of the captions (i.e., correctness and throughness) at a range of 1 - 5 , see Appendix C for details of the human evaluation process.
The correlation is evaluated using Pearson p , Kendall’s τ and Spearman’s r correlation coefﬁcients.
In Table 2, the reward of AIRL/rAIRL is the sum of the word-wise reward g θ , and the reward of rAIRL+SPICE is a linear combination of g θ and the SPICE score.
Among the reward-learning methods, AIRL poorly correlates with human, whereas the proposed rAIRL improves AIRL on all the correlation metrics, especially on the Pearson correlation (from 0 .
04 to 0 . 43 ).
Furthermore, a simple combination of SPICE and the recovered reward leads to an increased correlation with the human scores, which proves the capacity of the discriminator as a caption evaluator.
5. E VALUATION ON THE G ENERATED C APTIONS .
[Table]
Content correctness.
For a comprehensive evaluation of the content correctness
[Table]
the results of both the handcrafted metrics and human studies are shown in Table 3.
For the handcrafted metrics
[Table]
we report scores of SPICE and the recently proposed CHAIR s and CHAIR i since they correlate well with human (Anderson et al., 2016; Rohrbach et al., 2018).
SPICE computes similarity with the ground truth captions based on scene graph whilst CHIAIR s and CHIAR i indicate ratio of hallucinated objects.
The full results of other handcrafted metrics can be found in Appendix E.
Compared with non-adversarial methods (i.e., MLE, RL), existing adversarial net (GAN) does not perform well on
SPICE due to the reward ambiguity problem, whereas our rAIRL improves GAN (from 16 .
8 to 18 . 7 )
by disentangling reward for each word, and even outperforms RL (from 18 .
1 to 18 . 7 )
on the robust split.
The lowest scores on CHIAIR s and CHIAR i suggest that object hallucination is less likely in rAIRL.
As for the human evaluation, HE in Table 3 indicates the percentage of captions that are considered the best among the ﬁve methods.
See Appendix C for details of the human evaluation process.
The descriptions generated by our rAIRL are considered the best for over 40% images, whilst RL has the lowest scores that are less than 10% .
Diversity.
The diversity of captions is evaluated on a corpus level, indicating to what extent the generated captions of different images have diverse expressions.
The results are presented in Table 3.
VC indicates vocabulary coverage, which is the number of vocabularies of the generated captions over number of vocabularies of the ground truth captions.
NS represents ratio of novel sentence, which is the ratio of sentences that do not appear in the training set.
The fact that RL has high ratio of novel sentence ( 88 .
but low vocabulary coverage ( 11 .
7% ) suggests that it uses high-frequency words (such as “in a”, “of a”) to reconstruct captions that appear to be different from the training set (Li et al., 2019a).
See Appendix A for a few examples.
Our rAIRL improves AIRL on the diversity metrics and outperforms other learning methods on vocabulary coverage, indicating its capability of generating diverse descriptions on a corpus level.
[Table]
Grammar.
We used LanguageTool 2 to check grammar of the generated captions.
Table 4 shows percentage of sentences that have grammar errors found by LanguageTool: 1) Redundancy means repeated phrases in a sentence; 2) Agreement Error means subject-verb agreement error, such as “people is”; 3) Article Misuse denotes inappropriate usage of indeﬁnite articles, such as using “a” before uncountable nouns or plural words; 4) Incomplete Sentence refers to incomplete sentence that lacks a subject.
We found captions produced by RL have the most grammar errors ( 5 . 64% on the standard split and 4 .
67% on the robust split), especially the Article Misuse.
On the other hand, by approximating the true data distribution of each word in the sentence, rAIRL and MLE have the least grammar errors among all learning methods ( 0 .
78% on the standard split and 0 .
57% on the robust split)).
We also noticed that each method except rAIRL is biased towards a particular type of grammar error: agreement error in MLE, article misuse in RL, redundancy in GAN, article misuse in AIRL.
On both splits, our rAIRL does not appear to be biased towards a speciﬁc type of these grammar errors.
Summary.
The proposed rAIRL constantly performs well on both splits of MS COCO and is capable of producing qualitative captions with few grammar errors.
As a new adversarial algorithm, rAIRL enhances GAN by disentangling compact reward for each word in the caption and improves
AIRL by shifting the stationary point towards Nash equilibrium.
In the following sections, we ﬁrst give ablation studies to see which component of our method explains the performance improvements, and then compare rAIRL with existing methods.
5. C OMPARISON R ESULTS
[Table]
Ablation studies.
We conducted ablation experiments to understand the importance of each component of our algorithm.
Speciﬁcally, the constant term in Eq.
(9) and the conditional term in Eq.
(16) is removed, respectively.
Scores of all the evaluation techniques mentioned above are presented in Table 5.
We found that all the scores drop after removing either one of the terms.
Comparing these two terms, the constant term seems more important in recognizing objects and relations in the image since removing it has larger drop on SPICE.
The lager drop on vocabulary coverage and ratio of novel sentence in the second row indicates that the conditional term plays a signiﬁcant role in increasing the diversity of the generated captions.
More results on using different model architectures are included in Appendix D.
[Table]
Comparison with existing methods.
Based on the learning methods, existing models are divided into three categories in Table
[Table]
and we chose recent proposed methods for comparison: Att2in (Rennie et al., 2017), G-GAN (Dai & Lin, 2017), NBT (Lu et al., 2018), Up-Down
(Anderson et al., 2018) and GAN 2 , GAN 3 (Dognin et al., 2019).
Although some metrics based on n -gram overlapping (BLEU4, CIDEr) do not correlate well with human, their results are also reported in Table 6 for fair comparison.
Among the adversarial methods (GAN category), our rAIRL performs the best on all metrics.
The results on COCO online server are given in Appendix E.
To further demonstrate the generalization ability of our algorithm, we built our algorithm on the non-adversarial based models.
The composite models are denoted with rAIRL+MLE and rAIRL+RL.
In rAIRL+MLE, the conditional term is replaced by the cross-entropy loss of MLE; in rAIRL+RL, the RL loss is added into the loss function of the generator.
In Table 6, our rAIRL+MLE further improves the MLE baseline (i.e., Up-Down using MLE loss) on SPICE, whereas rAIRL+RL does not improve the RL baseline (i.e., Up-Down using RL loss) on these evaluation metrics.
This is caused by the difﬁculty of normalizing the learned reward and the handcrafted reward to the same order of magnitude (Shelton Christian, 2001), and we leave this problem to our future work.
C ONCLUSION
In this paper, we address the reward ambiguity problem in image captioning and propose a reﬁned
Adversarial Inverse Reinforcement Learning (rAIRL) method that solves the problem by disentangling reward for each word in a sentence.
Moreover, it achieves stable adversarial training by reﬁning the loss function to shift the stationary point towards Nash equilibrium and mode control technique is incorporated to mitigate mode collapse.
It is demonstrated that our method can learn compact reward through extensive experiments on MS COCO
A V ISUALIZED R ESULTS OF G ENERATED C APTIONS
B D IAGNOSE A ND I MPROVE C APTIONS
[Table]
Since the proposed rAIRL learns a word-wise reward
[Table]
it’s also applicable to diagnose a given caption by ﬁnding the wrong word (e.g., the word whose reward decreases sharply compared with that of the previous word) and rewriting the caption to improve its quality.
For example, improving a man is playing soccer to a man and a kid are playing soccer can be done by rewriting the caption from is .
Therefore, we choose to rewrite a given caption (source caption) from the word whose reward has a decrease rate larger than 50% .
However, we found that even rewriting the source caption from a random position using rAIRL can also improve the evaluation scores.
Thus, rewriting from a random position is selected as the baseline to compare with rewriting from the located position.
Table 7 shows results of rewriting from the located position, where the source captions are given by MLE, RL,
GAN and AIRL.
Beside each score we report its improvement relative to rewriting from a random position, whose values are mostly positive.
This demonstrates that the proposed rAIRL can diagnose the caption at a word level, and further improves the caption quality by rewriting from the located position.
C H UMAN E VALUATION P ROCESS
We conducted two types of human studies, one for evaluating the learned reward (in Section 5.2), and the other for examining quality of the generated captions (in Section 5.3).
In the ﬁrst human study experiment (in Section 5.2), we used the human scores in the COMPOSITE 3 dataset (Aditya et al., 2017), whose images are subsets from Flickr8k, Flickr30k and MS COCO.
The descriptions from this dataset are either ground truth captions or generated sentences by (Aditya et al., 2017; Johnson et al., 2015).
In the human evaluation process, the AMT worker was asked to give a score at range of [Table] 1-5 to evaluate the correctness and throughness of each sentence.
Captions with length exceedingwere discarded, resulting a total of 11 , 657 sentences.
Full results of the correlation is shown in Table 8.
SPICE correlates better with human evaluation when compared with other handcrafted metrics, whilst the composite metric rAIRL+SPICE further increases the correlation
[Table]
In the second human study experiment (in Section 5.3)
[Table]
we randomly selected 500 test images from the standard split and robust split of MS COCO, respectively.
The worker was asked “which caption is the best” by given an image with ﬁve sentences generated from the adversarial and non-adversarial methods, as shown in Figure 7.
The worker was allowed but not encouraged to make multiple choices if he/she thinks these captions are equally correct.
The order of captions produced by different methods was randomized.
Following (Shetty et al., 2017), each image in the test set was evaluated by 5 workers.
D A BLATION E XPERIMENTS ON M ODEL A RCHITECTURES
Theoretically, our algorithm is model-agnostic since it is independent of the design of model architecture.
For empirical support of the claim, we show results of using Att2in (Rennie et al., 2017) and Up-Down (Anderson et al., 2018) architectures in Table 9.
We report the metrics used in the original paper for fair comparison.
The proposed rAIRL mainly improves SPICE, which correlates well with human evaluations, on both architectures. [Table]
E F ULL R ESULTS ON MS COCO
We adopt SPICE to evaluate content correctness in the paper because it has better correlation with human judgments(Anderson et al., 2016).
Table 10 gives full results of the handcrafted metrics on two splits of MS COCO.
Comparing the adversarial (GAN, AIRL, rARIL) and non-adversarial (MLE,
RL) methods, RL outperforms other methods on most metrics.
In adversarial methods, the proposed rAIRL performs the best.
Table 11 shows results on the MS COCO online test server.
The proposed rAIRL improves AIRL on all the metrics.
[Table]
[Table]
F D ISCUSSION ON L OSS F UNCTIONS
[Table]
We compare the formula of the proposed loss function with existing methods in Table [Table] including
MLE, RL and GAN.
n is the length of a sentence.
r is the handcrafted metric, such as BLEU, CIDEr and SPICE.
π t is the probability of the t th generated word [Table] and π t true is the probability of the t th true word.
The loss functions are rewritten using similar symbols for direct comparison.
MLE maximizes the probability of the true data π true t whist RL and GAN optimize the reward by sampling from π t .
GAN is different from RL in that its reward is learned from the discriminator adversarially instead of being predeﬁned.
GAN is capable of mimicking human-written captions by adversarial learning, but the estimated reward function D gen of a full trajectory can be explained by multiple optimal policies and thus is too ambiguous.
The proposed rAIRL further disentangles the reward into D gen t at each time step t , as well as incorporating the true data for better diversity.
From the perspective of loss functions, rAIRL can be regarded as an integration of GAN and MLE using coefﬁcients σ − 1 ( D gen t ) and σ − 1 ( D true t ) .
G G ENERATOR O BJECTIVE
Below we show the deductions details of the generator objective (i.e., Eq.
(16) in section 4.2).
Taking the integral of ∇ ψ E w t ∼ π ψ [ · ] w.r.t.
ψ
E
[Equation]
According to Eq.
[Equation]
Thus we have
[Equation]
Similarly, replacing w t with w true t we get
[Equation]
Thus
[Equation]
the generator objective is:
[Equation]