L EARNING TO G ENERATE G ROUNDED V ISUAL C APTIONS WITHOUT L OCALIZATION S UPERVISION
Anonymous authors
Paper under double-blind review
A BSTRACT
When automatically generating a sentence description for an image or video, it often remains unclear how well the generated caption is grounded, or if the model hallucinates based on priors in the dataset and/or the language model.
The most common way of relating image regions with words in caption models is through an attention mechanism over the regions that are used as input to predict the next word.
The model must therefore learn to predict the attentional weights without knowing the word it should localize.
This is difﬁcult to train without grounding supervision since recurrent models can propagate past information and there is no explicit signal to force the captioning model to properly ground the individual decoded words.
In this work, we help the model to achieve this via a novel cyclical training regimen that forces the model to localize each word in the image after the sentence decoder generates it, and then reconstruct the sentence from the localized image region(s) to match the ground-truth.
Our proposed framework only requires learning one extra fully-connected layer (the localizer), a layer that can be removed at test time.
We show that our model signiﬁcantly improves grounding accuracy without relying on grounding supervision or introducing extra computation during inference for both image and video captioning tasks.
I NTRODUCTION
Vision and language tasks, such as visual captioning, combine linguistic descriptions with data from real-world scenes.
Deep learning models for such tasks have achieved great success, driven in part by the development of attention mechanisms that focus on various objects in the scene while generating captions.
The resulting models, however, are known to have poor grounding performance (Liu et al., 2017), leading to undesirable behaviors such as object hallucinations (Rohrbach et al., 2018), despite having high captioning accuracy.
That is, they often do not correctly associate generated words with the appropriate image regions ( e.g., objects) in the scene, resulting in models that lack interpretability.
Several existing approaches have tried to improve the grounding of captioning models.
One class of methods generate sentence templates with slot locations explicitly tied to speciﬁc image regions.
These slots are then ﬁlled in by visual concepts identiﬁed by off-the-shelf object detectors (Lu et al., 2018).
Other methods have developed speciﬁc grounding or attention modules that aim to attend to the correct region(s) for generating visually groundable word.
Such methods, however, rely on explicit supervision for optimizing the grounding or attention modules (Liu et al., 2017; Zhou et al., 2019) and require bounding box annotations for each visually groundable word.
In this work, we propose a novel cyclical training regimen that is able to signiﬁcantly improve grounding performance without any grounding annotations.
The key insight of our work is that current models use attention mechanisms conditioned on the hidden features of recurrent modules such as LSTMs, which leads to effective models with high accuracy but entangle grounding and decoding.
Since LSTMs are effective at propagating information across the decoding process, the network does not necessarily need to associate particular decoded words with their corresponding image region(s).
However, for a captioning model to be visually grounded, the model has to predict attentional weights without knowing the word to localize.
Based on this insight, we develop a cyclical training regimen to force the network to ground individual decoded words: decoding −→ localization −→ reconstruction .
Speciﬁcally, the model of the decoding stage can be any state-of-the-art captioning model; in this work, we follow GVD (Zhou et al., [Figure] Is
[Figure]
2019) to extend the widely used Up-Down model (Anderson et al., 2018).
At the localization stage, each word generated by the ﬁrst decoding stage is localized through a localizer , and the resulting grounded image region(s) are then used to reconstruct the ground-truth caption in the ﬁnal stage.
Both decoding and reconstruction stages are trained using a standard cross-entropy loss.
Key to our method, both stages share the same decoder, thereby causing the localization stage to guide the decoder to improve its attention mechanism.
Our method is simple and only adds a fully-connected layer to perform localization.
During inference, we only use the (shared) decoder, thus we do not add any computational cost.
We benchmark our proposed method on the challenging Flickr30k Entities image captioning dataset (Plummer et al., 2015) and the ActivityNet-Entities video captioning dataset (Zhou et al., 2019) on both captioning and grounding performances.
In addition to the existing grounding metric that calculate the grounding accuracy for each object class (Zhou et al., 2019), we further include a grounding metric that compute grounding accuracy for each generated sentence.
This new metric on each sentence removes the stringency of the original evaluation metric (as we discuss in Sec. 4) and provides an alternative way of measuring the grounding performance.
Despite the simplicity of our proposed method, we are able to signiﬁcantly surpass prior unsupervised models quantitatively and qualitatively on both datasets.
We achieve around 18% relative improvements in terms of bridging the gap between the unsupervised baseline and supervised methods on
Flickr30k Entites and around 34% on ActivityNet-Entities.
We further ﬁnd that our method can even outperform the supervised method on infrequent words, owing to its self-supervised nature.
Contributions summary.
We propose object re-localization as a form of self-supervision for grounded visual captioning and present a cyclical training regimen that re-generates sentences after re-localizing the objects conditioned on each word, implicitly imposing grounding consistency.
We evaluate our proposed approach on both image and video captioning tasks.
We show that the proposed training regime can boost grounding accuracy over a state-of-the-art baseline, enabling grounded models to be trained without bounding box annotations, while retaining high captioning quality across two datasets and various experimental settings.
Our code will be publicly released and can be found in supplemental.
R ELATED WORK
Visual captioning.
Neural models for visual captioning have received signiﬁcant attention recently (Anderson et al., 2018;
Ma et al., 2018;
Lu et al., 2018; Donahue et al., 2015; Venugopalan et al., 2015; Rohrbach et al., 2017b; Venugopalan et al., 2017; Rohrbach et al., 2017a;
Shetty et al., 2017;
Park et al., 2019).
Most current state-of-the-art models contain attention mechanisms, allowing the process to focus on subsets of the image when generating the next word.
These attention mechanisms can be deﬁned over spatial locations (Vinyals et al., 2015), semantic metadata (Li et al., 2018; Yao et al., 2017;
You et al., 2016; Zhou et al., 2017) or a predeﬁned set of regions extracted via a region proposal network (Ma et al., 2018; Zanﬁr et al., 2016; Anderson et al., 2018; Lu et al., 2018; Das et al., 2013; Kulkarni et al., 2013).
In the latter case, off-the-shelf object detectors are ﬁrst used to extract object proposals (Ren et al., 2015; He et al., 2017) and the captioning model then learns to dynamically attend over them when generating the caption.
Visual grounding.
Although attention mechanisms are generally shown to improve captioning quality and metrics, it has also been shown that they don’t really focus on the same regions as a human would (Das et al., 2017).
This make models less trustworthy and interpretable, and therefore creating grounded image captioning models, i.e., models that accurately link generated words or phrases to speciﬁc regions of the image, has recently been an active research area.
A number of approaches have been proposed, e.g., for grounding phrases or objects from image descriptions (Rohrbach et al., 2016;
Hu et al., 2016; Xiao et al., 2017; Deng et al., 2018;
Zhou et al., 2019; Zhang et al., 2019), grounding visual explanations (Hendricks et al., 2018), visual co-reference resolution for actors in video (Rohrbach et al., 2017a), or improving grounding via human supervision (Selvaraju et al., 2019).
Recently, Zhou et al.
(2019) presented a model with self-attention based context encoding and direct grounding supervision that achieves state-of-the-art results in both the image and video tasks.
They exploit ground-truth bounding box annotations to signiﬁcantly improve the visual grounding accuracy.
In contrast, we focus on reinforcing the visual grounding capability of the existing captioning model via a cyclical training regimen without using bounding box annotations and present a method that can increase grounding accuracy while maintaining comparable captioning performance with state of the arts.
Cyclical training.
Cycle consistency (Wang et al., 2013; Zhu et al., 2017; He et al., 2016; Chen & Lawrence Zitnick, 2015) has been used recently in a wide range of domains, including machine translation (He et al., 2016), unpaired image-to-image translation (Zhu et al., 2017), visual question answering (Shah et al., 2019), question answering (Tang et al., 2018), image captioning (Chen &
Lawrence Zitnick, 2015), video captioning (Wang et al., 2018; Duan et al., 2018), captioning and drawing (Huang et al., 2018) as well as domain adaptation (Hosseini-Asl et al., 2019).
While the cyclical training regime has been explored vastly in both vision and language domains, it has not yet been used for enforcing the visual grounding capability of a captioning model.
M ETHOD
Notation.
For a visual captioning task, we denote the input image as I (or input video as V ) and the target sentence as S .
Each image (or video) is represented by spatial feature map(s) extracted by a ResNet-101 model and a bag of regions obtained from Faster-RCNN (Ren et al., 2015) as
R =
[ r 1 , r 2 , ..., r N ] ∈ R d × N .
The target sentence is represented as a sequence of one-hot vectors y ∗ t ∈ R s , where T is the sentence length, t ∈ 1 , 2 , ..., T , and s is the dictionary size.
3. B ASELINE
We reimplemented the model used in GVD (Zhou et al., 2019) without self-attention for region feature encoding (Ma et al., 2018; Vaswani et al., 2017) as our baseline.
It is an extension of the state-of-the-art Up-Down
(Anderson et al., 2018) model with the grounding-aware region encoding (see Appendix A.5).
Speciﬁcally, our baseline model uses two LSTM modules: Attention LSTM and
Language LSTM.
The Attention LSTM identiﬁes which visual representation in the image is needed for the Language LSTM to generate the next word.
It encodes the global image feature v g , previous hidden state output of the Language LSTM h
Lt − 1 , and the previous word embedding e t − 1 into the hidden state h
At .
[Equation]
where [; ] denotes concatenation, and W e are learned parameters.
We omit the Attention LSTM input hidden and cell states to avoid notational clutter in the exposition.
The Language LSTM uses the hidden state h
At from the Attention LSTM to dynamically attend on the bag of regions R for obtaining visual representations of the image ˆ r t to generate a word y t .
[Equation]
A person is sitting on a stone .
A man is sitting on a bench .
Decoding
Localization
Reconstruction
Decoding loss
Recon.
loss
D e c od
i ng
Lo c a li z a t i on
R e c on .
+ 𝑟̂ # Attended ROIs 𝑟̂ #$ Localized ROIs where W aa and W a are learned parameters. The conditional probability distribution over possible output words y t is computed as: [Equation] where y 1: t − 1 is a sequence of outputs ( y 1 , ..., y t − 1 ) .
We refer the Language LSTM and the output logit layer as the complete language decoder.
3. O VERVIEW
Our goal is to enforce the generated caption to be visually grounded, i.e., attended image regions correspond speciﬁcally to individual words being generated, without ground-truth grounding supervision.
Towards this end, we propose a novel cyclical training regimen that is comprised of decoding , localization , and reconstruction stages, as illustrated in Figure 2.
The intuition of our method is that the baseline network is not forced to generate a correct correspondence between the attended objects and generated words, since the LSTMs can learn priors in the data instead of looking at the image or propagate information forward which can subsequently be used to generate corresponding words in future time steps.
The proposed cyclical training regimen, in contrast, aims at enforcing visual grounding to the model by requiring the language decoder (Eq. 3) to rely on the localized image regions ˆ r lt to reconstruct the ground-truth sentence, where the localization is conditioned only on the generated word from the decoding stage.
Our cyclical method can therefore be done without using any annotations of the grounding itself.
Speciﬁcally, let y dt
= D d (ˆ r t ; θ d ) be the initial language decoder with parameters θ d (Eq. 3), trained to sequentially generate words y d t .
Let G ( y d t ; θ g ) deﬁne a localizer unit with parameters θ g , that learns to map (ground) each generated word to region(s) in the image, i.e., ˆ r lt =
G
( y dt , R ; θ g ) .
Finally, let y lt = D l (ˆ r lt ; θ l ) be a second decoder, that is required to reconstruct the ground-truth caption using the localized region(s), instead of the attention computed by the decoder itself.
We deﬁne the cycle:
[Equation]
where D d and D l share parameters
Although parameters are shared, the inputs for the two language decoders differ, leading to unique LSTM hidden state values during a run.
Note that the Attention
LSTMs and logit layers in the two stages also share parameters, though they are omitted for clarity.
Through cyclical joint training, both D d and D l are required to generate the same ground-truth sentence.
They are both optimized to maximize the likelihood of the correct caption:
[Equation]
During training, the localizer regularizes the region attention of the reconstructor and the effect is further propagated to the baseline network in the decoding stage, since the parameters of Attention 𝑣 "
[Figure]
$ ℎ $& ℎ $’(& ℎ $’() ℎ $) ℎ $’( [Figure] 𝑊 + 𝑦 $’( 𝑟 ( 𝑟𝑟 .
𝛼 $ …
Soft [Figure] attention
[Figure]
[Figure] [Figure] $ ℎ $)
Localizer
[Figure] LSTM [Figure]
Logit
Localization [Figure] Recon
[Figure]
Decoding [Figure]
Localizer
Localizer
[Figure]
LSTM and Language LSTM are shared for both decoding and reconstruction stages.
Note that the gradient from reconstruction loss will not backprop to the decoder D d in the decoding stage since the generated words used as input to the localizer are leafs in the computational graph.
The network is implicitly regularized to update its attention mechanism to match with the localized image regions ˆ r t (cid:55)→ ˆ r lt .
In Sec. 4.3, we demonstrate that the localized image regions ˆ r lt indeed have higher attention accuracy than ˆ r t when using ground-truth words as inputs for the localizer.
3. C YCLICAL T RAINING
We now describe each stage of our cyclical model in detail, as illustrated in Figure 3.
Decoding.
We ﬁrst use the baseline model presented in Sec.
3.1 to generate a sequence of words y
=
[ y d 1 , y d 2 , ..., y dT ] , where T is the ground-truth sentence length.
Localization.
Following the decoding process, a localizer G is then learned to localize the image regions from each generated word y t .
[Equation]
where e t is the embedding for the word generated during decoding stage at step t , r n is the image representation of a region proposal, and W e and W l are the learned parameters.
Based on the localized weights β t , the localized region representation can be obtained by ˆ r lt = Rβ .
Reconstruction.
Finally, the shared language decoder D l relies on the localized region representation ˆ r lt to generate the next word.
The probability over possible output words is:
[Equation]
Given the target ground truth caption y ∗ 1: T and our proposed captioning model parameterized with θ , we minimize the following cross-entropy losses:
[Equation]
where λ 1 and λ 2 are weighting coefﬁcient selected on the validation split.
E XPERIMENTS
Datasets.
We use the Flickr30k Entities image dataset (Plummer et al., 2015) and the ActivityNetEntities video dataset (Zhou et al., 2019) for evaluating our proposed approach.
Flickr30k Entities
Captioning evaluation metrics.
We measure captioning performance using four language metrics, including BLEU (Papineni et al., 2002), METEOR (Banerjee & Lavie, 2005), CIDEr (Vedantam et al., 2015), and SPICE (Anderson et al., 2016).
Grounding evaluation metrics.
Following the grounding evaluation from GVD (Zhou et al., 2019), we measure the attention accuracy on generated sentences, denoted by F1 all and F1 loc .
In F1 all , a region prediction is considered correct if the object word 1 is correctly predicted and also correctly localized.
We also compute F1 loc , which only considers correctly-predicted object words.
Please see illustration of the grounding metrics in Appendix A.1.
In the original formulation, the precision and recall for the two F1 metrics are computed for each object class , and it is set to zero if an object class has never been predicted.
The scores are computed for each object class and averaged by the total number of classes.
Such metrics are extremely stringent as captioning models are generally biased toward certain words in the vocabulary, given the long-tailed distribution of words.
In fact, both the baseline and proposed method generate about 45% of the annotated object words within the val set in Flickr30k Entities.
The grounding accuracy of the other 55% of the classes are therefore zero, making the averaged grounding accuracy seemingly low.
Measuring grounding per generated sentence.
Instead of evaluating grounding on each object class (which might be less intuitive), we include a new grounding evaluation metric per sentence to directly reﬂect the grounding measurement of each generated sentence.
The metrics are computed against a pool of object words and their ground-truth bounding boxes (GT bbox) collected across ﬁve GT captions on Flickr30k Entities (and one GT caption on ActivityNet-Entities).
We use the same
Prec all , Rec all , Prec loc , and Rec loc as deﬁned previously, but their scores are averaged on each of the generated sentence.
As a result, the F1 loc per sent measures the F1 score only on the generated words.
The model will not be punished if some object words are not generated, but it also needs to maintain diversity to achieve high captioning performance.
4. I MPLEMENTATION AND T RAINING D ETAILS
Region proposal and spatial features.
Following GVD (Zhou et al., 2019), we extracted 100 region proposals from each image (video frame) and encode them via the grounding-aware region encoding .
Please refer to Appendix A.5 for more implementation details.
Training.
We train the model with ADAM optimizer (Kingma & Ba, 2015).
The initial learning rate is set to 1 e − 4 .
Learning rates automatically drop by 10x when the CIDEr score is saturated.
The batch size is 32 for Flickr30k Entities and 96 for ActivityNet-Entities.
We learn the word embedding layer from scratch for fair comparisons with existing work (Zhou et al., 2019).
4. C APTIONING AND G ROUNDING P ERFORMANCE C OMPARISON
Flickr30k Entities.
We ﬁrst compare the proposed method with our baseline with or without grounding supervision on the Flickr30k Entities test set (see Table 1).
To train the supervised baseline, we train the attention mechanism as well as add the region classiﬁcation task using the ground-truth grounding annotation, similar to GVD (Zhou et al., 2019).
We train the proposed baselines and our method on the training set and choose the best performing checkpoints based on their CIDEr score on the val set.
Our experimental results are reported by averaging across ﬁve runs on the test set.
We report only the mean of the ﬁve runs to keep the table uncluttered.
When compared to the existing state of the arts, our proposed baselines achieve comparable captioning evaluation performances and grounding accuracy.
Using the resulting supervised baseline as the upper bound, our proposed method with cyclical training statistically achieves around 20 to 25% relative grounding accuracy improvements for both F 1 all and F 1 loc and 10 to 15% for F 1 all per sent and F 1 loc per sent without utilizing any grounding annotations or additional computation during inference.
ActivityNet-Entities.
We adapt our proposed baselines and method to the ActivityNet-Entities video dataset (see Table 2).
We can see that our proposed method signiﬁcantly improved the grounding
[Table] are considered to be bolded. [Table] accuracy around 25% to 30% relative grounding accuracy improvements for both F 1 all and F 1 loc and around 40% for F 1 all per sent and F 1 loc per sent .
4. A NALYSIS
Are localized image regions better than attended image regions during training?
Given our intuition described in Sec. 3, we expect the decoder to be regularized to update its attention mechanism to match with the localized image regions ˆ r t (cid:55)→ ˆ r lt .
This indicates that the localized image regions should be more accurate than the attended image regions by the decoder during training.
To verify this, we compute the attention accuracy for both decoder and localizer over ground-truth sentences following (Rohrbach et al., 2016;
Zhou et al., 2018).
The attention accuracy for localizer is 20.4% and is higher than the 19.3% from the decoder at the end of training, which conﬁrms our hypothesis.
Grounding performance when using a better object detector.
In Table 1 and 2 we showed that our proposed method signiﬁcantly improved the grounding accuracy for both image and video captioning.
These experimental settings follow the widely used procedure for visual captioning systems: extract regional proposal features and generate visual captions by attending to those extracted visual features. [Figure] [Table] [Figure]
[Table] [Figure] One might ask, what if we have a better object detector that can extract robust visual representation that are better aligned with the word embeddings?
Will visual grounding still an issue for captioning?
To answer this, we ran two sets of experiments (Table 3): (1) Perfect object detector : we replace the
ROIs by ground-truth bbox and represent the new ROIs by learning embedding features directly from ground-truth object words associated with each ground-truth bbox.
This experiment gives an estimate of the captioning and grounding performance if we have (almost) perfect ROI representations (though unrealistic).
We can see that the fully-supervised method achieves an F1 all of only 23%, which further conﬁrms the difﬁculty of the metric and the necessity of our grounding metric on a per sentence level (note that F1 loc per sent shows 90%).
We can also see that baseline (unsup.) still leaves room for improvement on grounding performance.
Surprisingly, our method improved both captioning and grounding accuracy and surpasses the fully-supervised baseline except on the F1 loc per sent .
We ﬁnd that it is because the baseline (sup.) overﬁts to the training set, while ours is regularized from the cyclical training.
Also, our generated object words are more diverse, which is critical for F1 all and F1 loc .
(2) Grounding-biased object detector : we extract ROI features from an object detector pre-trained on Flickr30k.
Thus, the ROI features and their associated object predictions are biased toward the annotated object words but do not generalize to predict diverse captions compared to the original object detector trained from Visual Genome, resulting in lower captioning performance.
We can see that our proposed method still successfully improves grounding and maintains captioning performance in this experiment setting as well.
How does the number of annotations affect grounding performance?
In Figure 4, we present the average F1-score on the Flickr30k Entities val set when grouping classes according to their frequency of appearance in the training set 3 .
We see that, unsurprisingly, the largest difference in grounding accuracy between the supervised and our proposed cyclical training is for the 50 most frequently appearing object classes, where enough training data exists.
As the number of annotated boxes decreases, however, the difference in performance diminishes, and cyclical training appears to be more robust.
Overall, we see that the supervised method is biased towards frequently appearing objects, while grounding performance for the proposed approach is more balanced among classes.
Qualitative analysis.
We additionally conduct qualitative analysis for comparing the baseline (Unsup.) and the proposed method in Figure 5.
Each highlighted word has a corresponding image region annotated on the original image.
The image regions are selected based on the region with the maximum attention weight in α t .
We can see that our proposed method signiﬁcantly outperformed the baseline (Unsup.)
in terms of both the quality of the generated sentence and grounding accuracy.
In addition, we also discuss a number of correct and incorrect examples of our proposed method in
Figure 8 in the Appendix.
Please refer to the Appendix A.4 for further discussions on the qualitative results and the complete sequence of attended image regions of examples in Figure 5.
C ONCLUSION
Working from the intuition that typical attentional mechanisms in the visual captioning task are not forced to ground generated words since recurrent models can propagate past information we devise a novel cyclical training regime to explicitly force the model to ground each word without grounding annotations.
Our method only adds a fully-connected layer during training, which can be removed during inference and we show thorough quantitative and qualitative results demonstrating around 20% or 30% relative improvements in visual grounding accuracy over existing methods for image and video captioning tasks
A A PPENDIX A. G ROUNDING E VALUATION M ETRICS ILLUSTRATED
To help better understand the grounding evaluation metrics used in this work, we illustrated the grounding evaluation metrics in Figure 6.
We deﬁne the number of object words in the generated sentences as A, the number of object words in the GT sentences as B, the number of correctly predicted object words in the generated sentences as
C and the counterpart in the GT sentences as D, and the number of correctly predicted and localized words as E (see illustration of the grounding metrics in Appendix A.1).
A region prediction is considered correct if the object word is correctly predicted and also correctly localized ( i.e., IoU with
GT box > 0.5).
We then compute two version of the precision and recall as Prec all = E A , Rec all = E B ,
Prec loc =
E C , and Rec
loc
= E D .
The original grounding evaluation metric proposed in GVD (Zhou et al., 2019) average the grounding for each object class.
We additionally calculate the grounding accuracy for each generated sentence as demonstrated in the ﬁgure.
From this example, we can see that while Precision all counts dog as a wrong prediction for the dog object class, the Precision loc only cares if man and hat are predicted and correctly localizer (IoU > 0.5).
[Table]
[Table]
A. A DDITIONAL A NALYSIS
Should we explicitly make attended image regions to be similar to localized image regions?
One possible way to regularize the attention mechanism of the decoder is to explicitly optimize ˆ r t (cid:55)→ ˆ r lt via KL divergence over two soft-attention weights α t and β t .
The experimental results are shown in Table 5 ( Attention consistency ).
We use a single run unsupervised baseline with a ﬁx random seed as baseline model for ablation study.
We can see that when explicitly forcing the attended regions to be similar to the localized regions, both the captioning performance and the grounding accuracy remain similar to the baseline (unsup.).
We conjecture that this is due to the noisy localized regions at the initial training stage.
When forcing the attended regions to be similar to noisy localized regions, the Language LSTM will eventually learn to not rely on the attended region at each step for generating sequence of words.
To verify, we increase the weight for attention consistency loss and observed that it has lower grounding accuracy (F1 all = 3.2), but the captioning will reach similar performance while taking 1.5x longer to reach convergence.
Is using only the generated word for localization necessarily?
Our proposed localizer (Eq. 6 and
Figure 3) relies on purely the word embedding representation to locate the image regions.
This forces the localizer to rely only on the word embedding without biasing it with the memorized information from the Attention LSTM.
As shown in the Table 5 (localizer using h A ), although this achieves comparable captioning performance, it has lower grounding accuracy improvement compared to our proposed method.
Can words that are not visually-groundable handled differently?
In the proposed method, all the words are handled the same regardless of whether they are visually-groundable or not.
Yet, typically words that are nouns or verbs are more likely to be grounded, and words like ”a”, ”the”, etc , are not visually-groundable. We explored a few method variants to handle nouns and verbs differently. Mainly, we explored with two variants. Cyclical (1): the reconstruction loss is only computed when the target word is either nouns or verbs.
Cyclical (2): the localized region representation will be invalid (set to zero) if the target word is neither nouns nor verbs.
The experimental results are shown in Table 6, 7, and 8.
For the ﬁrst variant, Cyclical (1), we observed that the captioning performance stays the same while grounding accuracy has a small improvement.
On the other hand, for the second variant, Cyclical (2), we can see that all captioning scores are improved over baseline with CIDEr improved 2.4.
We can also see that grounding accuracy on per sentence basis further improved as well.
We then conducted further experiments on both ActivityNet-Entities and Flickr30k Entities with unrealistically perfect object detector , but the improvements however are not consistent.
In summary: on the Flickr30k Entities test set, we observed that CIDEr is better and grounding per sentence better, on the ActivityNet-Entities val set,
[Table]
[Table]
the captioning performances are about the same but grounding accuracy became worse, and on the
Flickr30k Entities test set with unrealistically perfect object detector, captioning performances are slightly worse but grounding accuracy improved.
A. H UMAN E VALUATION ON GROUNDING
We conduct a human evaluation on the perceptual quality of the grounding.
We asked 10 human subjects to pick the best among two grounded regions (by baseline and Cyclical) for each word.
The subjects have three options to choose from: 1) grounded region A is better, 2) grounded region B is better, and 3) they are about the same (see Figure 7 for example).
Authors or other colleagues familiar with the proposed method were excluded from the study.
Each of the human subjects were given 25 images, each with a varying number of groundable words.
Each image was presented to two different human subjects in order to be able to measure inter-rater agreement.
To avoid being biased towards the object words deﬁned in the dataset for automatic grounding evaluation, for the study we deﬁne a word to be groundable if it is either a noun or verb.
The order of approaches was randomized for each sentence.
Our experiment on the Flickr30k Entities val set showed that: 28.1% of words are more grounded by
Cyclical, 24.8% of words are more grounded by baseline, and 47.1% of words are similarly grounded.
We also measured inter-rater agreement between each pair of human subjects: 72.7% of ratings are the same, 4.9% of ratings are the opposite, and 22.4% of ratings could be ambiguous ( e.g., one chose
A is better, the other chose they are about the same).
We would also like to make a note that the grounded words judged to be similar largely consisted of very easy or impossible cases.
For example, words like mountain, water, street, etc, are typically rated to be ”about the same” since they usually have many possible boxes and is very easy for both models to ground the words correctly.
On the other hand, for visually ungroundable cases, e.g., stand appears a lot and the subject would choose about the same since the image does not cover the fact that the person’s feet are on the ground.
We see that the human study results follow the grounding results presented in the paper and show an improvement in grounding accuracy for the proposed method over a strong baseline.
The improvement is achieved without grounding annotations or extra computation at test time.
A man and woman, each wearing a white shirt, are walking down the sidewalk.
A
[Figure]
A is better.
[Figure]
is better
[Figure]
A and B are about the
[Figure]
A is
[Figure]
B is better
[Figure]
A and B are about the
[Figure]
A is better
[Figure]
B is
[Figure]
A
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
A. A DDITIONAL Q UALITATIVE R ESULTS
In Figure [Figure] we show a number of correct and incorrect examples of our proposed method.
We observe that while the model is able to generate grounded captions for the images, it may sometimes overlook the semantic meaning of the generated sentences, for example, ”A young girl [...] walking down a brick wall” .
Similarly, the model can overlook the spatial relationship between the objects, for instance, ”A man [...] is holding up a ﬂag” .
While a ﬂag is present in the scene and was able to be successfully located with the corresponding word, the man in a black shirt is spatially far from the ﬂag.
In Figure 9, 10, 11, 12, 13, 14,
[Figure] and [Figure] we illustrated the sequence of attended image region when generating each word for a complete image description.
At each step, only the top-1 attended image region is shown.
This is the same as how the grounding accuracy is measured.
Please see the description for Figure 9 - 16 for further discussions on the qualitative results.
A. A DDITIONAL I MPLEMENTATION D ETAILS
Region proposal features.
We use a Faster-RCNN model (Ren et al., 2015) pre-trained on Visual
Genome (Krishna et al., 2017) for region proposal and feature extraction.
In practice, besides the region proposal features, we also use the Conv features ( conv4 ) extracted from an ImageNet pretrained ResNet-101.
Following GVD (Zhou et al., 2019), the region proposals are represented using the grounding-aware region encoding , which is the concatenation of i) region feature, ii) region-class similarity matrix, and iii) location embedding.
For region-class similarity matrix, we deﬁne a set of object classiﬁers as W c , and the region-class similarity matrix can be computed as M s = softmax ( W (cid:62) c R ) , which captures the similarity between regions and object classes.
We omit the ReLU and Dropout layer after the linear embedding layer for clarity.
We initialize W c using the weight from the last linear layer of an object classiﬁers pre-trained on the Visual Genome dataset (Krishna et al., 2017).
For location embedding, we use 4 values for the normalized spatial location.
The 4-D feature is then projected to a d s = 300 -D location embedding for all the regions.
Software and hardware conﬁguration.
Our code is implemented in PyTorch.
All experiments were ran on the 1080Ti, 2080Ti, and Titan Xp GPUs.
Network architecture.
The embedding dimension for encoding the sentences is 512.
We use a dropout layer with ratio 0.5 after the embedding layer.
The hidden state size of the Attention and Language LSTM are 1024.
The dimension of other learnable matrices are: W e ∈ R
d
v × 512 ,
W a ∈ R 1024 × 512 , W aa ∈ R 512 × 1 , W o ∈ R 1024 ×
d v , W l ∈ R 512 × 512 , where the vocabulary size d v is 8639 for Flickr30k Entities and 4905 for ActivityNet-Entities.
A.5. T RAINING DETAILS .
The hyper-parameters λ
and λ 2 are set to 0.5 after hyper-parameter search between 0 and 1.
Flickr30k Entities.
Images are randomly cropped to 512 × 512 during training, and resized to 512 × 512 during inference.
Before entering the proposed cyclical training regimen, the decoder was pre-trained for about 35 epochs.
The total training epoch with the cyclical training regimen is around 80 epochs.
The total training time takes about 1 day.
ActivityNet-Entities.
Before entering the proposed cyclical training regimen, the decoder was pretrained for about 50 epochs.
The total training epoch with the cyclical training regimen is aroundepochs.
The total training time takes about 1 day.
While the model is able to correctly locate objects such as horse , rider , helmet , shirt , and hurdle , it mistakenly describes the rider as wearing a blue helmet, while it’s actually black, and with white shirt while it’s blue.