Retrieve, Caption, Generate: Visual Grounding for Enhancing Commonsense in Text Generation Models
Abstract
We investigate the use of multimodal information contained in images as an eﬀective method for enhancing the commonsense of Transformer models for text generation.
We perform experiments using BART and T5 on concept-to-text generation, speciﬁcally the task of generative commonsense reasoning, or CommonGen .
We call our approach VisCTG: Visually Grounded Concept-to-Text Generation .
VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process.
Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, ﬂuency, and speciﬁcity.
1. Introduction
Transformer-based models have seen increasing popularity for NLP tasks.
This includes
SOTA text generation models such as BART [Lewis et al., 2020] and T5 [Raﬀel et al., 2020].
Model improvements such as larger and better pretrained text generators is a large reason for performance gains.
Despite increasing attention on the commonsense reasoning capabilities of models through works like COMET [Bosselut et al., 2019], studies have shown that even large pretrained language models still struggle with commonsense tasks that humans can reason through very easily [Talmor et al., 2019].
We believe that there is commonsense information present in other modalities such as vision, beyond what is present simply in text, which can possibly be used to inject commonsense into text generation models.
In this paper, we show this is true by improving the performance of Transformer-based text generators on concept-to-text generation using visual-grounding, which we call VisCTG:
Visually-Grounded Concept-to-Text Generation.
Concept-to-text generation is a highlevel formulation of several constrained text generation and data-to-text natural language generation (NLG) tasks.
These are challenging tasks that have seen increasing interest, and involve generating natural language outputs given certain pre-conditions, e.g. speciﬁc words in the outputs, and structured or semi-structured inputs.
They typically involve converting a set of inputs into natural language text.
These inputs can normally be thought of as
[Table]
concepts , or high-level words or structures, that play an important role in the generated text.
Multimodal work has seen increasing popularity, but its exploration for constrained and data-to-text NLG has been limited [Baltrusaitis et al., 2019, Gao et al., 2020].
We investigate the task of generative commonsense reasoning, or CommonGen [Lin et al., 2020], which involves generating sentences that eﬀectively describe everyday scenarios from concepts sets, or words that must appear in the output.
CommonGen is challenging as eﬀective relational reasoning ability using commonsense knowledge is required.
Models must also possess compositional generalization ability to piece together diﬀerent concepts.
CommonGen is an eﬀective benchmark for constrained text generation and commonsense reasoning as its task formulation and evaluation methodology are rather broadly applicable.
We experiment on CommonGen using diﬀerent sizes of BART and T5.
An initial qualitative analysis ( § 3.1) of baseline generations shows several issues related to commonsense, speciﬁcity, and ﬂuency.
We hypothesize that these issues can be addressed through using images and their captions ( § 3.2).
Images representing everyday scenarios are quite commonplace, and are typically logical and grounded in commonsense.
Further, image captioning models can normally produce decent captions for everyday images, which can be used to guide and enhance the generation process.
See Table 1 for examples.
Expounding on this, we use a pretrained image captioning model on MSCOCO captions [Lin et al., 2015] to caption the top retrieved images for each concept set ( § 4.1,4.2).
We use these captions as additional information to augment inputs to our generation models ( § 4.3).
Extensive evaluation ( § 6) demonstrates that VisCTG improves model performance while addressing baseline inadequacies with regards to commonsense, speciﬁcity, and ﬂuency.
2. Dataset, Models, and Metrics 2.1 CommonGen Dataset
The original CommonGen dataset is made up of 35,141 concept sets (consisting of 3 tokeywords each) and 79,051 sentences, split into train, dev, and test splits.
Since the original test set is hidden, we partition the original dev set into new dev and test splits for the majority of our experiments.
We do, however, ask the CommonGen authors to evaluate our best VisCTG models on the original test set (more in § 6).
The training set remains the same.
We refer to the original dev and test sets as dev O and test O , and these new splits as train CG , dev CG , and test CG .
Table 2 contains information about these splits.
Their relative sizes and distribution of concept set sizes within each are kept similar to the originals.
2.2 Models: T5 and BART
[Table]
We use pretrained text generation models T5 and BART, both base and large versions.
Both are seq2seq Transformer models.
T5 has strong multitask pretraining.
BART is pretrained as a denoising autoencoder to reproduce original from noised text.
Their HuggingFace implementations are used for our experiments.
We train two seeded versions of each model on train CG and evaluate their performance on dev O .
These serve as the baselines for our experiments.
Using the numbers in Lin et al.
[2020] as comparison, we validate our implementations.
We use the hyperparameters from
Lin et al.
[2020], beam search for decoding, and select the ﬁnal epoch as the one reaching maximum ROUGE-
[Lin and Hovy, 2003] on the dev split.
From Table 3, we observe that our re-implementations reach or exceed reported results in Lin et al.
[2020] on most metrics.
[Table] [Table]
2.3 Evaluation Metrics
We use several evaluation metrics, including those in Lin et al.
[2020] such as BLEU [Papineni et al., 2002], CIDEr [Vedantam et al., 2015], SPICE [Anderson et al., 2016]
[Table]
and coverage (cov).
These (other than cov) assess token-level similarity between human references and generations.
Cov measures the average percentage of concepts covered by the generations.
We also use BERTScore [Zhang et al., 2019] and Perplexity (PPL).
BERTScore measures
BERT [Devlin et al., 2019] embeddings similarity between individual tokens, serving as a more semantic-level similarity measure.
We multiply by 100 when reporting BERTScore.
Concept Set
Generated Output
Human Reference { horse, carriage, draw } horse drawn in a carriage The carriage is drawn by the horse. { bathtub, bath, dog, give } A dog giving a bath in a bathtub.
The teenager made a big mess in the bathtub giving her dog a bath.
{ dog, house, eat } A dog eats hay in a house The dog eats food inside the house. { cow, horse, lasso }
A cow is lassoing a horse.
A group of men riding horses lassoing a cow.
PPL serves as a measure of ﬂuency, with lower values representing higher ﬂuency.
We use
GPT-2 [Radford et al., 2019] for PPL.
For all other metrics, higher means better performance.
3. Initial Analysis and Motivation 3.1 Baseline Model Generations
We conduct an initial analysis of the baseline model outputs, and observe that several outputs lack ﬂuency.
Some are more like phrases than full coherent sentences, e.g. “body of water on a raft” .
Others are missing important words, e.g. “A listening music and dancing in a dark room” is clearly missing a noun before listening .
A large portion of generated texts are generic and bland, e.g. “Someone sits and listens to someone talk” .
This may be an instance of the dull response problem faced by generation models [Du and Black, 2019, Li et al., 2015], where they prefer safe, short, and frequent responses independent of the input.
Many generations also appear to lack commonsense.
For example, “body of water on a raft” is illogical as the phrases “body of water” and “a raft” are pieced together incorrectly.
A similar issue occurs with the { horse, carriage, draw } example in Table 4.
Further, at times the models cannot understand what certain nouns can or cannot do, e.g. “A dog checking his phone on a pier.”
Several other examples of this can be found in Table 4.
3.2 Images and Captions
Images that represent everyday scenarios are quite prevalent for almost any reasonable input concept set.
Further, the images are typically grounded in commonsense.
For example, searching { cow, horse, lasso } will result in many images of cowboys riding horses and lassoing cows, rather than the illogical situation of “A cow is lassoing a horse.”
described by the baseline generation in Table 4.
Many everyday images are relatively similar to those in image captioning datasets such as MSCOCO, so pretrained image captioning models should work quite eﬀectively.
We thus hypothesize that using images and their captions to visually-ground concept-to-text generation can potentially deal with issues mentioned in 3.1.
Retrieved images with corresponding captions generated by a pretrained image captioning model (see § 4.2) and ﬁnal generations for select input concept sets can be found in Table 1.
Textual corpora also suﬀer from reporting bias [Gordon and Van Durme, 2013], where everyday, commonsense albeit “uninteresting” actions (walking), objects (bench) and facts (bananas are yellow) are underrepresented compared to real-world frequency, while “newsworthy” actions (murdering), objects (spaceships) and facts (blue GMO bananas) are exaggerated.
This also seeps into large pretrained text models [Shwartz and Choi, 2020].
Using visual data and models dampens this bias, likely improving commonsense of generations.
4. Methodology 4.1 Image Retrieval
We ﬁrst obtain images for each concept set in the three splits.
Image captioning datasets such as MSCOCO and Flickr are typically too small and focused to be eﬀective since we must cover numerous diﬀerent concept sets.
Further, a search engine is more generalizable.
We decide to use Google Images.
On a sample of concept sets, the retrieved images using other search engines were inappropriate; they did not incorporate most input keywords nor handle homonyms well.
For example, “sports+fan+watch” yields images of fans watching a sports game on Google images, but images of hand watches on Bing and DuckDuckGo.
We queried input concept sets by concatenating keywords with plus signs (+), and used simple-image-scraper 1 to obtain URLs of the top 30 resulting images.
The image was scraped only if the URL ended in .png , .jpeg , .jpg , or .gif .
Finally, the received content was veriﬁed to be valid images using pillow 1 , and skipped otherwise.
Retrieved images were typically of high quality and corresponded well to the input concepts.
See Table 1 for examples.
4.2 Image Captioning
After retrieving images, we use a PyTorch-based implementation 2 of the FC image captioning model [Luo et al., 2018, Rennie et al., 2017], which generates a caption sequence via an LSTM model.
The LSTM is initialized with a pseudo token, obtained by feeding the input image into a deep CNN followed by a linear projection.
For our experiments, we use a pretrained
FC model trained on the MSCOCO dataset with pretrained Resnet-101 image features.
As most of our retrieved images represent everyday scenarios and are relatively similar to those in MSCOCO, the pre-trained model performs quite well (see example captions in Table 1).
4.3 Caption Selection and Input Augmentation
[Table]
[Figure]
age
[Figure]
[Table]
[Figure]
DevCG
TestCG
After we have captions S c
= { c 1 , c 2 , ..., c n } for each concept set in all three splits, we reorder them by descending coverage to the concept set to obtain S c (cid:48)
= { c (cid:48) 1 , c (cid:48) 2 , ..., c (cid:48) n } .
If two captions are tied for coverage, we keep them in their original search result order.
This allows us to select the captions that have highest coverage and are most relevant.
Since most retrieved images and corresponding captions cover only a fraction of the entire concept set, and the quality of each varies, we hypothesize that using multiple captions for generation may lead to more robust and higher-quality outputs with more coverage.
The models may learn to piece together information from caption(s) while generating ﬁnal texts.
Hence, we try experiments using diﬀerent numbers of top captions within S c (cid:48) , a parameter we call NT C (Number of
[Table]
Top Captions).
We try NT C = 1 , 2 , 3 , 5 , 7 , 10, and do not go above NT C = 10 as Figureshows that coverage gains from 10 → 30 are minor.
The captions are concatenated together and onto the input concept set using < s > separator tokens.
These serve as augmented inputs to the BART and T5 models.
The models learn to convert these augmented inputs to human references during training, and are fed the augmented inputs (corresponding to the value of NTC) during validation and testing.
Some examples of augmented inputs and ﬁnal generations can be found in Table 5.
5. Experiments 5.1 Model Training and Selection
For training VisCTG models, we mainly follow baseline hyperparameters, barring learning rate (LR) which is tuned per NTC value, and the maximum encoder length which is chosen depending on the tokenizer and value of NTC to ensure the entire input token sequence can ﬁt onto the encoder.
We train two seeds per model.
See Appendix B for more details.
We choose the epoch corresponding to highest ROUGE-2 on dev CG , and use beam search for decoding.
NTC itself is a hyperparameter, so while we train separate versions of each model corresponding to diﬀerent NTC values, the ﬁnal chosen models correspond to the
NTC values that performed best on dev CG when averaged over both seeds.
We then use the ﬁnal chosen models to generate on both test CG and test O , and report the results in § 6.
5.2 Human Evaluation
We ask annotators to evaluate 72 test CG examples, containing the VisCTG and baseline outputs using BART-large and T5-base.
These two are chosen as they cover both model types and sizes.
See Appendix § C for further details.
We ask annotators to evaluate the text’s ﬂuency and commonsense on scales of 1-5.
Fluency is a measure of how grammatical, natural, and human-like the text is.
Commonsense is the plausibility of the events described by the text.
Coverage is more objective compared to ﬂuency and commonsense and we do not evaluate it as the automatic metric suﬃces.
6. Results and Analysis
Automatic evaluation results on test CG are in Tables 6 and 7, and results on test O are in
Table 8.
3 Graphs displaying BLEU-4, CIDEr, and SPICE (the metrics on the CommonGen
[Table] [Table]
leaderboard 4 ) on test CG over diﬀerent values of NTC are in Figure 2.
Human evaluation results on test CG are in Table 9.
Optimal NTC values for BART-base, BART-large, T5-base, and T5-large are 5, 2, 2, and 1, respectively.
These are the VisCTG results reported in the aforementioned tables.
Table 10 contains qualitative examples, with more in Appendix § D.
6.1 Analysis of Evaluation Results
We see from Tables 6 and 7 that VisCTG outperforms the baselines on all metrics across the models on test CG .
Performance gains are strong and statistically signiﬁcant for BART-base,
BART-large, and T5-base.
VisCTG appears relatively less eﬀective for T5-large which is the strongest baseline, and hence further improving its performance may be more diﬃcult.
Table 9 shows that the BART-large and T5-base VisCTG models noticeably outperform their respective baselines in both ﬂuency and commonsense, as rated by human annotators.
[Table]
[Figure] [Figure] [Table]
[Figure]
From Table
[Figure]
we see that VisCTG models substantially outperform corresponding baselines reported in Lin et al.
[2020] on test O .
T5-base VisCTG outperforms the reported T5-base and large baselines across metrics [Figure] and BARTbase VisCTG performs similarly to the reported BART-large baseline.
BART-large VisCTG outperforms the reported baseline, EKI-BART [Fan et al., 2020], and KG-BART [Liu et al., 2020].
These are SOTA published CommonGen BART models that use external knowledge from corpora and KGs.
We show that visual-grounding is more eﬀective, and BART-large
VisCTG would place very high on the leaderboard.
4 T5-large VisCTG outperforms the reported baseline, but lags behind the SOTA published RE-T5 [Wang et al., 2021].
Figure 2 shows that as NTC increases, BLEU-4, CIDEr, and SPICE increase to a peak, and taper oﬀ after.
This is expected as we saw in Figure 1 that the rate of increase of coverage declines with larger NTC.
The latter images and captions are of diminishing quality, and hence using too many negatively aﬀects model performance.
6.2 Qualitative Analysis
Table 10 shows several baseline outputs that contain issues from § 3.1, e.g. incomplete/illogical sentences.
Human references are all ﬂuent and logical.
VisCTG can usually generate much better text than the baselines, addressing issues with ﬂuency, commonsense, and speciﬁcity.
The baseline outputs for ex. 1-2 are phrases lacking arguments, and are all illogical for ex.
Using captions, VisCTG successfully adjusts semantic roles of entities, replaces incorrect subjects, ﬁxes dependency structure, and grounds generations in commonsense.
For ex. 1, the captions are of the form “ { X } sitting on a chair with { Y } ”, where { X } is a subject and { Y } is an object.
VisCTG output has a similar structure, being ﬂuent+logical with higher coverage.
For ex. 2, the baseline output treats “hand of a bird” as a single entity, the subject.
Captions separate “bird” and “hand” into two, likely guiding the VisCTG output to do so.
For ex. 3, the baseline misplaces “bus” as subject.
Captions are of form “ { X } sitting on a bench { Y } ”, where { X } is a logical subject and { Y } is an expression.
The VisCTG output has this structure, with correct subject and commonsense, and higher coverage.
For ex. 4, the baseline output lacks a subject that the captions both contain, likely guiding the VisCTG output to contain one: “a man” .
For ex. 5, the baseline output is generic due to uses of “someone” .
VisCTG’s output is more speciﬁc and refers to “man” , likely because the caption (although not very ﬁtting) includes a “man” subject.
Even for captions that ﬁt the concepts less, the structure and ﬂuency can still potentially be exploited.
VisCTG is imperfect.
For ex. 6, its output is less logical and lower coverage than the baseline’s.
The captions are all simplistic and low coverage; the ﬁrst is illogical, and some others are of the form “a bunch of apples { ... } on a tree” , likely negatively impacting the generation.
Ex. 4’s human reference is creative, which is an area where VisCTG still lacks in comparison.
For ex. 5, while VisCTG edits “someone” to “man” , it is unable to merge the two instances of “man” or adjust the sentence to be more coherent.
These weaknesses are likely because captions tend to be simplistic (due to the captioning model’s training data), limiting VisCTG’s ability to make heavier edits.
VisCTG, unsurprisingly, appears to depend quite heavily on the captions, and hence quality of the images and captioning model.
7. Related Work
Constrained Text Generation:
There have been several works that investigate constrained text generation.
Miao et al.
[2019] use Metropolis-Hastings sampling to determine
Levenshtein edits per generation step, and show gains on several tasks.
Feng et al.
[2019] propose Semantic Text Exchange to adjust text semantics given a replacement entity .
Data-to-text NLG:
E2E-NLG [Duˇsek et al., 2018] and WebNLG [Gardent et al., 2017] are two popular NLG benchmarks with structured inputs - meaning representation (MR) and triple sequences, respectively.
Montella et al.
[2020] use Wiki sentences with parsed
OpenIE triples as weak supervision for WebNLG.
Tandon et al.
[2018] permute input MRs to augment examples for E2E-NLG.
Kedzie and McKeown [2019] inject Gaussian noise into a trained decoder’s hidden states and sample augmented examples for E2E-NLG.
Commonsense Reasoning and Incorporation:
Talmor et al.
[2019] show that not all pretrained LMs can reason through commonsense tasks.
Other works investigate commonsense injection into models; one popular way being knowledge graphs (KGs).
One large commonsense KG is COMET, which trains on KG edges to learn connections between words and phrases.
COSMIC [Ghosal et al., 2020] is a model that uses COMET to inject commonsense.
EKI-BART
[Fan et al., 2020] and KG-BART [Liu et al., 2020] use external knowledge (e.g. from corpora and KGs) to improve BART’s performance on CommonGen.
Distinctly, VisCTG uses visual-grounding and shows higher performance (see § 6). SY
Multimodal Machine Learning and NLP:
There has been more work on multimodal
ML and NLP, in fundamental areas such as multimodal representation and fusion, and application areas such as speech recognition and video captioning, but little for constrained and data-to-text NLG tasks, e.g. CommonGen [Baltrusaitis et al., 2019, Gao et al., 2020].
[Table]
8. Conclusion and Future Work
In conclusion, we motivated and explored the use of visual grounding for improving the commonsense of Transformer models for text generation.
We investigated this for concept-totext generation, calling our method VisCTG: Visually Grounded Concept-to-Text Generation.
Extensive experiments on BART and T5 showed its eﬃcacy on the CommonGen task.
Comprehensive evaluation and analysis showed that VisCTG boosts model performance while addressing baseline deﬁciencies related to commonsense, ﬂuency
[Table]
and speciﬁcity.
Potential future work includes improving image search and captioning, e.g. a method for better selection of images during retrieval or using a stronger captioning model.
Video captioning and image generation rather than retrieval can also be explored.
Further, VisCTG can be investigated for other data-to-text NLG tasks such as WebNLG
[Table]
Appendix A. Pretrained FC Image Captioning Model Details
The image encoder is a pretrained Resnet-101 [Ren et al., 2015], where the global average pooling of the ﬁnal convolutional layer output, a vector of dimension 2048, is taken per image.
The spatial features are extracted from the output of a Faster R-CNN [Ren et al., 2015,
Anderson et al., 2018] with ResNet-
[He et al., 2016], trained by object and attribute annotations from Visual Genome [Krishna et al., 2016].
For captioning, the dimensions of
LSTM hidden state, image feature embedding, and word embedding are all set to 512.
Please see Luo et al.
[2018], particularly Sections 3.3 and 5.1, and Rennie et al.
[2017], particularly
Sections 2 and 5, for more.
Appendix B. BART and T5 Model Training and Generation Details
T5-large consists of 770M params, T5-base 220M params, BART-large 406M params, and
BART-base 139M params.
We train two seeded versions of each baseline model and VisCTG model.
For all models, we use beam search with a beam size of 5, decoder early stopping, a decoder length penalty of 0.6, maximum encoder lengths of 32 for the baselines and up to 160 for BART and 256 for T5 for VisCTG models, decoder maximum lengths of 32, and a decoder minimum length of 1.
For model training, we use a batch size of 64 for T5-base and
BART-base, 32 for BART-large, and 8 for T5-large.
For T5-base, T5-large, and BART-base, we use 400 warmup steps, and 500 for BART-large.
We train all models up to a reasonable number of epochs (e.g. 10 or 20) and perform early stopping using our best judgment (e.g. if metrics have continually decreased for multiple epochs).
The learning rates for VisCTG models were determined by trying a range of values (e.g. from 1e-6 to 1e-4), and ﬁnding ones which led to good convergence behavior (e.g. validation metrics increase at a decently steady rate and reach max.
after a reasonable number of epochs).
Training was done using Google Colab instances which alternately used a single V100 or
P100 GPU.
The vast majority of the training was done on a single V100 per model.
T5-base models trained in approx.
1.5 hours, BART-base models in approx.
1 hour, T5-large models in approx.
6 hours, and BART-large models in approx.
2 hours.
Appendix C. Human Evaluation Details
The human evaluation study was performed through paid annotators on AMT.
Annotators were from Anglophone countries with > 97% approval rate.
Each example was evaluated by two annotators.
Speciﬁc instructions and a question snippet can be seen in Figure 3.
Note that T5-base and BART-large studies were conducted separately, so results across both model types are not relatively comparable.
The main purpose of the evaluation is to show the relative improvement between the VisCTG models over the corresponding baselines.
The time given for each AMT task instance or HIT was 8 minutes.
Suﬃcient time to read the instructions, as calibrated by authors, was also considered in the maximum time limit for performing each HIT/task.
Annotators were paid 98 cents per HIT.
The rate of payment (7.35 $ /hour) exceeded the minimum wage rate for the USA (7.2 $ /hour) and hence constitutes fair pay.
We neither solicit, record, request or predict any personal information pertaining to the AMT crowdworkers, during the studies.
[Figure]
Appendix D. Further Qualitative Examples
See Table 11 for further qualitative examples.
[Table]
Human
[Table]
Captions a dog is standing in the grass with a frisbee < s > a dog is
[Table]
with a frisbee in the
[Table]
owner throws a ball to his dog during a chase
[Table]