HNC : Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities (2023)

Abstract: Image-Text-Matching (ITM) is one of the defacto methods of learning generalized representations from a large corpus in Vision and Language (VL). However, due to the weak association between the web-collected image–text pairs, models fail to show fine-grained understanding of the combined semantics of these modalities. To this end, we propose Hard Negative Captions (HNC): an automatically created dataset containing foiled hard negative captions for ITM training towards achieving fine-grained cross-modal comprehension in VL. Additionally, we provide a challenging manually-created test set for benchmarking models on a fine-grained cross-modal mismatch with varying levels of compositional complexity. Our results show the effectiveness of training on HNC by improving the models’ zero-shot capabilities in detecting mismatches on diagnostic tasks and performing robustly under noisy visual input scenarios. Also, we demonstrate that HNC models yield a comparable or better initialization for fine-tuning. Our code and data are publicly available.

## introduction pre-trained vision and language models (vlms) @xcite @xcite , when fine-tuned on downstream tasks, show promising performance thanks to their learned generalized information (or even knowledge) @xcite @xcite . these models are typically trained on a combination of several datasets under self-supervised training objectives, such as image-text-matching (itm), masked language modeling (mlm), and masked region modeling (mrm). itm defines the objective of predicting whether the textual and visual modalities entail one another. to learn this entailment, for already weakly-associated image-caption pairs, the negative captions are typically sampled from mini-batch training data which results in negative captions that do not align with the image, i.e., the mismatch between the modalities can be detected easily since the images and captions are semantically unrelated. consequently, the compositional understanding capabilities of vlms are rather limited, e.g., they tend to show weaknesses in correctly grounding linguistic concepts in their visual counterparts @xcite @xcite . these vlms, when tested against foiled inputs, fail against fine-grained mismatches in multimodal data (vision and language) @xcite . to address the aforementioned limitations we focus on improving vlms by automatically creating a dataset that enables learning from hard negative captions, i.e., negative captions that are minimally contradictory to their corresponding images. we state the hypothesis that such hard negative captions increase the general comprehension capabilities of pre-trained vlms. we summarize our contributions as follows: our method is resource-lean in constructing the hard negatives, and flexible in that it can be extended to other phenomena which is necessary for this fast-developing vision and language (vl) research field. ## related work probing vlms for fine-grained visual grounding several works revealed shortfalls in visual grounding capabilities of vlms at various levels by creating foiled visual descriptions in which they alter the nouns @xcite , words belonging to other part-of-speech (pos) tags such as adjectives or adverbs @xcite , s(ubject)-v(erb)-o(object) triples (hendricks and nematzadeh, 2021), person entities @xcite . these studies collectively suggest that vlms struggle with fine-grained imagecaption matching. moreover, several works studied the compositional understanding of vlms in visual grounding. @xcite propose winoground to evaluate visual grounding robustness using captions with the same set of words but different syntactic structures. their findings suggest that vlms exhibit bag-of-words behavior @xcite . @xcite introduce compositional visual reasoning (covr) to examine models' compositional generalization on unseen logical operations, e.g., quantifiers or aggregations, and conclude that reasoning over complex structures remains challenging. while above works aim to create probing datasets to identify vlms' potential shortfalls in visual grounding, our research goal goes beyond that: we propose a creation method for large-scale itm datasets, useful for further pretraining (or fine-tuning) models towards fine-grained cross-modal comprehension abilities. addressing shortfalls in fine-grained visual grounding capabilities of vlms given that vlms are usually pre-trained with web-crawled weakly-aligned image-caption pairs, e.g., conceptual captions @xcite , their ability to address cross-modal misalignments is questionable. the aforementioned empirical probes support this claim and suggest that vlms tend to suffer from overprediction in that they consider a somewhat related image-caption pair to be associated. previous works address this issue as a part of the training strategy @xcite @xcite , the model architecture @xcite , or by augmenting training data @xcite @xcite . we contribute to the last line of research and propose to augment hard negative captions for itm training by leveraging scene graphs towards achieving a fine-grained vl comprehension. ## hnc: hard negative captions we use the structural information provided by scene graphs @xcite to automatically generate hard negative image-text pairs with various caption types. we leverage the ground-truth scene graphs provided by the gqa (hudson and manning, 2019) dataset, which contains a total of +80k images paired with scene graphs in the training and validation set. we define a positive caption as a textual description that locally describes an image, i.e., the caption describes a part of the image and does not aim to provide an exhaustive description of the entire scene. a hard negative caption, in turn, is minimally contradictory to the image and is obtained by altering a piece of information in the corresponding positive caption, i.e., without that minimal change, it would be a positive caption. ## human-annotated challenge set as we rely on scene graphs and an automatic generation procedure to create our training and validation data, we believe in the importance of providing a quality test set ideally free from any noise introduced by our automatic procedure. to this end, we had 19 annotators foot_2 to write down pairs of captions for all caption types. annotation guidelines for each image, the annotators were asked to provide a positive and a negative caption pair per their assigned caption type(s). we set the following conditions for the annotation: 1) stay true to the vocabulary: the words in the captions must come from within the global gqa vocabulary. 2) choose visually challenging objects: the objects introduced as the foiled information in the captions must come from within the scene. 3) chose linguistically challenging attributes and predicates: the attributes and predicates introduced as the foiled information in the negative captions must be linguistically challenging, e.g., brown @xmath0; meaning that both captions are equally plausible. the annotators were instructed to skip creating a caption pair for the respective type in cases where at least one of the negative or positive captions cannot be created for a given image. dataset statistics in total, we obtain captions for 100 images. with 12 caption types, annotation results in 3201 captions with an average length of 8.42. per caption type, we get 32 captions on average. the annotated captions went under a quality check performed by another group that did not take part in the annotation. ## experiments we use the visiolinguistic transformer architectures (volta) framework @xcite as a unified testing suite to run our experiments. specifically, we use its controlled setup foot_3 and initialize all five models from the pre-trained weights provided by volta. we then further train the itm head on the training set of both hnc and foil. for a fair comparison with foil, which is substantially smaller (197k data points in the training split); in addition to the full-data setting (hnc full ), we include an hnc subset setting subsampled to 197k data points. we experiment with both single-stream and dual-stream architectures and analyze their performance difference (if any): uniter, visual-bert, vilbert, lxmert, vl-bert (tan and bansal, 2019; chen et al., 2020b; lu et al., 2019; li et al., 2019; su et al., 2020) 6 . to test whether training on hnc yields similar results on more recent and bigger models, we include experiments with blip @xcite , which are presented in a.1.2. evaluation we compare the performances of the models before and after further pre-training on hnc on two types of tasks: (1) linguistic comprehension tasks, and (2) real-world downstream reasoning tasks (sec. 5.1 and 5.2, resp.). the hnc subset results are averaged over five randomly sub-sampled splits, while the rest of the results come from a single run. ## results we report the results, i.e. classification accuracies, on the aforementioned four tasks 7 . we compare dual-stream and single-stream models to assess the effects of different modality integration methods on models' ability to detect mismatches. we display the results obtained from our further pre-trained 7 we only discuss the statistically significant results. ## dataset analysis next, we analyze our caption generation process: how robust are the different negative sampling strategies, and which results in less/more linguistic bias that a model could exploit as a shortcut? we discuss the challenges of automatic hard negative caption generation, the biases introduced in captions as a result of this automatic procedure, and how to mitigate them. we then perform a modality ablation study to ensure the quality of our humanannotated test set. we provide further qualitative analyses in appendix a.3. ## conclusion in this work, we introduced hard negative captions (hnc), a dataset for further pre-training vision and language models to improve their modality integration capabilities on a fine-grained level and demonstrated improvements across models and tasks. we proposed a novel automatic dataset construction procedure for constructing hard negative captions to be used for image-text-matching (itm) training as well as a challenging test set annotated by humans. we provided detailed analyses of the challenges in automatic creation of hard negative captions and proposed methods to mitigate them. lastly, we demonstrated the benefits of hnc by obtaining significant model performance gains on various tasks, including the diagnostic dataset valse, our hnc test set as well as a commonsense probing task (cpt), and down-stream performance gains after supervised fine-tuning on gqa, both of which require real-world reasoning.