Targeted Image Data Augmentation Increases Basic Skills Captioning Robustness (2023)

Abstract: Artificial neural networks typically struggle in generalizing to out-of-context examples. One reason for this limitation is caused by having datasets that incorporate only partial information regarding the potential correlational structure of the world. In this work, we propose TIDA (Targeted Image-editing Data Augmentation), a targeted data augmentation method focused on improving models’ human-like abilities (e.g., gender recognition) by filling the correlational structure gap using a text-to-image generative model. More specifically, TIDA identifies specific skills in captions describing images (e.g., the presence of a specific gender in the image), changes the caption (e.g., “woman” to “man”), and then uses a text-to-image model to edit the image in order to match the novel caption (e.g., uniquely changing a woman to a man while maintaining the context identical). Based on the Flickr30K benchmark, we show that, compared with the original data set, a TIDA-enhanced dataset related to gender, color, and counting abilities induces better performance in several image captioning metrics. Furthermore, on top of relying on the classical BLEU metric, we conduct a fine-grained analysis of the improvements of our models against the baseline in different ways. We compared text-to-image generative models and found different behaviors of the image captioning models in terms of encoding visual encoding and textual decoding.

## introduction humans and animals develop all kinds of cognitive abilities from a very early age that allow them to interact with their world @xcite . for instance, infants display numerical cognition abilities @xcite , can recognize emotions @xcite or even the 1 code will be available online after submission. danger associated with other agents' action plans @xcite . comparatively, animals also display similar numerical cognition abilities @xcite , or recognize emotions in order to better communicate within a social group @xcite . these abilities are crucial in order to build models of the world that are necessary for planning, reasoning, and solving complex decision-making tasks @xcite . deep learning systems can solve these tasks by optimizing an objective function via supervised, semi-supervised or unsupervised learning @xcite . within this framework, it has been shown that deeper layers progressively represent increasingly abstract concepts @xcite , akin to what has been observed in the human visual or auditive processing pathways @xcite . moreover, empirical work has shown that pretrained stateof-the-art transformer models @xcite encode factual knowledge within sets of knowledge neurons @xcite ; strongly related to the concepts of "grandmother" cells in neuroscience @xcite . importantly, not only factual knowledge but also conceptual knowledge (such as "sentiment" in a text or "written language" in an image) are encoded by nodes in deep layers @xcite . whereas recent methods have been proposed to access and edit factual knowledge @xcite , and thus evaluate how and where facts are being encoded in deep networks @xcite , it is much harder to evaluate the abilities associated with conceptual knowledge stored in these networks. yet, possessing such a conceptual knowledge base is crucial for out-of-distribution generalization @xcite . although deep networks seem to encode conceptual knowledge that allows them to display humanlike abilities such as counting, emotion, gender, color, and sentiment recognition/categorization @xcite @xcite @xcite , these same networks typically struggle in producing out-of-context (or out-of-distribution) generalizations @xcite @xcite @xcite . these limitations are due to the inherent functioning of artificial neural networks (anns). indeed, generalization performances of anns largely depend on their ability to extract the correlational structure in the training data set, memorize this structure, and extrapolate it to a novel (test) data set @xcite . indeed, given that the performance of vanilla deep networks is constrained by the structural correlation observed in the training data set, a straightforward way to maximize the generalization performance in anns is to augment data sets in targeted ways @xcite . thereby, targeted data augmentation would increase the span of potential correlations that could be observed in the world, and as such improve the human-like abilities of deep networks. by targeting specific human-like abilities and augmenting the data set to encapsulate unseen examples associated with these abilities, we hypothesize that models can increase their conceptual knowledge, and thus improve their performance on specific benchmarks we discuss below. moreover, similar to editing unique factual knowledge @xcite , one would ideally want to target unique conceptual knowledge (e.g., gender, color, numerosity, emotion, shape...) to induce such ability-selective performance, which has been widely studied @xcite . we will propose a simple way to overcome the issues raised above, for image captioning (ic) task. interestingly, novel text-to-image generation models @xcite in combination with text-generation or manipulation @xcite @xcite affords novel possibilities for targeted data augmentation for visionlanguage tasks. hence, we propose to enhance the capabilities of an image captioning model by using a targeted data-augmentation on several specific abilities (or skills). we use simple regular expressions (regex) to identify these skills in the caption, to change the caption for another version of it, and to generate the image related to this caption. the main contributions of this work are twofold. first, we propose a simple method to identify data related to a specific human-like ability in image captioning (e.g., color identification, emotion recognition...). second, we propose a novel data augmentation method based on image-to-text generation models that allows one to generate data sets that can selectively improve a single or combinations of human-like skills in image captioning performance. instead of manipulating or fine-tuning information processing within image captioning models, our method increases the span of potential object correlations and thus allows us to generalize image captioning abilities to a broader spectrum of situations that can be observed in the real world @xcite . in what follows, we first describe related work while specifying the original contribution of our work. subsequently, we describe the targeted image-editing data augmentation (tida; see figure 1 ) method and present the results associated with fine-tuning models with our tida-augmented data sets. finally, we discuss the implications of our work. ## previous and related work image captioning image captioning (ic) models provide human-like captions to images @xcite . such an ability lies in the intersection between computer vision and natural language processing @xcite , and is therefore, in essence, a multimodal problem. early ic models proposed to sequentially combine convolutional neural networks (cnn) with recurrent neural networks (rnn) into a single imagedconditioned language model @xcite @xcite . given the success of these models and their potential industrial applications, subsequent work has focused on improving the models' image captioning ability by focusing on specific properties of ic models. for instance, it has been shown that top-down visual attention mechanisms improve captioning performance @xcite . alternatively, focusing on the learning process, it has been shown that implementing self-critical sequence training (a variant of the re-inforce algorithm) improves ic performances by avoiding the exposure bias @xcite and directly optimizing the relevant task metrics @xcite . furthermore, many ic models are pre-trained using tasks like masked language modeling (mlm) and image-text matching (itm). these tasks possess losses that differ from image captioning (or other downstream tasks), and thus ic models require further fine-tuning. hence, recent work has focused on unifying generative vision-language models through text generation @xcite , in order to optimize knowledge transfer from train to test. lastly, novel methods have focused on optimally leveraging language caption supervision during pretraining, as small datasets with large caption variability can lead to detrimental effects @xcite . symbolic knowledge vision-language (vl) tasks can also be improved by incorporating symbolic knowledge into the vl models. for instance, providing a knowledge base, instantiated as subjectrelation-object triplets associated with the images, both improve performance in vision-question answering (vqa) tasks, on top of allowing to explain the vqa model's predictions @xcite . in the same vein, adding high-level (semantic) attributes as inputs to ic models can increase captioning benchmarks @xcite . alternative efforts have shown that using object tags to facilitate the semantic image-text alignment during pre-training, and improves benchmark metrics in downstream fine-tuned image captioning tasks @xcite . moreover, aligning directional semantic and spatial relationships between text and image (i.e., relation-level alignment) improves compositional reasoning @xcite . finally, symbolic knowledge and reasoning capability aim to enhance textual model's robustness when faced with out-of-distribution examples, thereby enabling them to engage in more humanlike reasoning @xcite . bias/bug detection, and evaluation tida enhances the likelihood of simultaneously observing distinct attributes in an image within the augmented dataset. thereby, our work relates to studies that focus on improving the predictive abilities of models in domains that suffer from bias-induced incorrect predictions. in line with this idea, the equalizer model is constrained to attend to the person attribute in images, increasing the ic abilities to detect the gender in the image @xcite . interestingly, other attributes such as numeracy (e.g., counting) naturally emerge in standard embeddings @xcite , and may thus be less prone to biased predictions. alternative debiasing methods focus on "decoupling" biased directions within text embeddings @xcite . other approaches focus on discovering the specific images where ic models fail (i.e., bugs). an instance of such a method uses a sequential pipeline that generates images from specific captions, classifies the object in the image, creates captions from the incorrectly classified images, generates captions of these images, and finally regenerates novel images based on the previously generated caption via a text-to-image generative process. these last images can be used to assess the robustness of vision models, as well as improve their performance @xcite . moreover, while image captioning is usually scored on automatic metrics like spice @xcite or cider @xcite , it has been suggested that metrics evaluating both precision and recall leading to better correlations with human judgments @xcite . finally, @xcite propose a method to compare image captioning models correlated with human judgment by leveraging llm @xcite . ## method and experiments we propose a two-step method that allows retrieving certain images using their captions, regarding a specific concept that we call skill. these skills refer to human-and animal-like basic abilities, such as gender categorization, counting, or recognizing colors. we first use a text mining method to detect whether or not a caption contains specific words that are related to the skill (subsection 3.1). second, we generate variants of the original skillrelated captions and create new images with these new captions in order to augment the dataset for each type of skill (subsection 3.2). an overview of the method is shown in figure 1 . ## conclusion and future work this paper assesses the effectiveness of generative data augmentation with current diffusion models for improving specific skills of image captioning models. we use the flickr30k image captioning dataset and ran experiments with blip, a recent vision-language state-of-the-art model. we show that tida, our targeted image data-augmentation techniques allows for gains regarding classical metrics that are recognized by the community like bleu or refclipscore. on top of that, we also propose a fine-grained analysis to analyze the results of the model by going beyond the classical opaque metrics by investigating the occurrences of specific semantic words related to the target skills. we found out that tida helps the image captioning model to use those words more efficiently. finally, we investigate the visual part, we probe the representations from the visual encoder and reveal that they do not contain more information on the skill, meaning the improvement relies on the textual decoder. our results open several avenues for further research. for instance, it remains unclear why we observe the boost in results on a specific skill when using data-augmentation on another skill. it would also be useful to investigate more in details the reasons of the improvement of performances the text decoder or the visual encoder, or to use a more precised metric powered by a llm like @xcite . it would also be useful to investigate more in details the reasons of the gain of performances of the text decoder or the visual encoder, or to using complex interpretable metrics from llm like the textto-image faithfulness evaluation with question answering @xcite . it would be to see improvements with text-to-image models known to be better at generating images related to color, counting, like attend-and-excite @xcite with newer versions of stable diffusion. finally, we would like to extend our method to visual question answering. using symbolic knowledge to extract the objects of the image-caption and the relation as implemented in @xcite , we can adapt the model to new situations and help to debias a vqa model. finally, given the recent results of @xcite , we should run a random data-augmentation on the train set and see whether this procedure may help to improve the results compared with tida. in terms of computational cost, tida's necessity to generate a number of new examples comparable to the original dataset size using costly neural image generation models signifies it is a challenge to apply to larger datasets and that the technique doesn't scale well to dataset size. and although each generated example can be leveraged many times, the process is heavily limited by the computation capabilities.