The Role of Data Curation in Image Captioning (2024)

Abstract: Image captioning models are typically trained by treating all samples equally, neglecting to account for mismatched or otherwise difficult data points. In contrast, recent work has shown the effectiveness of training models by scheduling the data using curriculum learning strategies. This paper contributes to this direction by actively curating difficult samples in datasets without increasing the total number of samples. We explore the effect of using three data curation methods within the training process: complete removal of an sample, caption replacement, or image replacement via a text-to-image generation model. Experiments on the Flickr30K and COCO datasets with the BLIP and BEiT-3 models demonstrate that these curation methods do indeed yield improved image captioning models, underscoring their efficacy.

## introduction image captioning is the task of generating grammatically correct and accurate descriptions of visual data, which involves understanding the identity of salient objects and their relationships @xcite . while existing models have made significant progress on this problem, there remains an inherent challenge: how to address the variations in learning difficulty that arise from diverse image-caption pairs @xcite . image captioning models are usually trained by treating the entire training dataset equally, which overlooks the variations in the complexity of each data point. one attempt at addressing this issue has been to apply data filtering as a preprocessing stage to large-scale datasets to remove noisy data from the pretraining process @xcite . several other image captioning techniques have relied on curriculum learning strategies @xcite , which schedule the training data with increased levels of complexity, effectively adapting the learning process to the difficulty of the task @xcite @xcite @xcite . in this paper, we aim to answer a fundamental question: can image captioning models be improved by not only recognizing variations in the data but also actively curating difficult samples? we introduce three data curation methods, each with the aim of improving the learning process while preserving the overall size of the training dataset. these methods include the complete removal of a sample, the replacement of captions, or the substitution of images using a text-to-image generation model. the targets of these methods are image-caption training samples that have unusually high losses with respect to the rest of the training dataset under the current model parameters. in other words, our approach focuses on the samples that are proving difficult to model @xcite . the main findings of this paper are: ## data curation for captioning our main goal is to assess whether actively curating image-caption pairs during training can improve image captioning models. there are many reasons for the existence of difficult samples, including mismatches between the image-caption or inconsistencies between the image and caption @xcite , e.g. the caption includes mentions of entities that cannot be seen in the image. for clarity in what follows, let d be an image captioning training dataset with k images, and let i k be the k-th image. each image is paired with j captions; let c j k be jth caption of image k, and thus, let (i k , c j k ) be an image-caption sample. ## results data curation improves captioning ## discussion curation amount matters the amount of data curated is an important hyperparameter. in addition to the best results reported above, we present finer-grained results of varying the amount of data curation. for remove and replacecap, we explore curating the top 1%, 5% and 10% of high-loss samples. for replaceimg, we explore 10%-80% curation ratios. in addition to fixed x% ratios, we also intereven on samples that have losses two standard deviations worse than the mean. the results of this analysis are shown in figure 3 . while the effective curation ratio for different curation approach ranges from 1%-50% for flickr30k, coco benefits from replaceimg on less than 10% of the top loss samples, and the effective curation ratio for remove and replacecap stops at 1%. this indicates that flickr30k may contain more noisy samples than the ms coco dataset. compared to ms coco, flickr30k contains more samples with long captions (figure 4 ), which may include overly-specific details that are inconsistent with other captions and are hard for the model to learn (figure 12 ). through our curation-based finetuning, these samples can be effectively iden0 10000 20000 30000 rank 10 0 10 1 10 2 10 3 10 4 10 5 10 6 log(frequency) method no curation remove replacecap replaceimg figure 6: zipfian distribution of words in flickr30k training samples for different curation approaches. note the clear changes made to the tail by remove. ## further analysis with the human study revealing the failure modes of the text-to-image model, we now provide insights on various techniques that are proved useful for improving image relevance in curating the image captioning datasets. round-trip captioning evaluation most previous work in text-to-image generation uses imageoriented measures like fid @xcite or clipscore @xcite . however, these measures are not suitable for our purpose as they are claimed to lack alignment with perceptual quality @xcite . we also found that clip-score cannot distinguish between low-and highloss samples in captioning (figure 8 ). alternatively, similar to @xcite , we use a fixed model to generate captions for synthesized images and then compare them to original captions in a three-step process (figure 9 ): (1) gen- stable diffusion text2img original validation sd validation predicted captions evaluation image captioning a person in a blue jacket is sitting against a wall covered in graffiti a person in a blue jacket is sitting against a wall covered in graffiti a person in a blue jacket is sitting against a wall covered in graffiti a person in a jacket and wearing jeans kneels down to take a picture of a graffiti-laden wall. ## conclusion in this paper, we have shown a simple, yet effective, data curation framework that can improve the performance of image captioning models. we investigated three approaches to data curation that dynamically update the training dataset based on high-loss image-caption samples. the methods involved either removing a sample, replacing the caption in a sample, or generating a new image from existing captions. experimental results on the flickr30k and ms coco datasets show the effectiveness of these approaches to data curation without increasing the total size of the training dataset. a deeper analysis of the images synthesized by the text-to-image model shows frequent errors on generating objects of a certain amount or color, and struggles with human body features. a human evaluation of the errors in those images shows a clear difference in images with high or low losses. in the future, we expect that better text-to-image generation models will lead to further improvements from using synthesized images to train image captioning models. from our insights in appendix a.4, there is also significant promise on building a hybrid model combining different curation methods. we believe that a more sophisticated learning scheme leveraging multiple methods will offer more flexibility when curating the dataset. we plan on verifying whether these findings extend to other image captioning models. moreover, we are also interested in applying the same framework to other multimodal tasks, especially those with under-complete datasets that cannot comprehensively cover the distributional space due to the cost of crowd-sourcing enough data, e.g. visual question answering, or visually-grounded dialog.