Cross-Modal Similarity-Based Curriculum Learning for Image Captioning (2022)

Abstract: Image captioning models require the high-level generalization ability to describe the contents of various images in words. Most existing approaches treat the image–caption pairs equally in their training without considering the differences in their learning difficulties. Several image captioning approaches introduce curriculum learning methods that present training data with increasing levels of difficulty. However, their difficulty measurements are either based on domain-specific features or prior model training. In this paper, we propose a simple yet efficient difficulty measurement for image captioning using cross-modal similarity calculated by a pretrained vision–language model. Experiments on the COCO and Flickr30k datasets show that our proposed approach achieves superior performance and competitive convergence speed to baselines without requiring heuristics or incurring additional training costs. Moreover, the higher model performance on difficult examples and unseen data also demonstrates the generalization ability.

## introduction image captioning has been widely investigated in computer vision and language research. however, most current methods treat image-caption pairs for training indistinctively, thus neglecting the difference in terms of learning difficulty. as illustrated in figure 1 , an image is annotated with multiple references with diverse styles and complexity levels. such diversity can introduce different levels of learning difficulty, and undertrained captioning models can be misled by wrong gradients when training on the difficult data @xcite . curriculum learning (cl) has demonstrated improvements in model performance and training speed by presenting data sorted according to the learning difficulty @xcite . existing image captioning approaches using cl have drawbacks in their difficulty measurements: 1) requir-similarity: 26.0 22.4 17.9 16.2 14.0 ing domain-specific knowledge or heuristics @xcite ; 2) adding up the mono-modal difficulty scores without considering the cross-modal features @xcite ; and 3) requiring additional computational resources to train models on the target data (for the cases of bootstrapping methods) @xcite . we propose a simple yet efficient difficulty measurement using a pretrained vision-language (vl) model. most vl models are pretrained with imagetext matching tasks, which involve the calculation of the cross-modal similarity. the similarity reveals the model confidence in the image-text data relevance and a lower score indicates hardto-determine or low-quality data @xcite . as shown in figure 1 , the vl model assigns higher scores to the highly relevant image-caption pairs usually with simple images and appropriate captions, while assigns lower scores to less relevant pairs often with complex images and low-quality captions. we consider the pairs with higher scores to be easier to learn and train the captioning model with training examples presented from easy to hard. in our experiments on the coco and flickr30k datasets, models trained with our similarity-based cl achieve superior performance and convergence speed without domain-specific heuristics or additional pretraining cost. moreover, using a vl pretrained model possessing the knowledge of the target data can further improve the performance. we also evaluate the trained model on difficult examples and unseen data, and the better performance of our method demonstrates the generalization ability. last, our method brings higher improvement when applied to a smaller model, suggesting its applicability to scenarios in which fine-tuning large models is unfeasible. mono-modal difficulty measurement difficulty measurement can be classified into predefined and automatic methods. predefined methods require heuristics based on the data feature, such as the variety @xcite or number @xcite of objects in an image and the length @xcite or word rarity @xcite in a sentence, to measure the data complexity for image classification and language generation. in contrast, automatic methods usually adopt a teacher model for difficulty measurement based on cross entropy @xcite or perplexity @xcite to determine the model confidence and uncertainty. cross-modal difficulty measurement difficulty measurement for caption data requires to consider the visual and language modalities. @xcite directly added visual and textual difficulty scores measured by wasserstein distance and tf-idf respectively for ultrasound image captioning. similarly, @xcite used both domainspecific heuristics and entropy from the bootstrap-ping model trained on the target data as difficulty measurements for medical report generation. in addition, @xcite trained several bootstrapping models to evaluate generated captions with the bleu score as the image difficulty. unlike these works, we propose an efficient measurement based on cross-modal similarity to improve the model performance in the general domain. ## conclusion in this paper, we propose an efficient cross-modal similarity-based difficulty measurement for image captioning. our proposed simi-cl method boosts the model performance and training speed especially for larger datasets, and the pretrained models fine-tuned with target data can lead to further improvement. the improvement for data with different difficulty levels and data from other dataset indicates that simi-cl achieves the highest model generalization ability. we also apply simi-cl to different model architectures, and the higher improvement for the simpler model shows its practicality when only small-size models can be implemented in real-world scenarios.