Learning from Children: Improving Image-Caption Pretraining via Curriculum (2023)

Abstract: Image-caption pretraining has been quite successfully used for downstream vision tasks like zero-shot image classification and object detection. However, image-caption pretraining is still a hard problem – it requires multiple concepts (nouns) from captions to be aligned to several objects in images. To tackle this problem, we go to the roots – the best learner, children. We take inspiration from cognitive science studies dealing with children’s language learning to propose a curriculum learning framework. The learning begins with easy-to-align image caption pairs containing one concept per caption. The difficulty is progressively increased with each new phase by adding one more concept per caption. Correspondingly, the knowledge acquired in each learning phase is utilized in subsequent phases to effectively constrain the learning problem to aligning one new concept-object pair in each phase. We show that this learning strategy improves over vanilla image-caption training in various settings – pretraining from scratch, using a pretrained image or/and pretrained text encoder, low data regime etc.

## introduction recently, there has been a tremendous interest in employing image-caption pretraining for downstream vision tasks like zero-shot object classification @xcite and zero-shot object detection @xcite . the idea is to learn a common semantic space where the visual embeddings of objects in images lie close to the textual embeddings of the concepts (objects' name/tag/label) in captions they refer to. this learned semantic space is later exploited for zero-shot object recognition by finding the concept embedding nearest to the objects' embeddings. a person standing near a horse and a dog a person rides on top of a horse. horse a person standing near a horse and a dog a person that is dressed up very nicely. person dog person horse person difficult to associate three unknown concepts to three visual objects. multiple concepts in a caption with multiple objects in an image, as shown in fig. 1 . different methods have tried to solve this problem from various angles -clip @xcite and align @xcite by using more data, albef @xcite by using more complex network architecture, florence @xcite and coca @xcite by using more tasks and ernie-vil 2.0 @xcite by using more data augmentations (views). we propose an alternative approach based on a novel learning strategy that is architecture agnostic and does not require any additional data or compute. we take inspiration from cognitive science research studying how children learn language (concepts) in early stages by just observing their surroundings (images). specifically, we refer to two studies showing that childern learn rapidly if the object of interest is unambiguous @xcite and by applying co-referential statistics across multiple scenes @xcite . we implement these two ideas via a curriculum learning approach (demonstrated in fig. 1 ): these two strategies effectively reduce the problem of aligning multiple object-concept pairs per training sample to aligning only one such pair. to the best of our knowledge, no prior work has applied curriculum leaarning to image-caption pretraining in this way. @xcite apply a curriculum based on the difficulty of negative samples in contrastive loss. whereas, @xcite design the curriculum based on the granularity of text: from words to phrases to sentences. although our proposed approach can be applied to any multimodal network architecture, we pick ovr-cnn @xcite due to its simplicity. we pretrain it with the proposed curriculum learning approach and evaluate on the downstream task of zero-shot object detection. we demonstrate that curriculum learning outperforms vanilla imagecaption pretraining on a variety of architectural settings -with and without a pretrained image encoder and/or a pretrained text encoder. we even show superior performance in low-data settings, suggesting our method can be leveraged in low-resource scenarios as well. ## method we propose a curriculum learning framework to improve image caption pretraining. in this work, we apply it to ovr-cnn as its architecture is simpler and easier/faster to train/evaluate. we begin the description of our approach with a brief background on ovr-cnn. next, we discuss how we modify it to implement the proposed curriculum learning framework. ## conclusion we proposed a curriculum learning framework to improve image-caption pretraining, using the number of concepts in captions. we also designed a novel curriculum aware loss to focus learning on the unaligned concept in each phase. our approach outperforms vanilla image-caption pretraining in various settings, including with/without pretrained encoders and small data. further, we extensively analysed our model to study the contribution of each component.