Exploring Curriculum Learning for Vision-Language Tasks: A Study on Small-Scale Multimodal Training (2024)

Abstract: For specialized domains, there is often not a wealth of data with which to train large machine learning models. In such limited data / compute settings, various methods exist aiming to \textit{do more with less} , such as finetuning from a pretrained model, modulating difficulty levels as data are presented to a model (curriculum learning), and considering the role of model type / size. Approaches to efficient \textit{machine} learning also take inspiration from \textit{human} learning by considering use cases where machine learning systems have access to approximately the same number of words experienced by a 13 year old child (100M words). We investigate the role of 3 primary variables in a limited data regime as part of the multimodal track of the BabyLM challenge. We contrast: (i) curriculum learning, (ii), pretraining (with text-only data), (iii) model type. We modulate these variables and assess them on two types of tasks: (a) multimodal (text+image), and (b) unimodal (text-only) tasks. We find that curriculum learning benefits multimodal evaluations over non-curriclum learning models, particularly when combining text-only pretraining. On text-only tasks, curriculum learning appears to help models with smaller trainable parameter counts. We suggest possible reasons based on architectural differences and training designs as to why one might observe such results.

## introduction recent vision-language models (vlms) have achieved superior performance on numerous benchmark datasets (such as the llama foot_0 and gemini models foot_1 ), and continue advancing rapidly as models are scaled up. the number of parameters of such models is often on the order of billions. these models require multiple days of compute, and hundreds of gpus (e.g., @xcite ), resulting in massive energy consumption @xcite . furthermore, to train such large models, we require massive amounts of pretraining data. for example, 70m image-text pairs were used to train the flava foundation model @xcite . pretraining vlms on such large scale data is often infeasible for independent researchers and university research labs with limited compute. in contrast to machine learning, human learning is much more efficient, a finding which has led researchers to consider which methods might promote more human-like learning in artificial neural networks. this was originally argued for in early work on curriculum learning @xcite , citing the fact that humans do not learn from randomly sampled data, but benefit from learning over structured chunks, typically increasing in difficulty (a curriculum). to this end, we explore the application of curriculum learning to vlms with limited input data as part of the babylm challenge @xcite . for the multimodal track, which contains a dataset of image-caption pairs, we take inspiration from phase-based curriculum methodology used in @xcite . we use part-of-speech (pos) linguistic features from the captions to categorize samples into different phases, to generate a learning curriculum. however, instead of training the model only one phase at a time (as used in @xcite ), we train the model on the current and previous phases such that the pool of data which can be sampled increases at each phase. from our experiments, we observe that: ## training and evaluation details training details: for the curriculum variants, we train the model for two epochs per each difficulty phase (of which there are four). we used a learning rate of 1e -5 , maximum token length of 50, and 32 samples per batch foot_8 , and adam optimizer 10 (kingma and ba, 2017). when training the t+c variants of our baseline and curriculum models, we first trained the model on the text-only dataset for twenty epochs (instead of eight epochs for image-caption data) and use the same hyperparameter values. we used an nvidia a5000 gpu with 24gb vram, with half-precision (fp16) to train the models. we provide the total time required to train each model variant in appendix a. for all experiments, we set the random seed to 0 to remove variation in the results due to different random sampling and initialization. we also hold out 5% of the full image-caption dataset to validate the model. we show the validation loss curves in appendix b. evaluation: to evaluate the performance of our models, we use the evaluation pipeline provided by challenge @xcite . we report the performance of all the variants of the git and f lamingo models on the multimodal, and text-based evaluation tasks. ## results as unimodal and multimodal tasks are qualitatively different, we analyze the three experimental vari-ables of interest (curriculum, pretraining & model type) in the context of each task type. namely, we report the results for all variants of git and f lamingo models across two main task types that differ with respect to their data inputs: (i) multimodal (image+captions), and (i) unimodal (textonly).