Curriculum Learning Effectively Improves Low Data VQA (2021)

Abstract: Visual question answering (VQA) models, in particular modular ones, are commonly trained on large-scale datasets to achieve state-of-the-art performance. However, such datasets are sometimes not available. Further, it has been shown that training these models on small datasets significantly reduces their accuracy. In this paper, we propose curriculum-based learning (CL) regime to increase the accuracy of VQA models trained on small datasets. Specifically, we offer three criteria to rank the samples in these datasets and propose a training strategy for each criterion. Our results show that, for small datasets, our CL approach yields more accurate results than those obtained when training with no curriculum.

## introduction visual question answering (vqa) models are commonly trained on large-scale datasets to achieve the state of the art performance @xcite @xcite . modular vqa models, in particular, require large data sets for training. these models dynamically combine a number of neural networks according to a pre-specified layout @xcite @xcite to form a new larger network that produces an answer to an input question. the layout, or program, is generated for each question on the fly. as a consequence, the architecture of the resulting network varies according to the program. combining neural networks often leads to a wide and deep network. training such a large-sized network with a varying architecture calls for a massive amount of labeled data, which is either expensive or very limited in many realistic settings. with insufficient data, a large and complex network can perform unsuccessfully. an example of this is our experience in training the vqa model by @xcite with only 20% of the clevr dataset @xcite . our results showed only 54.24% accuracy compared to the accuracy of 96.90% on the full dataset according to the authors' report @xcite . motivated by this experience, the work presented in this paper studies vqa in low data scenarios, and sheds light on the performance of current modular vqa models under data scarcity conditions. to the best of our knowledge, this is the first study to investigate vqa models in low-data regime. many approaches have been investigated to improve the performance of deep learning models when training on limited data, ranging from data augmentations @xcite and pretraining @xcite to semi-supervised learning @xcite and transfer learning @xcite . however, these works mostly deal with the scarcity of labeled data by assuming help from available unlabeled data, or by transferring knowledge from similar domains. unlike them, our goal is to train a modular vqa model from scratch by using only a small amount of labeled data without using any other resources. specifically, we take the cl approach to tackle the problem of vqa models' low performance under low data conditions. curriculum learning @xcite was introduced as a method to supervise the order in which data examples are exposed to the model. our hope is to maximize the usage of training samples by performing supervision on the order of training data that are fed into the model. the underlying idea of cl is to start learning from easy examples, and gradually consider harder ones, rather than using examples in a random sequence. to rank training examples from easy to hard, cl must define the concepts of easy and hard examples. such a ranking is a key challenge in cl. many of the ranking criteria introduced in the cl literature are problem-specific heuristics @xcite or automated measures based on model performance (hacohen and weinshall, 2019). in this paper, we propose and analyze the performance of three ranking criteria: (1) a length-based criterion, which considers longer questions as more complex than shorter questions, and ranks the examples in increasing order of their program length; in addition to the ranking heuristics, in §5, we propose a cl training strategy for each criterion. we also argue that under cl training in low data regimes, a model is very susceptible to overfitting and poor generalization. employing a regularizer is crucially important to prevent the model from becoming over-confident on the training data. we demonstrate that the proposed training strategies, when coupled with l2-norm regularization, lead to a significant improvement in performance, in some cases over 30% increase in accuracy. we apply our approach to the model proposed by @xcite as a modular vqa model. the model originally consists of two main components: (1) a program generator that takes a question and generates a program; and (2) an execution engine that combines neural modules according to the program in order to create a network to produce an answer from the input image. @xcite demonstrate that the program generator can produce acceptable programs by training only on a small fraction of all possible programs (≤ 4%). thus, we focus on training the execution engine in a low-data setting and use ground-truth programs as input to the execution engine. to simulate a low data regime, we use four randomly chosen small subsets of the clevr dataset @xcite for training. our results show that our cl approach yields more accurate results than those obtained when training with no curriculum. ## background visual question answering is the task of inferring the answer by reasoning on the input question and image. most of the current approaches map question-image pairs into a cross-modal common embedding space. a question is usually treated holistically in such approaches, thus the reasoning process is hard to explain @xcite @xcite . in contrast, modular approaches perform visual reasoning by semantically parsing the question and generating a reasoning chain called a program @xcite . the program shows the reasoning steps required for answering the question as a layout for the modules. the algorithm then combines the modules according to the program. modules are small neural networks treated as single-task functions that are combined into a larger network to accomplish a complex job. the resulting network is executed on the input image to predict the answer. modular approaches naturally have a strong potential for interpretability. @xcite showed human evaluators can more clearly understand their modular vqa model compared to a non-modular model @xcite . thus, we are interested in studying modular models. similar to other vqa models, modular approaches call for a large amount of annotated data for both the semantic parser (program generator) and the executor. this issue has led to recent studies on sample efficient training strategies, ranging from multi-task learning @xcite and active learning @xcite to disentangling reasoning from vision and language understanding @xcite . for instance, @xcite propose an agent that, instead of operating on the training set, interactively learns by asking questions. regarding the simulated low data setting in our work, efficient use of training data becomes extremely important. we employ curriculum learning in §4 and §5 as a method of making the best use of limited available data where a model can establish its understanding on simple concepts and gradually develop it by seeing harder examples over training. ## vqa model in a vqa task, a model receives as input a pair (x, q) of image x and a question q about the image. the model learns to select an answer @xmath0. the vqa model @xcite ) includes two main components: a program generator g and an execution engine e. the program generator predicts a program p to address a question q. the execution engine combines the modules according to the program, and executes the obtained network on the image to produce an answer. @xcite train the model using a semi-supervised learning approach. they demonstrate that the program generator can produce acceptable programs while training on only a small fraction of possible programs (≤ 4%). to evaluate e's performance in a low data regime, we conducted a number of vanilla supervised training experiments with decreasing sized training sets. note that we use ground truth program and image pairs as the input to e in all experiments. figure 1 shows the best accuracy of each experiment on clevr's validation set while the execution engine is trained on a subset of the clevr's train set e.g., 50% (see figure 2 for some examples of the clevr dataset). the results verify execution engine's poor performance on the small sized training subsets. ## curriculum heuristics for vqa studies introduce various heuristics for measuring the hardness of examples. some heuristics define hardness based on human judgment, in the sense that an example can be challenging for a machine if a human finds it difficult. such criteria take features of examples into consideration such as word frequency and sentence length for texts @xcite @xcite and shape complexity for images @xcite . the ordering of examples provided by these heuristics is task-dependent and does not change during training. in contrast, more general criteria determine the ordering of examples by incorporating the machine's response, e.g., a teacher network supervises the learning process (hacohen and weinshall, 2019) or the progress of a model is taken into account @xcite @xcite . in this study, we explore the heuristics described in the rest of this section. ## curriculum learning for vqa we describe now our training procedure. a generic curriculum learning requires a model m and a training dataset d as inputs. it also requires the existence of a hardness criterion n , a curriculum scheduler e, a selection function l, and a performance measure p . according to traditional curriculum learning, at every training iteration, the scheduler e decides when to update the curriculum. curriculum learning is applied on top of the conventional training loop in machine learning. the output of each training loop is usually the model's performance measure, which may be used by the scheduling function l to specify the appropriate moment for modifying the curriculum. the scheduler can also decide merely based on the number of training iterations. a curriculum update typically includes re-ranking training examples according to the hardness criterion n . in the next step, the algorithm selects algorithm 1 scheduled training with curriculum 1: e: execution engine 2: {(xi, pi, ai)} n @xmath1: training examples 3: γ: ∈ [0, 1], discount factor for reducing subset size 4: t : number of iterations 5: t0: number of warm-starting iterations 6: procedure hemtraining 7: phase1: warm-starting 9: @xmath2: else phase2: hard example mining 11: for @xmath3, ..., n} do 12: @xmath4: end for 14: normalise(pi) 15: s t ←sample k t district elements from categorical( p) 16: w t ←w t-1 +π ∇w @xmath5,e(p i ,x i ;w t-1 )) ## experiment we use our implementation of the execution engine model @xcite . a vanilla training of the model posts the lowest threshold of the performance in our setting. we also implemented and compared the three heuristics for the hardness criterion: program length ( §4.1), answer hierarchy ( §4.2) and hard example ( §4.3). the length-based curriculum can be seen as a baseline to the answer hierarchy criterion, while both of them play the role of baseline for the hard example curriculum. we do not compare with the state of the art, because the goal of our paper is to study vqa in a low-data regime, and to the best of our knowledge, there is no other work that conducts similar research. thus, we focus on improving the performance of our baseline models. we assessed our baselines under the following conditions: i) no-reg when no regularizer is applied. ii) dropout when we apply dropout technique to the final linear layer (classification layer) in e. iii) l2-norm when l2-norm regularizer is applied as a weight decay to the optimizer. ## conclusion this paper studied vqa in low data settings and shed light on the low performance of vqa models under the data scarcity condition. to improve the performance, we propose three curriculum learning approaches based on length, answer hierarchy, and hard examples. we also stressed the problem of overfitting and poor generalization that becomes crucially important in the absence of sufficient data. we explored the effect of using generalization techniques on a models' performance in low data regimes. our results show that the proposed cl algorithms outperform the baseline in many cases while fail in some others. however, the algorithms when coupled with l2 regularization lead to improvements.