Rethinking Task Sampling for Few-shot Vision-Language Transfer Learning (2022)

Abstract: Despite achieving state-of-the-art zero-shot performance, existing vision-language models still fall short of few-shot transfer ability on domain-specific problems. Classical fine-tuning often fails to prevent highly expressive models from exploiting spurious correlations. Although model-agnostic meta-learning (MAML) presents as a natural alternative for few-shot transfer learning, the expensive computation due to implicit second-order optimization limits its use on large-scale vision-language models such as CLIP. While much literature has been devoted to exploring alternative optimization strategies, we identify another essential aspect towards effective few-shot transfer learning, task sampling, which is previously only be viewed as part of data pre-processing in MAML. To show the impact of task sampling, we propose a simple algorithm, Model-Agnostic Multitask Fine-tuning (MAMF), which differentiates classical fine-tuning only on uniformly sampling multiple tasks. Despite its simplicity, we show that MAMF consistently outperforms classical fine-tuning on five few-shot image classification tasks. We further show that the effectiveness of the bi-level optimization in MAML is highly sensitive to the zero-shot performance of a task in the context of few-shot vision-language classification. The goal of this paper is to provide new insights on what makes few-shot learning work, and encourage more research into investigating better task sampling strategies.

## introduction while existing machine learning models have achieved human-level performance at various individual tasks, they generally lack the ability of fast adaptation and generalization. in recent years, transfer learning has been proven to be effective on a wide range of computer vision @xcite and natural language processing @xcite tasks. specifically, recent advances in large-scale vision-language models @xcite @xcite have demonstrated strong zero-shot ability on a wide range of tasks. however, these models still have certain limitations on concepts that require extensive domain knowledge, such as fungi classification. we identify two major limitations in current few-shot transfer learning literature, from both evaluation and algorithm perspectives. limitation on evaluation: in current transfer learning paradigm, the testing instances of a downstream task are drawn from the same distribution as the training set. this evaluation setting can fail to faithfully reflect whether a model has truly learned a new concept, since modern deep neural networks can easily memorize and exploit spurious correlations from the training set @xcite . thus, we first propose a new evaluation scheme for few-shot transfer learning where we replace the original testing phase with meta-testing (section 3). with meta-testing, the testing distribution are distinguished from the training. limitation on algorithm: to make an arbitrary pretrained vision-language model learn new concepts with few examples, model-agnostic metalearning (maml) @xcite presents as a natural candidate. one major limitation of the original maml method is the expensive computation overhead due to implicit second-order optimization. most follow-up work @xcite @xcite @xcite has focused on improving the optimization strategy. however, we found that they all achieved comparable performance despite of using different optimization algorithms. this observation motivates us to ask: if the specific choice of optimization method is not the key to the empirical success of maml, what would be? inspired by related work in the area of multitask learning @xcite , we conjecture that task sampling itself is an essential ingredient in learning new concepts efficiently. to verify this hypothesis, we propose a simple fine-tuning algorithm, model-agnostic multitask fine-tuning (mamf), which simplifies maml by using only first-order gradient-based optimization while keeping the uniform task sampling procedure intact. the goal is not to propose yet another complex algorithm, but to investigate what is the most important aspect for effective fewshot transfer learning. we compare mamf with classical fine-tuning, which does not perform uniform task sampling, and first-order maml (fo-maml) @xcite , which adopts complex bi-level optimization upon sampled tasks. our empirical result demonstrates the importance of uniform task sampling and reveals limited effectiveness of the bi-level optimization of maml in the context of few-shot transfer learning. we hope our work encourages more research into exploring better task sampling strategies for improving few-shot transfer learning and meta-learning algorithms. ## problem formulation we are interested in a few-shot classification problem where we have a pretrained vision-language model f with initial parameters θ. let τ tr be a training task sampled from a distribution p(τ tr ), and τ ts be a testing task sampled from p(τ ts ), where a task is defined to an induced sub-problem by restricting the output space from the original problem. specifically, for an original classification problem with m classes in total, we define a task as a sub-problem where the output space is a subset of n classes randomly sampled from the m classes. we further denote n tr and n ts as the number of classes in each training and testing task. t tr and t ts as the total number of sampled tasks respectively. the classical fine-tuning setting is depicted in figure 1 (a), where we have t @xmath0, and t @xmath1. that is, both training and testing sets are treated as one single task containing data points from all m classes. ## reformulating classical fine-tuning evaluation with meta-testing our goal is to enable and evaluate a model's capability of generalizing to new concepts with few examples. the classical fine-tuning setting is not sufficient since the training and testing data points are drawn from the same distribution. therefore, we propose to replace the original joint testing in classical fine-tuning with meta-testing. meta-testing is first introduced by related work in meta-learning @xcite @xcite . as shown in the testing phase of figure 1 (b , c , d ), we first sample t ts tasks (t @xmath2, each containing data points from n ts classes (1 < n @xmath3. for each sampled testing task τ ts , we further randomly split the data points into two disjoint sets, i.e., support set a and query set b, with corresponding loss l τ ts ,a and l τ ts ,b . then we further update the model parameters on the support set and evaluate on the query set. by randomly sampling multiple tasks during meta-testing, we can distinguish the testing distribution from training, which largely prevents the model from exploiting spurious correlations in the training set. essentially, we make the original problem more challenging by requiring the model to quickly generalize to potentially unseen task distributions during testing. the objec-tive is to find an updated model parameter θ that minimizes the expected loss on all testing tasks e τ ts ∼p(τ ts ) l τ ts ( θ) . specifically, under this setting, maml's objective can be written as follows: ## model-agnostic multitask fine-tuning as shown above, previous maml-like methods update model parameters iteratively via a complex bi-level optimization scheme @xcite @xcite , which is computationally expensive. we hypothesize that the task sampling process itself is more important than specific choice of optimization method. to verify this hypothesis, we propose a simple algorithm, model-agnostic multitask fine-tuning (mamf), where we keep the uniform task sampling strategy as maml but perform simple first-order gradient-based optimization on each task sequentially. unlike maml, mamf does not further split the tasks into support and query sets. the objective of mamf can be written as: ## conclusion in this paper, we demonstrate the importance of task sampling by proposing a simple yet effective fine-tuning method mamf. we further show novel insights on the limited effectiveness of the bi-level optimization. we hope our work encourage more research on improving few-shot transfer learning via better task sampling beyond uniform sampling.