Learning to Learn Semantic Factors in Heterogeneous Image Classification (2021)

Abstract: Few-shot learning is to recognize novel classes with a few labeled samples per class. Although numerous meta-learning methods have made significant progress, they struggle to directly address the heterogeneity of training and evaluating task distributions, resulting in the domain shift problem when transitioning to new tasks with disjoint spaces. In this paper, we propose a novel method to deal with the heterogeneity. Specifically, by simulating class-difference domain shift during the meta-train phase, a bilevel optimization procedure is applied to learn a transferable representation space that can rapidly adapt to heterogeneous tasks. Experiments demonstrate the effectiveness of our proposed method.

## introduction deep learning methods are now widely used in diverse applications. however, their efficacy is largely contingent on a large amount of labelled data in the target task and domain of interest @xcite . different from humans that can easily learn to accomplish new tasks with a few examples, it is difficult for machines to rapidly generalize to new concepts with very little supervision, which calls considerable attention to the challenging few-shot learning (fsl) setting. for example, few-shot classification problem requires models to classify unlabeled samples into novel classes with only a few labeled samples available for training @xcite . commonly understood as learning to learn, meta-learning paradigm has made significant progress in fsl by transferring knowledge extracted from a collection of previous tasks @xcite . such taskagnostic knowledge can contribute to the current testing task with optimizing learning algorithms. however, beyond its recent achievements, metalearning still faces the problem of generalization. in contrast to supervised machine learning methods which assume that training and testing data are sampled i.i.d. from the same distribution, fsl aims to learn to address tasks from different distributions with limited data. this refers to the realistic scenario that the label spaces of future testing tasks can not be obtained in advance and are often disjoint with the label spaces of training tasks. in experiments, this is actualized by splitting all categories in the dataset into non-overlapping base classes and novel classes, while training tasks are sampled from base classes and testing tasks are samples from novel classes. therefore, due to the class label difference, meta-learning approaches suffer from natural heterogeneous distributions of tasks. as each task can be regarded as having a separate domain, it can be considered as a special case of domain shift that is extremely serious when a large gap of semantic relationship exists between base classes and novel classes. as most of the current meta-learning approaches make a strong assumption that training tasks and testing tasks are drawn from the similar distributions and share the same characteristics, @xcite has shown the limitations of existing approaches in cross-domain fsl scenarios where base classes and novel classes are from different datasets. however, few works have focused on this issue to improve existing approaches. for example, as a representative work of metric-based meta-learning, prototypical network @xcite learns a metric space where embeddings of query samples in one class are close to the centroid of support samples in the same class, and far from centroids of other classes in the task. while prototypical network benefits from a simple but effective inductive bias, it lacks adaptation to new tasks or domains. in this paper, we propose to improve such metricbased approaches with a bilevel optimization procedure. specifically, we simulate class-differencecaused domain shift during meta-training by simultaneously sampling multiple tasks with nonoverlapping class sets. each time one of the tasks is prepared as the target task for outer level optimization and the others are first used as the source tasks for inner level optimization of the network. following this training strategy during the metatrain phase, the model can better adapt to the testing tasks from heterogeneous distributions with an adaptation step. moreover, different from some usual options of inner objective, we use shannon entropy as an unsupervised factorization loss to constrain the learned representations as near-binary codes @xcite . this can be viewed as learning a discriminative latent factor space for each task where each factor can be interpreted as a latent attribute that is corresponding to abstract visual concepts. to summarize, our main contributions are :1) considering the challenge of heterogeneous task distributions faced by few-shot learning, we simulate the class-difference-caused domain shift in the meta-train phase, and devise a metric-based metalearning approach integrated with a bilevel optimization for better generalization; 2) we propose to utilize an unsupervised factorization loss as the inner objective, making representations to be nearbinary codes that reduce the difficulty of classifier learning. meanwhile, due to the bilevel optimization between heterogeneous few-shot tasks during meta-training, the model can rapidly learn the representation space for testing tasks; 3) we conduct extensive experiments and analysis to demonstrate that our approach effectively improves the performance and interpretability under both conventional and cross-domain few-shot settings without introducing additional architectures, and thus it can be regarded as a better baseline. as a simple but effective model for fsl learning, prototypical network (protonet) @xcite use an embedding function f θ with parameters θ to encode each sample into a representation vector. for each class c in the class set c of the task t , a prototype vector p c is defined as the mean vector of the embedded support samples in the class, which can be expressed as when inferring, the probability over classes for a query sample x i is a softmax over the inverse of squared euclidean distances between the query representation and prototype vectors, expressed the classification loss is the sum of negative logprobability of each query sample in task t with its ground-truth class label: ## conclusion in this paper, we propose meta-protonet to handle the challenge of heterogeneous task distributions in few-shot scenarios, aiming to learn a latent factor space in which metric-based classification of heterogeneous tasks can be better performed. extensive experiments show that our proposed approach can be considered as a stronger baseline in both conventional and cross-domain few-shot settings.