On the Limits of Multi-modal Meta-Learning with Auxiliary Task Modulation Using Conditional Batch Normalization (2024)

Abstract: Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that cross-modal learning can improve representations for few-shot classification. More specifically, language is a rich modality that can be used to guide visual learning. In this work, we experiment with a multi-modal architecture for few-shot learning that consists of three components: a classifier, an auxiliary network, and a bridge network. While the classifier performs the main classification task, the auxiliary network learns to predict language representations from the same input, and the bridge network transforms high-level features of the auxiliary network into modulation parameters for layers of the few-shot classifier using conditional batch normalization. The bridge should encourage a form of lightweight semantic alignment between language and vision which could be useful for the classifier. However, after evaluating the proposed approach on two popular few-shot classification benchmarks we find that a) the improvements do not reproduce across benchmarks, and b) when they do, the improvements are due to the additional compute and parameters introduced by the bridge network. We contribute insights and recommendations for future work in multi-modal meta-learning, especially when using language representations.

## introduction it is widely recognized that humans can learn new concepts based on very little supervision, i.e. with few examples (or "shots"), and generalize these concepts to unseen data @xcite . recent advances in deep learning on the other hand have mostly relied on datasets with large amounts of labeled examples, primarily due to overfitting figure 1 : architectural overview of the method we experimented with. it consists of three components: a classifier, an auxiliary network, and a bridge network. the few-shot classifier and auxiliary network receive the same input example. the bridge network transforms high-level features of the auxiliary network into modulation parameters for layers of the few-shot classifier through conditional batch normalization. although this setting is often used to illustrate the remaining gap between human capabilities and machine learning, we could argue that the lack of context poses a serious disadvantage to machine learning models. indeed, these models typically work based on a single-pass analysis while humans can first look at and understand contextual information before trying to interpret new classes @xcite . it has been observed many times in the past that training models with contextual information such as auxiliary modalities can help build a more robust task-independent feature space @xcite @xcite . auxiliary tasks however often require large support datasets with good label distributions and a delicate adjustment of network capacity to really help improve performance on the main task @xcite . multi-modal information can be difficult to process using a simple backbone architecture due to the varied structure and high-level nature of some typically used modalities, although recent transformer-based works have shown it's possible, albeit costly @xcite . we refer to the appendix a for a more comprehensive study of the related work. we propose studying whether multitask learning with multi-modal objectives could be beneficial for few-shot learning even with commonly-used lowcapacity feature extraction backbones, and without weight sharing between the main and auxiliary tasks. we study a way to condition multiple layers of our main feature extractor using an embedding produced by an entirely separate auxiliary network working on the same input data. the conditioning is applied to normalization layer parameters using a bridge network and it helps specialize the representations produced by the main feature extractor without affecting its architecture. our idea here is to mimic the way humans can leverage context to help solve the recognition problem by combining low-level and high-level cues. in other words, we allow the main feature extractor to decide ahead of time what it should focus on based on task-level contextual knowledge. the proposed model architecture is illustrated in figure 1 . in contrast with previous works that also studied feature extraction conditioning and multi-modal learning, our approach is simple and can be applied to any feature extractor with batch normalization layers. the bridged-parallel-network design we propose also simplifies the feature alignment process since both branches process the same input data. finally, the need for only a single input modality at test time leads to a more practical design for downstream applications. however, after evaluating the proposed approach on two popular few-shot classification benchmarks we find that a) the improvements do not reproduce across benchmarks, and b) when they do, the improvements are due to the additional compute and parameters introduced by the bridge network. we contribute insights and recommendations for future work in multi-modal meta-learning, especially when using language representations. ## proposed method in this section, we formulate conditional batch normalization in the context of few-shot learning. we propose a model, simpaux, with two feature extractors that predict high-level (language-based) attributes of images as well as their semantic class. the embeddings of the attribute prediction pipeline (or "auxiliary" pipeline) are used to condition the batch normalization layers of the main visual feature extractor, which is based on a protonet architecture @xcite . more specifically, we use protonet++ improvement introduced in @xcite , with a resnet-12 @xcite , which is a common choice in few-shot learning settings (e.g. @xcite . the conditioning happens through a bridge connection composed of dense layers that translates the auxiliary embedding into batch normalization statistics. these three components are shown in figure 1 and are described in the following subsections. note that we use the same input modality (imagery) for the auxiliary and main feature extractors. however, our method is not limited to this modality: it was primarily chosen for compatibility with existing datasets. we refer the reader to appendix b for a review of the fundamental ideas required to better understand our proposed few-shot learning solution from a technical standpoint. ## experimental results we evaluate simpaux against the baseline, protonet++, (the improved version of pro-tonets suggested in @xcite ) on two popular few-shot learning benchmarks, @xcite and mini-imagenet @xcite in 5-shot learning settings, using attributes for cub and embeddings on synthetic captions for mini-imagenet for the auxiliary visual processing network. we refer to appendix c for additional implementation details. ## discussion and recommendations from the experimental results, we conclude that a) the improvements provided by simpaux do not reproduce across benchmarks, and b) when these improvements do indeed take place, they seem to be due to the additional compute and parameters provided by the bridge network.we hypothesize three non-mutually exclusive reasons why image captioning as auxiliary task modulation via conditional batch normalization did not consistently improve the results: 1) a lack of quality of the image captions, attributes, or caption embeddings, 2) the limited impact of the conditional batch normalization approach, and 3) the difficulty of learning the auxiliary task. while improving the quality of captions, attributes and caption embeddings with better annotations or more powerful models could alleviate 1), the following recommendations and observations look at other aspects involved in this work. caution when evaluating systems with auxiliary multi-modal information. training models with contextual information such as auxiliary modalities have been shown to build a more robust taskindependent feature space @xcite @xcite . however, spurious improvements with multi-modal data are not new. for instance, @xcite empirically raises doubts about whether existing multi-modal translation systems, combining visual and textual data, actually make use of the visual information. similarly, we have seen the other way around: it is perfectly possible to outperform a unimodal baseline with a multi-modal one without actually making use of the textual information; simpaux's improvements in cub were due to the additional parameters introduced by the bridge network. thus, we recommend extra care when concluding that multimodal information helps in a certain task, which is definitely possible but could be due to other factors. importance of implementation details. we experimented with different activation functions, including relu (agarap, 2018), selu @xcite , and silu @xcite . we found that silu consistently yielded slightly better results across benchmarks and settings. ensuring that weight decay was not applied to bias parameters, which is not the default behavior in pytorch @xcite , also turned out to be key to re-producing few-shot works originally implemented in tensorflow @xcite . hyperparameter search. in the hyperparameter search, we generally observed consistent results. however, we also observed a few outliers, which can be particularly extreme under certain settings in few-shot learning, and if used as empirical evidence, could totally change the conclusions. thus, we reiterate the need for reporting averages and variances instead of the results of a single run, and also recommend caution at extracting certain conclusions when performing large-scale hyperparameter searches, as noted by @xcite . ## conclusion in this work, we have studied a new multi-modal architecture for few-shot learning consisting of an image classifier, an auxiliary network trained with image captions, and a modulating network based on conditional batch normalization to connect the two. while initially promising, we have observed the limits of this approach and how these limits could inform future research. fang zhao, jian zhao, shuicheng yan, and jiashi feng. 2018. dynamic conditional networks for few-shot learning. in proceedings of the european conference on computer vision (eccv), pages 19-35. imtiaz masud ziko, jose dolz, eric granger, and ismail ben ayed. 2020. laplacian regularized few-shot learning. arxiv preprint arxiv:2006.15486.