Data-Efficient Language Shaped Few-shot Image Classification (2021)

Abstract: Many existing works have demonstrated that language is a helpful guider for image understanding by neural networks. We focus on a language-shaped learning problem in a few-shot setting, i.e., using language to improve few-shot image classification when language descriptions are only available during training. We propose a data-efficient method that can make the best usage of the few-shot images and the language available only in training. Experimental results on dataset ShapeWorld and Birds show that our method outperforms other state-of-the-art baselines in language-shaped few-shot learning area, especially when training data is more severely limited. Therefore, we call our approach data-efficient language-shaped learning (DF-LSL).

## introduction few-shot image classification is well aligned with the practical application scenarios where labeled images are costly to acquire. building effective few-shot image classifiers is challenged by the difficulty to improve the classifier generalizability given few labeled images in each class. recent efforts have been dedicated to design metricbased approaches @xcite , augmentation-based methods, @xcite @xcite , and meta-learning methods @xcite @xcite . another stream of work introduces language information to guide the image classification @xcite , because nature languages are a kind of reflection of the world and convey rich information and knowledge for understanding the visual patterns. in this paper, we target on addressing the few-shot image classification by efficiently using the language description as a guide during the training of image classification model. different from @xcite and @xcite , we aim to deal with a more challenging scenario where we have no language information during testing period. all language descriptions are only available during training. our study shares the same setting with only one recent work in @xcite . we design a different model that can make the best usage of the few images available for each class and the language information in the training process. the key difference is two-fold. first, all few-shot images are asked to participate in the language-shaping stage to enhance the guidance on image understanding. second, extra supervision tasks are introduced to enlarge the communication channel between language description and images. our proposed approach, named data-efficient language shaped learning (df-lsl), is shown in extensive evaluation to perform better than stateof-the-art baselines. comparing to the strongest baseline lsl in @xcite , our proposed method df-lsl has 1.8% higher accuracy on the cub benchmark dataset, and 0.5%-1.8% higher accuracy on the shapeworld benchmark dataset. learning from other domains: zero-shot learning. zero-shot learning is a kind of problem setup where a model needs to predict the class of samples, without giving samples in those classes in training phase. usually, a zero-shot model has to utilize some side information from other domains, to learn about those zero-shot classes. there are approaches using attribute information to give descriptions about the unseen classes @xcite . there are also approaches @xcite try to transfer language information into zeroshot image classification and achieved good results. language related learning. nature language shapes the way we know about the world, and thus has been introduced to assistant various tasks. for example, language descriptions are generated to explain the decisions of neural network (belle, 2017) for improving the explainability of deep learning methods. language can also provide guidance during learning. this idea is applied on many different learners such as monte-carlo framework @xcite and reinforcement learning @xcite . moreover, @xcite try to use language information to guide the image classification, which is also our study purpose. ## the proposed method our proposed df-lsl method has two key differences from lsl, as shown in figure 1 . the details are discussed next. ## experiments in general, we implement by the same settings as lsl @xcite for the sake of fair comparison. for predictions, we average the probabilities across k prototypes. we use shapeworld @xcite and cub @xcite dataset to evaluate our method. the details of experimental settings and model descriptions can be found in our source code foot_0 . non-overlapping shapes, and the language descriptions are related to the special information between two shapes. following the same setting in lsl, we set k, the number of image samples per class to 4, and a language description is associated with that class (a universal description for those four images). query set has positive and negative samples, where positive samples can match that language description while the negative ones cannot. our entire dataset contains 9000 training tasks, 1000 validation tasks and 4000 tasks. no augmentation method was employed on this dataset, because the images in shapeworld dataset are classified accroding to their colors, shapes and positions. cropping, flipping and color jittering will be harmful to those properties. due to the special case of binary classification, we simply apply dot-product operation to be our similarity metric s, then use a sigmoid function to scale the similarity value from 0 to 1, which will be a suitable representation for probability. caltech-ucsd birds. images in shapeworld dataset are synthetic by computers and only contain several basic shapes and a black background. in real world scenarios, we have more complicated shapes and more noisy background information. moreover, we only have one language description per class in shapeworld, which is not enough for us to analysis the influence of the amount of language information. therefore, we perform our experiments on another challenging dataset caltech-ucsd birds (cub) @xcite , which contains 200 bird species and their images. all language descriptions are from @xcite , and describe each bird image with ten different sentences. for the purpose of pre-processing and augmentation, we apply pixel normalization, color jittering, horizontal flipping and random cropping. we use a matrix w as the similarity function, which means s(a, b) can be calculated by a t w b. ## conclusion this paper proposes a data-efficient language shaped learning (df-lsl) model, which aims to improve the few-shot image classification model by language information. experiment results show that the overall performance of our approach surpasses all other baselines on two benchmark datasets. this verifies the effectiveness of the proposed two key innovations in df-lsl.