Improving Few-Shot Image Classification Using Machine- and User-Generated Natural Language Descriptions (2022)

Abstract: Humans can obtain the knowledge of novel visual concepts from language descriptions, and we thus use the few-shot image classification task to investigate whether a machine learning model can have this capability. Our proposed model, LIDE (Learning from Image and DEscription), has a text decoder to generate the descriptions and a text encoder to obtain the text representations of machine- or user-generated descriptions. We confirmed that LIDE with machine-generated descriptions outperformed baseline models. Moreover, the performance was improved further with high-quality user-generated descriptions. The generated descriptions can be viewed as the explanations of the model’s predictions, and we observed that such explanations were consistent with prediction results. We also investigated why the language description improves the few-shot image classification performance by comparing the image representations and the text representations in the feature spaces.

## introduction humans can efficiently learn about new concepts from language @xcite . hence, in this paper, we focus on the few-shot image classification problem to verify machine learning models' capability to understand new concepts from language. this problem is a kind of meta-learning problem in which a model first learns from the concepts of classes by training on a few instances and then learns unseen classes in the same way. in our problem setting, the model can use language descriptions of images as additional information. this setting is similar to teaching a new concept to others by explaining it deductively from small amounts of data, unlike most machine learning models that learn inductively from large amounts of data. for this setting, we propose a new model, called lide (learning from image and description). as shown in figure 1 , lide consists of an image encoder, an image classifier, a text decoder, and a text encoder. lide has the advantage of providing explainablity. it passes an image representation encoded by the image encoder to the text decoder, which then generates a language description of the image as an explanation of the model's prediction. the image classifier then classifies the image in accordance with the text representation, which is encoded by the text encoder, in addition to the image representation. lide also provides high accuracy due to its explainability. it has been difficult to use machinegenerated descriptions to improve image classification performance because of their low quality @xcite . therefore, we design a training algorithm and a text encoding method to obtain robust text representations. in addition, lide includes a feature fusion module added to the image classifier to combine the information from both the image representation and the text representation. moreover, lide can take user-generated descriptions as input instead of machine-generated descriptions. we can use a description that contains textual features captured from an input image by human perception or a post-edited text in the text decoder output. the resulting high-quality descriptions provided by users can improve the image classification accuracy. our contributions are summarized as follows: ## conclusion we tackled the few-shot image classification task through learning of novel concepts from language descriptions of images. we observed that machineand user-generated descriptions improved the fewshot image classification performance. we also found that the generated captions explained the input image and were consistent with the prediction performance. our experiments also revealed four reasons why the text representation improved the performance: the inner-class distances of the multi-modal representations are smaller and the inter-class distances are larger than those of image representations; multi-modal representations are embedded in a space with a smaller latent dimension; multimodal representations are robust for noisy images; and certain types of knowledge are easily recovered from text representations. humans can learn concepts from language and explain them with language, but this is still difficult for machine learning models. this study sheds light on the importance of interactivity in explanating with language in machine learning.