Retrieval-enriched zero-shot image classification in low-resource domains (2024)

Abstract: Low-resource domains, characterized by scarce data and annotations, present significant challenges for language and visual understanding tasks, with the latter much under-explored in the literature. Recent advancements in Vision-Language Models (VLM) have shown promising results in high-resource domains but fall short in low-resource concepts that are under-represented (e.g. only a handful of images per category) in the pre-training set. We tackle the challenging task of zero-shot low-resource image classification from a novel perspective. By leveraging a retrieval-based strategy, we achieve this in a training-free fashion. Specifically, our method, named CoRE (Combination of Retrieval Enrichment), enriches the representation of both query images and class prototypes by retrieving relevant textual information from large web-crawled databases. This retrieval-based enrichment significantly boosts classification performance by incorporating the broader contextual information relevant to the specific class. We validate our method on a newly established benchmark covering diverse low-resource domains, including medical imaging, rare plants, and circuits. Our experiments demonstrate that CoRE outperforms existing state-of-the-art methods that rely on synthetic data generation and model fine-tuning.

## introduction low-resource domains refer to those rare domains where the data or its annotation is truly scarce. similarly, low-resource languages are those that have significantly less content available online @xcite with respect to other highresource languages, like english. there exist abundant research on the topic in the context of natural language processing @xcite @xcite ., * currently at apple ## background to establish a foundation in understanding our method, we provide a brief introduction to vision-language models and web-scale databases, the two essential elements in our method design. ## core we focus on zero-shot low-resource image classification, following the paradigm of vlms as described in sec 3.1. low-resource domains are generally not well represented by vlms given their limited availability in the pre-training dataset @xcite . there is a need to steer the original data representation towards more specialized low-resource domains represented by the query image q. to this end, our training-free method core extracts domain-relevant information from a large text-image database d, to enrich the representation of both class prototypes and the query image. for the class prototypes, i.e. textual representation of the classes {c n } n @xmath0, we retrieve semantically close captions for each class with the embedding of prompt text regarding the c. on one hand, the retrieved captions contain rich domainlevel information while being less specific to the exact class. on the other hand, the class prompt is only specific to the class of interest without much prior of the domain information. therefore, we further join the retrieved captions with the prompt class text to obtain the textual class prototypes w + enriched with the domain context. a similar rationale is applied to enrich the query image representation via image-to-text retrieval, implemented by the image encoder f v v lm of a pretrained vlm. with the visual embedding z q of the query image q, we retrieve the set of captions that are the most aligned to z q in the shared latent space, and use the retrieved captions to obtain an enriched image representation z + q . the final class is predicted using z + q and w + similar to eq. 1: we describe each retrieval branch in detail in the following sections and we show an overview of our core in fig. 2 . ## experiments we evaluate our proposed method core in comparison with state-of-the-art vision-language approaches using three challenging low-resource datasets. we describe the dataset used and the evaluation protocol we follow. we discuss the results w.r.t. the baseline methods, and we ablate the proposed components and architectural choices. datasets. we consider three datasets covering different low-resource scenarios for our analysis: we employ the circuit diagram classification dataset @xcite 1 that comprises 1,332 circuit diagram images covering 32 different classes, the images are scraped from the web 1 the other released datasets relate to retrieval tasks. and textbooks. the authors split the data into 154 images for training and the remaining for testing, resulting in an average of ∼ 5 samples per class available at training time. the second dataset we consider is inaturalist 2021 @xcite which contains 10,000 species with a fine-grained classification. the dataset features many visually similar species, captured in a wide variety of situations. in order to remain in the rare domain setting, we restrict our analysis to the rarest 100 species in terms of available training samples and set the maximum amount of training shots to 5. we test on 10 images for each class, therefore the test set contains 1,000 samples. the last dataset we employ is ham10000 @xcite , which comprises dermatoscopic images from different populations. the dataset has 7 classes that include a representative collection of all important diagnostic categories in the realm of pigmented lesions. the test set includes 1,511 images. database(s). we use cc12m @xcite as our source dataset to build the database d, following previous works @xcite . we also use a subset (10%) of coyo-700m to test the scaling laws of the retrieval database. we focus on their textual corpus as we are interested in retrieving the relevant pieces of text. the retrieval databases are implemented using faiss @xcite on pre-extracted embeddings. we use the siglip @xcite text encoder for the image-to-text retrieval, while we employ a textonly encoder to obtain a stronger textual semantic in text-to-text retrieval. in particular, we use sfr-embedding-mistral @xcite as f llm . evaluation protocol. as a common practice in image classification tasks, we report the perfor-mance as the top-1 and top-5 accuracy. for vlms, we compare the index of the highest logit to the ground truth, while for large multimodal models (lmm) we parse the lmm output to extract the predicted class name and match it with the dataset class names. ## conclusion we presented core, a training-free retrieval-based zero-shot solution for low-resource image classification. through the retrieval of semantically relevant textual information for both image representations and class prototypes, core is able to enhance the richness of feature representations and achieves state-of-the-art performance. remarkably, it does so without the need for additional training or labeled data, outperforming existing trainingbased methods under extremely low-resource conditions. moreover, we established a comprehensive benchmark using representative datasets and baseline models, providing a robust testing ground for the low-resource image classification task. the results demonstrate the efficacy and generalization capability of core across various low-resource domains, representing a significant step forward in low-resource image classification.