PEEB : Part-based Image Classifiers with an Explainable and Editable Language Bottleneck (2024)

Abstract: CLIP-based classifiers rely on the prompt containing a class name that is known to the text encoder. Therefore, they perform poorly on new classes or the classes whose names rarely appear on the Internet (e.g., scientific names of birds). For fine-grained classification, we propose PEEB – an explainable and editable classifier to (1) express the class name into a set of text descriptors that describe the visual parts of that class; and (2) match the embeddings of the detected parts to their textual descriptors in each class to compute a logit score for classification. In a zero-shot setting where the class names are unknown, PEEB outperforms CLIP by a huge margin (∼10× in top-1 accuracy). Compared to part-based classifiers, PEEB is not only the state-of-the-art (SOTA) on the supervised-learning setting (88.80% and 92.20% accuracy on CUB-200 and Stanford Dogs-120, respectively) but also the first to enable users to edit the text descriptors to form a new classifier without any re-training. Compared to concept bottleneck models, PEEB is also the SOTA in both zero-shot and supervised-learning settings.

## introduction fine-grained classification @xcite is a long-standing computervision challenge. furthermore, it is also important to explain how sota classifiers make a decision, e.g., which bird traits make a model think a given bird is painted bunting? (fig. 1 ) first, many bird classifiers claim to be explainable @xcite by comparing the input image with a set of learned, part prototypes (fig. 1b ) or natural-language concepts (fig. 1a ). yet, such prototypes are feature vectors and therefore not editable by users. textual concepts are often compared against entire * equal contribution back: vibrant green coloring beak: conical, silver-gray belly: rich red hue ... throat: bright red plumage image for classification and it is unknown what image details match a given descriptor @xcite . third, most vision-language classifiers need the prompt to have a known {class name} (like a special code instead of an expressive, natural description) that matches the input image @xcite . fourth, most classifiers require either training-set images in a supervised-learning setting or demonstration images in a zero-shot setting @xcite . this requirement is impractical when building a classifier for a novel species whose photos do not yet exist in the database. to address the above four problems, we propose peeb, a part-based image classifier that is explainable and editable via a natural-language bottleneck. peeb classifies images by grounding the textual descriptor of object parts provided by humans or gpt-4 (no images needed) to detected parts in the image (fig. 1c ). while peeb leverages clip's encoders @xcite , it uses no class names (e.g., indigo bunting) in the prompt. in contrast, clip-based models @xcite @xcite rely so heavily on the known class names that their accuracy drops significantly when the names are removed or replaced by less-common ones (sec. 5.1). for birds , we first define the parts of interest for identifying a bird. we take the 15 parts defined in cub @xcite and reduce them to 12 by merging similar parts, e.g. left wing and right wing are merged into wings. using gpt-4 (openai, 2023), we construct a textual descriptor to describe each bird part of every species (see appendix c). next, peeb localizes the 12 bird parts in the image and computes their matching scores with corresponding text descriptors (fig. 2 ). the sum of the 12 dot products between the paired visual and textual part embeddings would be the unnormalized distance (logits) between the input image and every class for classification (fig. 3 ). for dogs , we use a similar procedure. to our knowledge, all existing public bird-image datasets (listed in table a4 ) are limited in size (less than 100k images per dataset) and in diversity (less than 1,500 species per dataset), impeding largescale, vision-language, contrastive learning. therefore, for our pre-training, we construct bird-11k, an exceptionally large dataset of bird images, comprising ∼290,000 images spanning across ∼11,000 species-essentially, all known bird species on earth (sec. 3). bird-11k is constructed from seven existing bird datasets and ∼55,000 new images that we collect from the macaulay library. similarly, we build dog-140, a large-scale dataset of 206k dog images. our main findings are: foot_0 1. clip-based classifiers rely mostly on class names in the prompt: the cub accuracy of m&v model @xcite drops drastically from 53.78% to 5.89% and 5.95% after class names are removed or replaced by scientific names (sec. 5.1). text-based concept bottlenecks like peeb, @xcite @xcite also match visual part embeddings to text embeddings. yet, they (1) do not use clip and instead rely on tf-idf text features; recent vision-language models (vlms) claim to be interpretable as they use textual concepts in the prompt. yet, some works that rely on class-wise differential captions @xcite or learned concept weights @xcite @xcite do not generalize to unseen classes. the most recent, similar work to peeb might be labo @xcite , which; however, operates at the image level instead of patch level, and does not generalize to unseen classes. many clip-based classifiers @xcite @xcite rely heavily on having seen class names in the prompt and thus are neither explainable nor editable to users. unlike clip-based models, peeb reveals what image details are being used for classification by matching descriptors to corresponding visual object parts (e.g. a bird's beak in fig. 3 ). @xcite employ a fixed set of attributes and train an attribute-to-label weight matrix for zero-shot classification. several studies @xcite @xcite highlight its effectiveness on datasets like cub, sun @xcite , and awa @xcite . yet, in practice, ale requires tabular data annotations for every new class in the dataset (e.g., 312 attributes per cub species), editing the weight matrix, and model re-training. in contrast, to add an unseen class to peeb, users would only need to describe its 12 bird parts in natural language. ## experiments & results we conduct extensive experiments to evaluate peeb on multiple bird datasets (cub, nabirds, inaturalist) and on gzsl (secs. 5.1 and 5.3), zsl (sec. 5.4) and also supervised learning settings. we also find peeb to perform well on dog image classification on dogs-120 (sec. 5.6).