Prompting Scientific Names for Zero-Shot Species Recognition (2023)

Abstract: Trained on web-scale image-text pairs, Vision-Language Models (VLMs) such as CLIP can recognize images of common objects in a zero-shot fashion. However, it is underexplored how to use CLIP for zero-shot recognition of highly specialized concepts, e.g., species of birds, plants, and animals, for which their scientific names are written in Latin or Greek. Indeed, CLIP performs poorly for zero-shot species recognition with prompts that use scientific names, e.g., “a photo of Lepus Timidus” (which is a scientific name in Latin). This is because these names are usually not included in CLIP’s training set. To improve performance, we explore using large-language models (LLMs) to generate descriptions (e.g., of species color and shape) and additionally use them in prompts. However, this method improves only marginally. Instead, we are motivated to translate scientific names (e.g., Lepus Timidus) to common English names (e.g., mountain hare) and use such in the prompts. We find that common names are more likely to be included in CLIP’s training set, and prompting them achieves 2~5 times higher accuracy on benchmarking datasets of fine-grained species recognition.

## introduction trained on large-scale image-text data, visionlanguage models (vlms) such as clip @xcite and align @xcite can recognize images of common objects in a zero-shot fashion, i.e., without further finetuning. such success is typically reported on standard benchmarks such as imagenet which contains 1,000 common classes, e.g., clip achieves 76.2% zero-shot accuracy vs. 11.5% by @xcite prior to clip. we are motivated to use clip for zero-shot recognition of highly specialized concepts, e.g., species of plants, birds, and animals. ## methods we start with the vanilla prompts that have been used in the contemporary literature of zero-shot recognition @xcite . vanilla prompt uses the prompt template "a photo of lepus timidus" to get a similarity measure for a given input image. note that for this method, we use species scientific names (written in latin or greek) in the prompts, hence call this method s-name. concretely, it feeds prompts to clip's text encoder for the text features, uses them to compute cosine similarities to the visual feature computed by the visual encoder for the given image. clip tends to produce a high similarity for the correct prompt (corresponding to ground-truth class name), hence achieving zero-shot recognition. prompt with additional descriptions enriches prompts with descriptions (e.g., shape, color, and size). these descriptions provide contextual information for the targeted concepts and help zero-shot recognition @xcite . recent work proposes to generate such descriptions using llms @xcite . although using llms does not always work on specific species (fig. 2 ), we follow this practice to generate descriptions using gpt4. when gpt4 fails to provide useful descriptions for any species, we use the simple prompts without any descriptions by default. note using additional descriptions apply to both vanilla prompt and methods below. prompt with common names translated from scientific names (c-name). we investigate open-clip's training set, i.e., laion400m @xcite , and find that it does not sufficiently cover scientific names (e.g., 468 out of 810 species in the semi-inat dataset have their scientific names in lainon400m). instead, many species have common names that are in laion400m. therefore, we turn to external resources (e.g., online collections and wikipedia) to translate scientific names (written in latin or greek) to common english names and use the latter in prompts. for the species that do not have any common english names (only a few in the benchmarking datasets), we use their original scientific names by default. prompt with more frequent names (f-name) between common and scientific names. we hypothesize that it is important to use the texts that are more frequently encountered by openclip during training. therefore, we further analyze laion400m @xcite to obtain ## experiments we conduct extensive experiments to validate our methods. we start with implementation details, datasets and the evaluation metric, and then analyze benchmarking results with in-depth analyses. ## conclusions we study the problem of zero-shot species recognition using pretrained vision-language models (vlms). for this problem, the typical prompt method that uses the species scientific names "by default" performs poorly, and additionally using descriptions in prompts improves only marginally. we find that scientific names written in latin or greek are not frequently seen in vlms' training set, explaining why they do not work well. yet, species english common names are more likely to be seen in vlms' training set. hence, we propose a rather simple method that translates scientific names to common english names, and use the latter in prompts. this method significantly boosts accuracy by 2-5 times on three benchmarking datasets.