Hugging Face's logo

imageomics
/
bioclip

Zero-Shot Image Classification
OpenCLIP
English
clip
biology
CV
images
animals
species
taxonomy
rare species
endangered species
evolutionary biology
multimodal
knowledge-guided
Model card Files Community
6
bioclip / README.md
egrace479's picture
egrace479
Add Model card
85baf1d
preview code
|
raw
history blame
11.7 kB
metadata
license: mit
language:
  - en
library_name: open_clip
tags:
  - zero-shot-image-classification
  - clip
  - biology
  - CV
  - images
  - animals
  - species
  - taxonomy
  - rare species
  - endangered species
  - evolutionary biology
  - multimodal
  - knowledge-guided
datasets:
  - TreeOfLife-10M
  - iNat21
  - BIOSCAN-1M

Model Card for BioCLIP

BioCLIP is a foundation model for the tree of life, built using CLIP architecture as a vision model for general organismal biology. It is trained on TreeOfLife-10M, our specially-created dataset covering over 450K taxa--the most biologically diverse ML-ready dataset available to date. Through rigorous benchmarking on a diverse set of fine-grained biological classification tasks, BioCLIP consistently outperformed existing baselines by 17% to 20% absolute. Through intrinsic evaluation, we found that BioCLIP learned a hierarchical representation aligned to the tree of life, which demonstrates its potential for robust generalizability.

Model Details

Model Description

BioCLIP is based on OpenAI's CLIP. We trained the model on TreeOfLife-10M from OpenAI's ViT-B/16 checkpoint, using OpenCLIP's code. BioCLIP is trained with the standard CLIP objective to imbue the model with an understanding, not just of different species, but of the hierarchical structure that relates species across the tree of life. In this way, BioCLIP offers potential to aid biologists in discovery of new and related creatures, since it does not see the 454K different taxa as distinct classes, but as part of an interconnected hierarchy.

  • Developed by: Samuel Stevens, Jiaman Wu, Matthew J. Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su
  • Model type: Vision Transformer (ViT-B/16)
  • License: MIT
  • Fine-tuned from model: OpenAI CLIP, ViT-B/16

Model Sources

Uses

BioCLIP has been extensively evaluated on species classification tasks across many different subtrees of the tree of life. The ViT-B/16 vision encoder is recommended as a base model for any computer vision task for biology; we expect it to outperform general domain models with the same architecture on biology-specific tasks.

Direct Use

See the demo here for examples of zero-shot classification. It can also be used in a few-shot setting with a KNN; please see our paper for details for both few-shot and zero-shot settings without fine-tuning.

Bias, Risks, and Limitations

This model was developed from the original CLIP model, thus many of the concerns discussed in (Radford et al. 2021) apply. We encourage the concerned/curious user to read their extensive ethics statement, while we focus our attention on the biological perspective which is unique to BioCLIP.

  • No specific geographic information (eg., GPS coordinates) are included in training, so the species classification does not pose a direct threat to animals through aiding poachers, as it cannot inform them of their location.
  • BioCLIP is designed to aid in scientific discovery through an association of images to the hierarchical taxonomy structure. As with many--if not all--models currently in production, it is important to retain the context that it is meant to assist biologists in their work, not replace them. As such, we caution against over-reliance on model predictions.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

BioCLIP can be used with the open_clip library:

import open_clip

model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:imageomics/bioclip')
tokenizer = open_clip.get_tokenizer('hf-hub:imageomics/bioclip')

Training Details

Compute Infrastructure

Training was performed on 8 NVIDIA A100-80GB GPUs distributed over 2 nodes on OSC's Ascend HPC Cluster with global batch size 32,768 for 4 days.

Based on Machine Learning Impact calculator presented in Lacoste et al. (2019), that's 132.71 kg of CO2 eq., or 536km driven by an average ICE car.

Training Data

This model was trained on TreeOfLife-10M, which is a compilation of images matched to Linnaean taxonomic rank from kingdom through species. They are also matched with common (vernacular) name of the subject of the image where available. For more information, please see our dataset, TreeOfLife-10M.

Training Hyperparameters

  • Training regime: fp16 mixed precision.

We resize images to 224 x 224 pixels. We use a maximum learning rate of 1e4 with 1000 linear warm-up steps, then use cosine decay to 0 over 100 epochs. We also use a weight decay of 0.2 and a batch size of 32K.

Evaluation

Testing Data

We tested BioCLIP on the following collection of 10 biologically-relevant tasks.

  • Meta-Album: Specifically, we used the Plankton, Insects, Insects 2, PlantNet, Fungi, PlantVillage, Medicinal Leaf, and PlantDoc datasets from Set-0 through Set-2 (Set-3 was still not released as of our publication/evaluation (Nov. 2023).
  • Birds 525: We evaluated on the 2,625 test images provided with the dataset.
  • Rare Species: A new dataset we curated for the purpose of testing this model and to contribute to the ML for Conservation community. It consists of 400 species labeled Near Threatened through Extinct in the Wild by the IUCN Red List, with 30 images per species. For more information, see our dataset, Rare Species.

For more information about the contents of these datasets, see Table 2 and associated sections of our paper.

Metrics

We use top-1 and top-5 accuracy to evaluate models, and validation loss to choose the best performing checkpoints from training.

Results

We compare BioCLIP to OpenAI's CLIP and OpenCLIP's LAION-2B checkpoint. Here are the zero-shot classification results on our benchmark tasks. Please see our paper for few-shot results.

Model Animals Plants & Fungi Rare Species Mean
Birds 525 Plankton Insects Insects 2 PlantNet Fungi PlantVillage Med. Leaf PlantDoc
CLIP 49.9 3.2 9.1 9.8 58.5 10.2 5.4 15.9 26.1 26.6 21.4
OpenCLIP 54.7 2.2 6.5 9.6 50.2 5.7 8.0 12.4 25.8 31.0 20.6
BioCLIP 74.7 5.4 32.7 21.2 91.0 51.8 24.0 48.1 27.5 39.2 41.5
iNat21 Only 55.7 2.7 29.9 12.0 89.3 42.7 16.4 22.2 18.8 19.4 30.9

Summary

BioCLIP outperforms general-domain baselines by 18% on average.

Model Examination

We encourage readers to see Section 4.6 of our paper. In short, BioCLIP forms representations that more closely align to the taxonomic hierarchy compared to general-domain baselines like CLIP or OpenCLIP.

Citation

BibTeX:

@software{bioclip2023,
  author = {Samuel Stevens and Jiaman Wu and Matthew J. Thompson and Elizabeth G. Campolongo and Chan Hee Song and David Edward Carlyn and Li Dong and Wasila M. Dahdul and Charles Stewart and Tanya Berger-Wolf and Wei-Lun Chao and Yu Su},
  doi = {<doi once generated>},
  month = nov,
  title = {BioCLIP},
  version = {v0.1},
  year = {2023}
}

Please also cite our paper: (Coming Soon)

Please also cite OpenCLIP:

@software{ilharco2021openclip,
  author = {Ilharco, Gabriel and Wortsman, Mitchell and Wightman, Ross and Gordon, Cade and Carlini, Nicholas and Taori, Rohan and Dave, Achal and Shankar, Vaishaal and Namkoong, Hongseok and Miller, John and Hajishirzi, Hannaneh and Farhadi, Ali and Schmidt, Ludwig},
  doi = {10.5281/zenodo.5143773},
  month = jul,
  title = {{OpenCLIP}},
  version = {v0.1},
  year = {2021}
}

APA:

Stevens, S, Wu, J., Thompson, M. J., Campolongo, E. G., Song, C. H., Carlyn, D. E., Dong, L., Dahdul, W. M., Stewart, C., Berger-Wolf, T., Chao, W.L., & Su, Y. (2023) BioCLIP (Version v0.1) [Computer software]. https://doi.org/

Please also cite OpenCLIP:

Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., & Schmidt, L. (2021). OpenCLIP (Version v0.1) [Computer software]. https://doi.org/10.5281/zenodo.5143773

Acknowledgements

The Imageomics Institute is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) Institute program under Award #2118240 (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning).

Model Card Authors

Elizabeth G. Campolongo, Samuel Stevens, and Jiaman Wu

Model Card Contact

stevens.994@osu.edu