--- license: mit language: - en library_name: open_clip tags: - zero-shot-image-classification - clip - biology - CV - images - animals - species - taxonomy - rare species - endangered species - evolutionary biology - multimodal - knowledge-guided datasets: - TreeOfLife-10M - iNat21 - BIOSCAN-1M --- # Model Card for BioCLIP BioCLIP is a foundation model for the tree of life, built using CLIP architecture as a vision model for general organismal biology. It is trained on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M), our specially-created dataset covering over 450K taxa--the most biologically diverse ML-ready dataset available to date. Through rigorous benchmarking on a diverse set of fine-grained biological classification tasks, BioCLIP consistently outperformed existing baselines by 17% to 20% absolute. Through intrinsic evaluation, we found that BioCLIP learned a hierarchical representation aligned to the tree of life, which demonstrates its potential for robust generalizability. ## Model Details ### Model Description BioCLIP is based on OpenAI's [CLIP](https://openai.com/research/clip). We trained the model on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M) from OpenAI's ViT-B/16 checkpoint, using [OpenCLIP's](https://github.com/mlfoundations/open_clip) code. BioCLIP is trained with the standard CLIP objective to imbue the model with an understanding, not just of different species, but of the hierarchical structure that relates species across the tree of life. In this way, BioCLIP offers potential to aid biologists in discovery of new and related creatures, since it does not see the 454K different taxa as distinct classes, but as part of an interconnected hierarchy. - **Developed by:** Samuel Stevens, Jiaman Wu, Matthew J. Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su - **Model type:** Vision Transformer (ViT-B/16) - **License:** MIT - **Fine-tuned from model:** OpenAI CLIP, ViT-B/16 ### Model Sources - **Repository:** [BioCLIP](https://github.com/Imageomics/BioCLIP) - **Paper:** Coming Soon - **Demo:** [BioCLIP Demo](https://huggingface.co/spaces/imageomics/BioCLIP) ## Uses BioCLIP has been extensively evaluated on species classification tasks across many different subtrees of the tree of life. The ViT-B/16 vision encoder is recommended as a base model for any computer vision task for biology; we expect it to outperform general domain models with the same architecture on biology-specific tasks. ### Direct Use See the demo [here](https://huggingface.co/spaces/imageomics/BioCLIP) for examples of zero-shot classification. It can also be used in a few-shot setting with a KNN; please see [our paper]() for details for both few-shot and zero-shot settings without fine-tuning. ## Bias, Risks, and Limitations This model was developed from the original CLIP model, thus many of the concerns discussed in ([Radford et al. 2021](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf)) apply. We encourage the concerned/curious user to read their extensive ethics statement, while we focus our attention on the biological perspective which is unique to BioCLIP. - No specific geographic information (eg., GPS coordinates) are included in training, so the species classification does not pose a direct threat to animals through aiding poachers, as it cannot inform them of their location. - BioCLIP is designed to aid in scientific discovery through an association of images to the hierarchical taxonomy structure. As with many--if not all--models currently in production, it is important to retain the context that it is meant to assist biologists in their work, not replace them. As such, we caution against over-reliance on model predictions. ### Recommendations Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations. ## How to Get Started with the Model BioCLIP can be used with the `open_clip` library: ```py import open_clip model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:imageomics/bioclip') tokenizer = open_clip.get_tokenizer('hf-hub:imageomics/bioclip') ``` ## Training Details ### Compute Infrastructure Training was performed on 8 NVIDIA A100-80GB GPUs distributed over 2 nodes on [OSC's](https://www.osc.edu/) Ascend HPC Cluster with global batch size 32,768 for 4 days. Based on [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://doi.org/10.48550/arXiv.1910.09700), that's 132.71 kg of CO2 eq., or 536km driven by an average ICE car. ### Training Data This model was trained on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M), which is a compilation of images matched to [Linnaean taxonomic rank](https://www.britannica.com/science/taxonomy/The-objectives-of-biological-classification) from kingdom through species. They are also matched with common (vernacular) name of the subject of the image where available. For more information, please see our dataset, [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M). ### Training Hyperparameters - **Training regime:** fp16 mixed precision. We resize images to 224 x 224 pixels. We use a maximum learning rate of 1e4 with 1000 linear warm-up steps, then use cosine decay to 0 over 100 epochs. We also use a weight decay of 0.2 and a batch size of 32K. ## Evaluation ### Testing Data We tested BioCLIP on the following collection of 10 biologically-relevant tasks. - [Meta-Album](https://paperswithcode.com/dataset/meta-album): Specifically, we used the Plankton, Insects, Insects 2, PlantNet, Fungi, PlantVillage, Medicinal Leaf, and PlantDoc datasets from Set-0 through Set-2 (Set-3 was still not released as of our publication/evaluation (Nov. 2023). - [Birds 525](https://www.kaggle.com/datasets/gpiosenka/100-bird-species): We evaluated on the 2,625 test images provided with the dataset. - [Rare Species](https://huggingface.co/datasets/imageomics/rare-species): A new dataset we curated for the purpose of testing this model and to contribute to the ML for Conservation community. It consists of 400 species labeled Near Threatened through Extinct in the Wild by the [IUCN Red List](https://www.iucnredlist.org/), with 30 images per species. For more information, see our dataset, [Rare Species](https://huggingface.co/datasets/imageomics/rare-species). For more information about the contents of these datasets, see Table 2 and associated sections of [our paper](). ### Metrics We use top-1 and top-5 accuracy to evaluate models, and validation loss to choose the best performing checkpoints from training. ### Results We compare BioCLIP to OpenAI's CLIP and OpenCLIP's LAION-2B checkpoint. Here are the zero-shot classification results on our benchmark tasks. Please see [our paper]() for few-shot results.
Model Animals Plants & Fungi Rare Species Mean
Birds 525 Plankton Insects Insects 2 PlantNet Fungi PlantVillage Med. Leaf PlantDoc
CLIP 49.9 3.2 9.1 9.8 58.5 10.2 5.4 15.9 26.1 26.6 21.4
OpenCLIP 54.7 2.2 6.5 9.6 50.2 5.7 8.0 12.4 25.8 31.0 20.6
BioCLIP 74.7 5.4 32.7 21.2 91.0 51.8 24.0 48.1 27.5 39.2 41.5
iNat21 Only 55.7 2.7 29.9 12.0 89.3 42.7 16.4 22.2 18.8 19.4 30.9
### Summary BioCLIP outperforms general-domain baselines by 18% on average. ### Model Examination We encourage readers to see Section 4.6 of [our paper](). In short, BioCLIP forms representations that more closely align to the taxonomic hierarchy compared to general-domain baselines like CLIP or OpenCLIP. ## Citation **BibTeX:** ``` @software{bioclip2023, author = {Samuel Stevens and Jiaman Wu and Matthew J. Thompson and Elizabeth G. Campolongo and Chan Hee Song and David Edward Carlyn and Li Dong and Wasila M. Dahdul and Charles Stewart and Tanya Berger-Wolf and Wei-Lun Chao and Yu Su}, doi = {}, month = nov, title = {BioCLIP}, version = {v0.1}, year = {2023} } ``` Please also cite our paper: (Coming Soon) Please also cite OpenCLIP: ``` @software{ilharco2021openclip, author = {Ilharco, Gabriel and Wortsman, Mitchell and Wightman, Ross and Gordon, Cade and Carlini, Nicholas and Taori, Rohan and Dave, Achal and Shankar, Vaishaal and Namkoong, Hongseok and Miller, John and Hajishirzi, Hannaneh and Farhadi, Ali and Schmidt, Ludwig}, doi = {10.5281/zenodo.5143773}, month = jul, title = {{OpenCLIP}}, version = {v0.1}, year = {2021} } ``` **APA:** Stevens, S, Wu, J., Thompson, M. J., Campolongo, E. G., Song, C. H., Carlyn, D. E., Dong, L., Dahdul, W. M., Stewart, C., Berger-Wolf, T., Chao, W.L., & Su, Y. (2023) BioCLIP (Version v0.1) [Computer software]. https://doi.org/ Please also cite OpenCLIP: Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., & Schmidt, L. (2021). OpenCLIP (Version v0.1) [Computer software]. https://doi.org/10.5281/zenodo.5143773 ## Acknowledgements The [Imageomics Institute](https://imageomics.org) is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) Institute program under [Award #2118240](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2118240) (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning). ## Model Card Authors Elizabeth G. Campolongo, Samuel Stevens, and Jiaman Wu ## Model Card Contact [stevens.994@osu.edu](mailto:stevens.994@osu.edu)