File size: 11,730 Bytes
a14378a
85baf1d
 
 
 
a14378a
 
 
85baf1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a14378a
85baf1d
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
---
license: mit
language:
- en
library_name: open_clip
tags:
- zero-shot-image-classification
- clip
- biology
- CV
- images
- animals
- species
- taxonomy
- rare species
- endangered species
- evolutionary biology
- multimodal
- knowledge-guided
datasets:
- TreeOfLife-10M
- iNat21
- BIOSCAN-1M
---


# Model Card for BioCLIP

<!-- 
This modelcard has been generated using [this raw template](https://github.com/huggingface/huggingface_hub/blob/main/src/huggingface_hub/templates/modelcard_template.md?plain=1). And further altered to suit Imageomics Institute needs -->

BioCLIP is a foundation model for the tree of life, built using CLIP architecture as a vision model for general organismal biology. 
It is trained on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M), our specially-created dataset covering over 450K taxa--the most biologically diverse ML-ready dataset available to date. 
Through rigorous benchmarking on a diverse set of fine-grained biological classification tasks, BioCLIP consistently outperformed existing baselines by 17% to 20% absolute. 
Through intrinsic evaluation, we found that BioCLIP learned a hierarchical representation aligned to the tree of life, which demonstrates its potential for robust generalizability.

## Model Details

### Model Description

BioCLIP is based on OpenAI's [CLIP](https://openai.com/research/clip). 
We trained the model on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M) from OpenAI's ViT-B/16 checkpoint, using [OpenCLIP's](https://github.com/mlfoundations/open_clip) code.
BioCLIP is trained with the standard CLIP objective to imbue the model with an understanding, not just of different species, but of the hierarchical structure that relates species across the tree of life. 
In this way, BioCLIP offers potential to aid biologists in discovery of new and related creatures, since it does not see the 454K different taxa as distinct classes, but as part of an interconnected hierarchy. 


- **Developed by:** Samuel Stevens, Jiaman Wu, Matthew J. Thompson, Elizabeth G. Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M. Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, and Yu Su
- **Model type:** Vision Transformer (ViT-B/16)
- **License:** MIT
- **Fine-tuned from model:** OpenAI CLIP, ViT-B/16

### Model Sources

- **Repository:** [BioCLIP](https://github.com/Imageomics/BioCLIP)
- **Paper:** Coming Soon
- **Demo:** [BioCLIP Demo](https://huggingface.co/spaces/imageomics/BioCLIP)

## Uses

BioCLIP has been extensively evaluated on species classification tasks across many different subtrees of the tree of life.
The ViT-B/16 vision encoder is recommended as a base model for any computer vision task for biology; we expect it to outperform general domain models with the same architecture on biology-specific tasks.


### Direct Use

See the demo [here](https://huggingface.co/spaces/imageomics/BioCLIP) for examples of zero-shot classification.
It can also be used in a few-shot setting with a KNN; please see [our paper]() for details for both few-shot and zero-shot settings without fine-tuning.


## Bias, Risks, and Limitations

This model was developed from the original CLIP model, thus many of the concerns discussed in ([Radford et al. 2021](https://proceedings.mlr.press/v139/radford21a/radford21a.pdf)) apply. 
We encourage the concerned/curious user to read their extensive ethics statement, while we focus our attention on the biological perspective which is unique to BioCLIP. 
 - No specific geographic information (eg., GPS coordinates) are included in training, so the species classification does not pose a direct threat to animals through aiding poachers, as it cannot inform them of their location.
 - BioCLIP is designed to aid in scientific discovery through an association of images to the hierarchical taxonomy structure. As with many--if not all--models currently in production, it is important to retain the context that it is meant to assist biologists in their work, not replace them. As such, we caution against over-reliance on model predictions.

### Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. 
More information needed for further recommendations.

## How to Get Started with the Model

BioCLIP can be used with the `open_clip` library:

```py
import open_clip

model, preprocess_train, preprocess_val = open_clip.create_model_and_transforms('hf-hub:imageomics/bioclip')
tokenizer = open_clip.get_tokenizer('hf-hub:imageomics/bioclip')
```

## Training Details

### Compute Infrastructure

Training was performed on 8 NVIDIA A100-80GB GPUs distributed over 2 nodes on [OSC's](https://www.osc.edu/) Ascend HPC Cluster with global batch size 32,768 for 4 days.

Based on [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://doi.org/10.48550/arXiv.1910.09700), that's 132.71 kg of CO<sub>2</sub> eq., or 536km driven by an average ICE car.

### Training Data

This model was trained on [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M), which is a compilation of images matched to [Linnaean taxonomic rank](https://www.britannica.com/science/taxonomy/The-objectives-of-biological-classification) from kingdom through species. They are also matched with common (vernacular) name of the subject of the image where available. For more information, please see our dataset, [TreeOfLife-10M](https://huggingface.co/datasets/imageomics/TreeOfLife-10M).

### Training Hyperparameters

- **Training regime:** fp16 mixed precision.

We resize images to 224 x 224 pixels.
We use a maximum learning rate of 1e4 with 1000 linear warm-up steps, then use cosine decay to 0 over 100 epochs.
We also use a weight decay of 0.2 and a batch size of 32K.

## Evaluation

### Testing Data

We tested BioCLIP on the following collection of 10 biologically-relevant tasks.
 - [Meta-Album](https://paperswithcode.com/dataset/meta-album): Specifically, we used the Plankton, Insects, Insects 2, PlantNet, Fungi, PlantVillage, Medicinal Leaf, and PlantDoc datasets from Set-0 through Set-2 (Set-3 was still not released as of our publication/evaluation (Nov. 2023).
 - [Birds 525](https://www.kaggle.com/datasets/gpiosenka/100-bird-species): We evaluated on the 2,625 test images provided with the dataset.
 - [Rare Species](https://huggingface.co/datasets/imageomics/rare-species): A new dataset we curated for the purpose of testing this model and to contribute to the ML for Conservation community. It consists of 400 species labeled Near Threatened through Extinct in the Wild by the [IUCN Red List](https://www.iucnredlist.org/), with 30 images per species. For more information, see our dataset, [Rare Species](https://huggingface.co/datasets/imageomics/rare-species).

For more information about the contents of these datasets, see Table 2 and associated sections of [our paper]().

### Metrics

We use top-1 and top-5 accuracy to evaluate models, and validation loss to choose the best performing checkpoints from training.

### Results

We compare BioCLIP to OpenAI's CLIP and OpenCLIP's LAION-2B checkpoint.
Here are the zero-shot classification results on our benchmark tasks. 
Please see [our paper]() for few-shot results.

<table cellpadding="0" cellspacing="0">
  <thead>
    <tr>
      <th rowspan="2">Model</th>
      <th colspan="4">Animals</th>
      <th colspan="5">Plants & Fungi</th>
      <th rowspan="2">Rare Species</th>
      <th rowspan="2">Mean</th>
    </tr>
    <tr>
      <th>Birds 525</th>
      <th>Plankton</th>
      <th>Insects</th>
      <th>Insects 2</th>
      <th>PlantNet</th>
      <th>Fungi</th>
      <th>PlantVillage</th>
      <th>Med. Leaf</th>
      <th>PlantDoc</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>CLIP</td>
      <td>49.9</td>
      <td>3.2</td>
      <td>9.1</td>
      <td>9.8</td>
      <td>58.5</td>
      <td>10.2</td>
      <td>5.4</td>
      <td>15.9</td>
      <td>26.1</td>
      <td>26.6</td>
      <td>21.4</td>
    </tr>
    <tr>
      <td>OpenCLIP</td>
      <td>54.7</td>
      <td>2.2</td>
      <td>6.5</td>
      <td>9.6</td>
      <td>50.2</td>
      <td>5.7</td>
      <td>8.0</td>
      <td>12.4</td>
      <td>25.8</td>
      <td>31.0</td>
      <td>20.6</td>
    </tr>
    <tr>
      <td>BioCLIP</td>
      <td><b>74.7</b></td>
      <td><b>5.4</b></td>
      <td><b>32.7</b></td>
      <td><b>21.2</b></td>
      <td><b>91.0</b></td>
      <td><b>51.8</b></td>
      <td><b>24.0</b></td>
      <td><b>48.1</b></td>
      <td><b>27.5</b></td>
      <td><b>39.2</b></td>
      <td><b>41.5</b></td>
    </tr>
    <tr>
      <td>iNat21 Only</td>
      <td>55.7</td>
      <td>2.7</td>
      <td>29.9</td>
      <td>12.0</td>
      <td>89.3</td>
      <td>42.7</td>
      <td>16.4</td>
      <td>22.2</td>
      <td>18.8</td>
      <td>19.4</td>
      <td>30.9</td>
    </tr>
  </tbody>
</table>


### Summary

BioCLIP outperforms general-domain baselines by 18% on average.

### Model Examination

We encourage readers to see Section 4.6 of [our paper]().
In short, BioCLIP forms representations that more closely align to the taxonomic hierarchy compared to general-domain baselines like CLIP or OpenCLIP.


## Citation 

**BibTeX:**

```
@software{bioclip2023,
  author = {Samuel Stevens and Jiaman Wu and Matthew J. Thompson and Elizabeth G. Campolongo and Chan Hee Song and David Edward Carlyn and Li Dong and Wasila M. Dahdul and Charles Stewart and Tanya Berger-Wolf and Wei-Lun Chao and Yu Su},
  doi = {<doi once generated>},
  month = nov,
  title = {BioCLIP},
  version = {v0.1},
  year = {2023}
}
```

Please also cite our paper: (Coming Soon)
<!--
```
@article{stevens2023bioclip,
  title    = "BioCLIP: A Vision Foundation Model for the Tree of Life",
  author   = "Samuel Stevens and Jiaman Wu and Matthew J. Thompson and Elizabeth G. Campolongo and Chan Hee Song and David Edward Carlyn and Li Dong and Wasila M. Dahdul and Charles Stewart and Tanya Berger-Wolf and Wei-Lun Chao and Yu Su",
  journal  = "<journal_name>",
  year     =  <year>,
  url      = "<DOI_URL>",
  doi      = "<DOI>"
}
```
-->

Please also cite OpenCLIP:
```
@software{ilharco2021openclip,
  author = {Ilharco, Gabriel and Wortsman, Mitchell and Wightman, Ross and Gordon, Cade and Carlini, Nicholas and Taori, Rohan and Dave, Achal and Shankar, Vaishaal and Namkoong, Hongseok and Miller, John and Hajishirzi, Hannaneh and Farhadi, Ali and Schmidt, Ludwig},
  doi = {10.5281/zenodo.5143773},
  month = jul,
  title = {{OpenCLIP}},
  version = {v0.1},
  year = {2021}
}
```

**APA:**

Stevens, S, Wu, J., Thompson, M. J., Campolongo, E. G., Song, C. H., Carlyn, D. E., Dong, L., Dahdul, W. M., Stewart, C., Berger-Wolf, T., Chao, W.L., & Su, Y. (2023) BioCLIP (Version v0.1) [Computer software]. https://doi.org/

Please also cite OpenCLIP:

Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Carlini, N., Taori, R., Dave, A., Shankar, V., Namkoong, H., Miller, J., Hajishirzi, H., Farhadi, A., & Schmidt, L. (2021). OpenCLIP (Version v0.1) [Computer software]. https://doi.org/10.5281/zenodo.5143773

## Acknowledgements

The [Imageomics Institute](https://imageomics.org) is funded by the US National Science Foundation's Harnessing the Data Revolution (HDR) Institute program under [Award #2118240](https://www.nsf.gov/awardsearch/showAward?AWD_ID=2118240) (Imageomics: A New Frontier of Biological Information Powered by Knowledge-Guided Machine Learning).


## Model Card Authors

Elizabeth G. Campolongo, Samuel Stevens, and Jiaman Wu

## Model Card Contact

[stevens.994@osu.edu](mailto:stevens.994@osu.edu)