|
--- |
|
language: en |
|
datasets: |
|
- laion2b |
|
--- |
|
|
|
# OpenFlamingo-9B (CLIP ViT-L/14, MPT-7B) |
|
|
|
[Blog post]() | [Code](https://github.com/mlfoundations/open_flamingo) | [Demo]() |
|
|
|
OpenFlamingo is an open source implementation of DeepMind's [Flamingo](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model) models. |
|
This 9B-parameter model uses a [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14) vision encoder and [MPT-7B](https://huggingface.co/mosaicml/mpt-7b) language model. |
|
|
|
## Model Details |
|
We follow the Flamingo modeling paradigm, outfitting the layers of a pretrained, frozen language model such that they cross-attend to visual features when decoding. Following Flamingo, we freeze the vision encoder and language model but train the connecting modules on web-scraped image-text sequences. Specifically, we use a mixture of [LAION-2B](https://arxiv.org/abs/2210.08402) and [Multimodal C4](https://arxiv.org/abs/2304.06939). |
|
|
|
## Uses |
|
OpenFlamingo models process arbitrarily interleaved sequences of images and text to output text. This allows the models to accept in-context examples and undertake tasks like captioning, visual question answering, and image classification. |
|
|
|
### Bias, Risks, and Limitations |
|
OpenFlamingo models inherit the risks of their parent models, especially the language model. As an open-source research effort, we highly value open, accessible, reproducible multimodal model research; however, it is crucial to be aware that these models are trained on web data, have not been finetuned for safety, and thus may produce unintended, inappropriate, unreliable, and/or inaccurate outputs. Please use caution before deploying OpenFlamingo models in real applications. We also hope that OpenFlamingo enables further safety and reliability research to address these issues. |
|
|
|
In an effort to mitigate current potential biases and harms, we have deployed a text content filter on model outputs in the OpenFlamingo demo. We continue to red-team the model to understand and improve its safety. |
|
|
|
## Evaluation |
|
<table> |
|
<tr> |
|
<th></th> |
|
<th>0-shot</th> |
|
<th>4-shot</th> |
|
<th>8-shot</th> |
|
<th>16-shot</th> |
|
<th>32-shot</th> |
|
</tr> |
|
<tr> |
|
<th>COCO (CIDEr)</th> |
|
<td>79.5 (0.2)</td> |
|
<td>89.0 (0.3)</td> |
|
<td>96.3 (0.1)</td> |
|
<td>98.8 (0.7)</td> |
|
<td>99.5 (0.1)</td> |
|
</tr> |
|
<tr> |
|
<th>VQAv2 (Accuracy)</th> |
|
<td>48.3 (0.1)</td> |
|
<td>49.4 (0.4)</td> |
|
<td>51.8 (0.4)</td> |
|
<td>51.3 (0.5)</td> |
|
<td>50.2 (0.6)</td> |
|
</tr> |
|
<tr> |
|
<th>Flickr-30K (CIDEr)</th> |
|
<td>59.5 (1.0)</td> |
|
<td>65.8 (0.6)</td> |
|
<td>62.9 (1.0)</td> |
|
<td>62.8 (1.0)</td> |
|
<td>61.3 (0.7)</td> |
|
</tr> |
|
<tr> |
|
<th>OK-VQA (Accuracy)</th> |
|
<td>34.7 (0.1)</td> |
|
<td>34.3 (0.1)</td> |
|
<td>38.4 (0.0)</td> |
|
<td>39.5 (0.1)</td> |
|
<td>38.1 (0.0)</td> |
|
</tr> |
|
<tr> |
|
<th>TextVQA (Accuracy)</th> |
|
<td>24.2 (0.5)</td> |
|
<td>28.2 (0.4)</td> |
|
<td>29.1 (0.1)</td> |
|
<td>27.3 (0.1)</td> |
|
<td>23.8 (0.2)</td> |
|
</tr> |
|
<tr> |
|
<th>Vizwiz (Accuracy)</th> |
|
<td>17.7 (0.7)</td> |
|
<td>23.1 (0.9)</td> |
|
<td>31.6 (1.5)</td> |
|
<td>38.0 (1.1)</td> |
|
<td>40.2 (0.7)</td> |
|
</tr> |
|
<tr> |
|
<th>ImageNet (Top-1 Accuracy)</th> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
</tr> |
|
<tr> |
|
<th>Hateful Memes (ROC AUC)</th> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
<td>-</td> |
|
</tr> |
|
</table |
|
|
|
|
|
|
