init commit

README.md

@@ -0,0 +1,99 @@
+---
+language: en
+datasets:
+- laion2b
+---
+
+# OpenFlamingo-4B (CLIP ViT-L/14, RedPajama-INCITE-Instruct-3B-v1)
+
+[Blog post]() | [Code](https://github.com/mlfoundations/open_flamingo) | [Demo]()
+
+OpenFlamingo is an open source implementation of DeepMind's [Flamingo](https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model) models. 
+This 4B-parameter model uses a [CLIP ViT-L/14](https://huggingface.co/openai/clip-vit-large-patch14) vision encoder and an instruction tuned [RedPajama-3B](https://huggingface.co/togethercomputer/RedPajama-INCITE-Instruct-3B-v1) language model.
+
+## Model Details
+We follow the Flamingo modeling paradigm, outfitting the layers of a pretrained, frozen language model such that they cross-attend to visual features when decoding. Following Flamingo, we freeze the vision encoder and language model but train the connecting modules on web-scraped image-text sequences. Specifically, we use a mixture of [LAION-2B](https://arxiv.org/abs/2210.08402) and [Multimodal C4](https://arxiv.org/abs/2304.06939).  
+
+## Uses
+OpenFlamingo models process arbitrarily interleaved sequences of images and text to output text. This allows the models to accept in-context examples and undertake tasks like captioning, visual question answering, and image classification. 
+
+### Bias, Risks, and Limitations
+OpenFlamingo models inherit the risks of their parent models, especially the language model. As an open-source research effort, we highly value open, accessible, reproducible multimodal model research; however, it is crucial to be aware that these models are trained on web data, have not been finetuned for safety, and thus may produce unintended, inappropriate, unreliable, and/or inaccurate outputs. Please use caution before deploying OpenFlamingo models in real applications. We also hope that OpenFlamingo enables further safety and reliability research to address these issues.
+
+In an effort to mitigate current potential biases and harms, we have deployed a text content filter on model outputs in the OpenFlamingo demo. We continue to red-team the model to understand and improve its safety.
+
+## Evaluation
+<table>
+  <tr>
+    <th></th>
+    <th>0-shot</th>
+    <th>4-shot</th>
+    <th>8-shot</th>
+    <th>16-shot</th>
+    <th>32-shot</th>
+  </tr>
+  <tr>
+    <th>COCO (CIDEr)</th>
+    <td>81.2 (0.3)</td>
+    <td>85.8 (0.5)</td>
+    <td>94.8 (0.2)</td>
+    <td>98.0 (0.3)</td>
+    <td>99.2 (0.3)</td>
+  </tr>
+  <tr>
+    <th>VQAv2 (Accuracy)</th>
+    <td>44.5 (0.2)</td>
+    <td>47.5 (0.3)</td>
+    <td>45.7 (0.3)</td>
+    <td>44.3 (0.6)</td>
+    <td>45.8 (0.0)</td>
+  </tr>
+  <tr>
+    <th>Flickr-30K (CIDEr)</th>
+    <td>55.6 (1.3)</td>
+    <td>61.2 (0.5)</td>
+    <td>59.0 (1.0)</td>
+    <td>54.8 (1.0)</td>
+    <td>53.0 (0.5)</td>
+  </tr>
+  <tr>
+    <th>OK-VQA (Accuracy)</th>
+    <td>29.7 (0.2)</td>
+    <td>34.3 (0.2)</td>
+    <td>32.4 (0.2)</td>
+    <td>30.7 (0.3)</td>
+    <td>32.5 (0.1)</td>
+  </tr>
+  <tr>
+    <th>TextVQA (Accuracy)</th>
+    <td>21.1 (0.4)</td>
+    <td>27.2 (0.3)</td>
+    <td>25.1 (0.2)</td>
+    <td>23.2 (0.1)</td>
+    <td>23.2 (0.2)</td>
+  </tr>
+  <tr>
+    <th>Vizwiz (Accuracy)</th>
+    <td>14.9 (0.1)</td>
+    <td>21.0 (0.4)</td>
+    <td>27.1 (1.4)</td>
+    <td>-</td>
+    <td>37.1 (0.3)</td>
+  </tr>
+  <tr>
+    <th>ImageNet (Top-1 Accuracy)</th>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+  <tr>
+    <th>Hateful Memes (ROC AUC)</th>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+    <td>-</td>
+  </tr>
+</table>

checkpoint.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:def3dbd2e3cbf471019bc5d8ee854fc644b3eca62dd3e0fc81e76dd6e0363c06
+size 15077874322