File size: 1,690 Bytes
ed340d9 e1fbeb3 7f7df37 e1fbeb3 7f7df37 e1fbeb3 7f7df37 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
---
license: bsd-3-clause
language:
- en
pipeline_tag: visual-question-answering
library_name: transformers
---
<br>
<br>
# LoViM Model Card
## Model details
**Model type:**
LoViM is an open-source Vision-Languagde model trained by initializing from InstructBLIP and alignment with Vicuna on multimodal instruction-finetuning data.
It composes of an EVA-CLIP vision encoder, a Q-Former, a projection layer and an auto-regressive language model, based on the decoder only transformer architecture.
**Model date:**
LoViM_FlanT5 was trained in July 2023.
**Paper or resources for more information:**
https://gordonhu608.github.io/lovim/
**License:**
BSD 3-Clause License
**Where to send questions or comments about the model:**
https://github.com/mlpc-ucsd/LoViM
## Intended use
**Primary intended uses:**
The primary use of LoViM FlanT5 is for commercial use on large multimodal models.
**Primary intended users:**
The primary intended users of this model is for commercial companies in computer vision, natural language processing, machine learning, and artificial intelligence.
## Training dataset
Pre-train data: 558K filtered image-text pairs from LAION,CC-3M, and SBU. Selected by LLaVA.
Instruction-finetuning data: COCO-Caption, TextCaps, VQAv2, OKVQA, A-OKVQA, LLaVA-150K, OCR-VQA.
## Evaluation dataset
For zero-shot evaluation on general image task, we selected Nocaps, Flickr30K, VizWiz, Visual Spaial Reasoning (VSR), IconQA, Visual Dialog, ScienceQA, MSRVTT QA, TextVQA and Hateful Memes.
For zero-shot evaluation on text-rich image OCR task, we selected ST-VQA, OCR-VQA, Text-VQA, and Doc-VQA.
More detials are in our github, https://github.com/mlpc-ucsd/LoViM |