|
--- |
|
tags: |
|
- vision |
|
- image-text-to-text |
|
--- |
|
NOTE: this model can only be used once https://github.com/huggingface/transformers/pull/29012 is merged |
|
|
|
# LLaVa-Next, leveraging [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) as LLM |
|
|
|
The LLaVA-NeXT model was proposed in [LLaVA-NeXT: Improved reasoning, OCR, and world knowledge](https://llava-vl.github.io/blog/2024-01-30-llava-next/) by Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, Yong Jae Lee. LLaVa-NeXT (also called LLaVa-1.6) improves upon [LLaVa-1.5](https://huggingface.co/transformers/main/model_doc/llava.html) by increasing the input image resolution and training on an improved visual instruction tuning dataset to improve OCR and common sense reasoning. |
|
|
|
Disclaimer: The team releasing LLaVa-NeXT did not write a model card for this model so this model card has been written by the Hugging Face team. |
|
|
|
## Model description |
|
|
|
LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases. |
|
|
|
 |
|
|
|
## Intended uses & limitations |
|
|
|
You can use the raw model for tasks like image captioning, visual question answering, multimodal chatbot use cases. See the [model hub](https://huggingface.co/models?search=llava-hf) to look for |
|
other versions on a task that interests you. |
|
|
|
### How to use |
|
|
|
We refer to the [documentation](https://huggingface.co/transformers/main/model_doc/llava_next.html#). |
|
|
|
### BibTeX entry and citation info |
|
|
|
```bibtex |
|
@misc{liu2023improved, |
|
title={Improved Baselines with Visual Instruction Tuning}, |
|
author={Haotian Liu and Chunyuan Li and Yuheng Li and Yong Jae Lee}, |
|
year={2023}, |
|
eprint={2310.03744}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CV} |
|
} |
|
``` |
