Update README.md

README.md

@@ -3,7 +3,6 @@ tags:
 - vision
 - image-text-to-text
 ---
-NOTE: this model can only be used once https://github.com/huggingface/transformers/pull/29012 is merged
 
 # LLaVa-Next, leveraging [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) as LLM
 
@@ -13,8 +12,12 @@ Disclaimer: The team releasing LLaVa-NeXT did not write a model card for this mo
 
 ## Model description
 
-LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases.
-
+LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases. LLaVA 1.6 improves on LLaVA 1.5 BY:
+- Using [Mistral-7B](https://mistral.ai/news/announcing-mistral-7b/) (for this checkpoint) and [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) which has better commercial licenses,
+  and bilingual support
+- More diverse and high quality data mixture
+- Dynamic high resolution
+  
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62441d1d9fdefb55a0b7d12c/FPshq08TKYD0e-qwPLDVO.png)
 
 ## Intended uses & limitations
@@ -24,7 +27,34 @@ other versions on a task that interests you.
 
 ### How to use
 
-We refer to the [documentation](https://huggingface.co/transformers/main/model_doc/llava_next.html#).
+Here's the prompt template for this model:
+```
+"<|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n<image>\nWhat is shown in this image?<|im_end|><|im_start|>assistant\n"
+```
+You can load and use the model like following:
+```python
+from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
+import torch
+from PIL import Image
+import requests
+
+processor = LlavaNextProcessor.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf")
+
+model = LlavaNextForConditionalGeneration.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True) 
+model.to("cuda:0")
+
+# prepare image and text prompt, using the appropriate prompt template
+url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
+image = Image.open(requests.get(url, stream=True).raw)
+prompt = "[INST] <image>\nWhat is shown in this image? [/INST]"
+
+inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
+
+# autoregressively complete prompt
+output = model.generate(**inputs, max_new_tokens=100)
+
+print(processor.decode(output[0], skip_special_tokens=True))
+```
 
 ### BibTeX entry and citation info