Update README.md

README.md

@@ -3,7 +3,6 @@ tags:
 - vision
 - image-text-to-text
 ---
-NOTE: this model can only be used once https://github.com/huggingface/transformers/pull/29012 is merged
 
 # LLaVa-Next, leveraging [liuhaotian/llava-v1.6-vicuna-7b](https://huggingface.co/liuhaotian/llava-v1.6-vicuna-7b) as LLM
 
@@ -13,8 +12,10 @@ Disclaimer: The team releasing LLaVa-NeXT did not write a model card for this mo
 
 ## Model description
 
-LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases.
-
+LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases. LLaVA 1.6 improves on LLaVA 1.5 BY:
+- More diverse and high quality data mixture
+- Dynamic high resolution
+  
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62441d1d9fdefb55a0b7d12c/FPshq08TKYD0e-qwPLDVO.png)
 
 ## Intended uses & limitations
@@ -24,7 +25,35 @@ other versions on a task that interests you.
 
 ### How to use
 
-We refer to the [documentation](https://huggingface.co/transformers/main/model_doc/llava_next.html#).
+Here's the prompt template for this model:
+```
+"A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\nWhat is shown in this image? ASSISTANT:"
+```
+You can load and use the model like following:
+```python
+from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
+import torch
+from PIL import Image
+import requests
+
+processor = LlavaNextProcessor.from_pretrained("llava-v1.6-vicuna-7b-hf")
+
+model = LlavaNextForConditionalGeneration.from_pretrained("llava-v1.6-vicuna-7b-hf", torch_dtype=torch.float16, low_cpu_mem_usage=True) 
+model.to("cuda:0")
+
+# prepare image and text prompt, using the appropriate prompt template
+url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
+image = Image.open(requests.get(url, stream=True).raw)
+prompt = "A chat between a curious human and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the human's questions. USER: <image>\nWhat is shown in this image? ASSISTANT:"
+
+
+inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
+
+# autoregressively complete prompt
+output = model.generate(**inputs, max_new_tokens=100)
+
+print(processor.decode(output[0], skip_special_tokens=True))
+```
 
 ### BibTeX entry and citation info