merve HF staff commited on
Commit
e69afc2
1 Parent(s): d932c54

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +33 -4
README.md CHANGED
@@ -3,7 +3,6 @@ tags:
3
  - vision
4
  - image-text-to-text
5
  ---
6
- NOTE: this model can only be used once https://github.com/huggingface/transformers/pull/29012 is merged
7
 
8
  # LLaVa-Next, leveraging [NousResearch/Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) as LLM
9
 
@@ -13,8 +12,12 @@ Disclaimer: The team releasing LLaVa-NeXT did not write a model card for this mo
13
 
14
  ## Model description
15
 
16
- LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases.
17
-
 
 
 
 
18
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62441d1d9fdefb55a0b7d12c/FPshq08TKYD0e-qwPLDVO.png)
19
 
20
  ## Intended uses & limitations
@@ -24,8 +27,34 @@ other versions on a task that interests you.
24
 
25
  ### How to use
26
 
27
- We refer to the [documentation](https://huggingface.co/transformers/main/model_doc/llava_next.html#).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
 
 
 
29
  ### BibTeX entry and citation info
30
 
31
  ```bibtex
 
3
  - vision
4
  - image-text-to-text
5
  ---
 
6
 
7
  # LLaVa-Next, leveraging [NousResearch/Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) as LLM
8
 
 
12
 
13
  ## Model description
14
 
15
+ LLaVa combines a pre-trained large language model with a pre-trained vision encoder for multimodal chatbot use cases. LLaVA 1.6 improves on LLaVA 1.5 BY:
16
+ - Using [Mistral-7B](https://mistral.ai/news/announcing-mistral-7b/) and [Nous-Hermes-2-Yi-34B](https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B) (for this checkpoint) which has better commercial licenses,
17
+ and bilingual support
18
+ - More diverse and high quality data mixture
19
+ - Dynamic high resolution
20
+
21
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62441d1d9fdefb55a0b7d12c/FPshq08TKYD0e-qwPLDVO.png)
22
 
23
  ## Intended uses & limitations
 
27
 
28
  ### How to use
29
 
30
+ Here's the prompt template for this model:
31
+ ```
32
+ "[INST] <image>\nWhat is shown in this image? [/INST]"
33
+ ```
34
+ You can load and use the model like following:
35
+ ```python
36
+ from transformers import LlavaNextProcessor, LlavaNextForConditionalGeneration
37
+ import torch
38
+ from PIL import Image
39
+ import requests
40
+
41
+ processor = LlavaNextProcessor.from_pretrained("NousResearch/Nous-Hermes-2-Yi-34B")
42
+
43
+ model = LlavaNextForConditionalGeneration.from_pretrained("NousResearch/Nous-Hermes-2-Yi-34B", torch_dtype=torch.float16, low_cpu_mem_usage=True)
44
+ model.to("cuda:0")
45
+
46
+ # prepare image and text prompt, using the appropriate prompt template
47
+ url = "https://github.com/haotian-liu/LLaVA/blob/1a91fc274d7c35a9b50b3cb29c4247ae5837ce39/images/llava_v1_5_radar.jpg?raw=true"
48
+ image = Image.open(requests.get(url, stream=True).raw)
49
+ prompt = ""<|im_start|>system\nAnswer the questions.<|im_end|><|im_start|>user\n<image>\nWhat is shown in this image?<|im_end|><|im_start|>assistant\n""
50
+
51
+ inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
52
+
53
+ # autoregressively complete prompt
54
+ output = model.generate(**inputs, max_new_tokens=100)
55
 
56
+ print(processor.decode(output[0], skip_special_tokens=True))
57
+ ```
58
  ### BibTeX entry and citation info
59
 
60
  ```bibtex