|
--- |
|
license: apache-2.0 |
|
library_name: vllm |
|
--- |
|
|
|
# Pixtral-12B-0910 |
|
|
|
> [!WARNING] |
|
> We still need to validate official evaluations with the below usage example. |
|
|
|
|
|
...TODO |
|
|
|
## Usage (VLLM) |
|
|
|
We recommend using Pixtral with the [vLLM library](https://github.com/vllm-project/vllm). |
|
|
|
### Installation |
|
|
|
**Important**: Make sure you install `vLLM >= v0.6.1.post1`: |
|
|
|
``` |
|
pip install --upgrade vllm |
|
``` |
|
|
|
Also make sure you have `mistral_common >= 1.4.0` installed: |
|
|
|
``` |
|
pip install --upgrade mistral_common |
|
``` |
|
|
|
You can also make use of a ready-to-go [docker image](https://hub.docker.com/layers/vllm/vllm-openai/latest/images/sha256-de9032a92ffea7b5c007dad80b38fd44aac11eddc31c435f8e52f3b7404bbf39?context=explore). |
|
|
|
**_Simple Example_** |
|
|
|
```py |
|
from vllm import LLM |
|
from vllm.sampling_params import SamplingParams |
|
|
|
model_name = "mistralai/Pixtral-12B-2409" |
|
|
|
sampling_params = SamplingParams(max_tokens=8192) |
|
|
|
llm = LLM(model=model_name, tokenizer_mode="mistral") |
|
|
|
prompt = "Describe this image in one sentence." |
|
image_url = "https://picsum.photos/id/237/200/300" |
|
|
|
messages = [ |
|
{ |
|
"role": "user", |
|
"content": [{"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": image_url}}] |
|
}, |
|
] |
|
|
|
outputs = llm.chat(messages, sampling_params=sampling_params) |
|
|
|
print(outputs[0].outputs[0].text) |
|
``` |
|
|
|
**_Advanced Example_** |
|
|
|
You can also pass multiple images per message and/or pass multi-turn conversations |
|
|
|
```py |
|
from vllm import LLM |
|
from vllm.sampling_params import SamplingParams |
|
|
|
model_name = "mistralai/Pixtral-12B-2409" |
|
max_img_per_msg = 5 |
|
|
|
sampling_params = SamplingParams(max_tokens=8192, temperature=0.7) |
|
|
|
# Lower max_num_seqs or max_model_len on low-VRAM GPUs. |
|
llm = LLM(model=model_name, tokenizer_mode="mistral", limit_mm_per_prompt={"image": max_img_per_msg}, max_model_len=32768) |
|
|
|
prompt = "Describe the following image." |
|
|
|
url_1 = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png" |
|
url_2 = "https://picsum.photos/seed/picsum/200/300" |
|
url_3 = "https://picsum.photos/id/32/512/512" |
|
|
|
messages = [ |
|
{ |
|
"role": "user", |
|
"content": [{"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": url_1}}, {"type": "image_url", "image_url": {"url": url_2}}], |
|
}, |
|
{ |
|
"role": "assistant", |
|
"content": "The images shows nature.", |
|
}, |
|
{ |
|
"role": "user", |
|
"content": "More details please and answer only in French!." |
|
}, |
|
{ |
|
"role": "user", |
|
"content": [{"type": "image_url", "image_url": {"url": url_3}}], |
|
} |
|
] |
|
|
|
outputs = llm.chat(messages=messages, sampling_params=sampling_params) |
|
print(outputs[0].outputs[0].text) |
|
``` |
|
|
|
You can find more examples and tests directly in vLLM. |
|
- [Examples](https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_pixtral.py) |
|
- [Tests](https://github.com/vllm-project/vllm/blob/main/tests/models/test_pixtral.py) |
|
|
|
**_Server_** |
|
|
|
You can also use pixtral in a server/client setting. |
|
|
|
1. Spin up a server: |
|
|
|
``` |
|
vllm serve mistralai/Pixtral-12B-2409 --tokenizer_mode mistral --limit_mm_per_prompt 'image=4' |
|
``` |
|
|
|
2. And ping the client: |
|
|
|
``` |
|
curl --location 'http://<your-node-url>:8000/v1/chat/completions' \ |
|
--header 'Content-Type: application/json' \ |
|
--header 'Authorization: Bearer token' \ |
|
--data '{ |
|
"model": "mistralai/Pixtral-12B-2409", |
|
"messages": [ |
|
{ |
|
"role": "user", |
|
"content": [ |
|
{"type" : "text", "text": "Describe this image in detail please."}, |
|
{"type": "image_url", "image_url": {"url": "https://s3.amazonaws.com/cms.ipressroom.com/338/files/201808/5b894ee1a138352221103195_A680%7Ejogging-edit/A680%7Ejogging-edit_hero.jpg"}}, |
|
{"type" : "text", "text": "and this one as well. Answer in French."}, |
|
{"type": "image_url", "image_url": {"url": "https://www.wolframcloud.com/obj/resourcesystem/images/a0e/a0ee3983-46c6-4c92-b85d-059044639928/6af8cfb971db031b.png"}} |
|
] |
|
} |
|
] |
|
}' |
|
``` |
