mistralai

/

Pixtral-12B-2409

like 522

Follow

Mistral AI_ 3,077

they need to rename params to config apparently

you could just download vllm==0.6.1 and infer out of the box. The reasoning behind changing the convention from config.json to param.json is given here: https://github.com/vllm-project/vllm/pull/8168#issuecomment-2330341084

Yes at Mistral we usually only release the consolidated checkpoints to begin with (see table here). The community then usually converts the models to transformers but this takes some time and then it takes even more time to wait for the next transformers release. So yes, I think the consolidated format will surely continued to be used!

Alright, that makes sense. It's just that there's so many other loaders that are in very common use. The KoboldAI server (whose primary loader KoboldCPP is based on llama.cpp) has been pretty interested in Pixtral and it's looking at this rate like we'll be waiting for another good while. Unfortunate, but what can you do.
Thanks for the responses.

Hello,

Is it working for you ? I have installed:

• vllm==0.6.1
• upgraded mistral_common

I am still having the same issue with config.json file

Attempting this again on a collab - VLLM only works on Linux - which is a small % of the population trying to use these items.

Attempting this again on a collab - VLLM only works on Linux - which is a small % of the population trying to use these items.

Then perhaps they need to step up and get off that garbage windows crap. And you might be surprised on the real % out there if you get out of your bubble.

It works fine via the Docker container on Windows.

https://hub.docker.com/r/vllm/vllm-openai

Then perhaps they need to step up and get off that garbage windows crap. And you might be surprised on the real % out there if you get out of your bubble.

I main Linux, but you're kidding yourself.

Yeah, I'm closing this before it turns into polemic. Sorry!