vllm

What're we doing here?

#2
by inflatebot - opened

This is a really weird drop, you guys. Why don't we have Transformers support? Why does the vLLM commit you link to not actually exist? The model appears to be completely unusable at the moment??
I'm not trying to be rude, but what the heck is going on?
Am I just not used to being this early?

they need to rename params to config apparently

you could just download vllm==0.6.1 and infer out of the box. The reasoning behind changing the convention from config.json to param.json is given here: https://github.com/vllm-project/vllm/pull/8168#issuecomment-2330341084

Yes at Mistral we usually only release the consolidated checkpoints to begin with (see table here). The community then usually converts the models to transformers but this takes some time and then it takes even more time to wait for the next transformers release. So yes, I think the consolidated format will surely continued to be used!

Alright, that makes sense. It's just that there's so many other loaders that are in very common use. The KoboldAI server (whose primary loader KoboldCPP is based on llama.cpp) has been pretty interested in Pixtral and it's looking at this rate like we'll be waiting for another good while. Unfortunate, but what can you do.
Thanks for the responses.

Hello,

Is it working for you ? I have installed:

  • vllm==0.6.1
  • upgraded mistral_common

I am still having the same issue with config.json file

Attempting this again on a collab - VLLM only works on Linux - which is a small % of the population trying to use these items.

Attempting this again on a collab - VLLM only works on Linux - which is a small % of the population trying to use these items.

Then perhaps they need to step up and get off that garbage windows crap. And you might be surprised on the real % out there if you get out of your bubble.

It works fine via the Docker container on Windows.

https://hub.docker.com/r/vllm/vllm-openai

Then perhaps they need to step up and get off that garbage windows crap. And you might be surprised on the real % out there if you get out of your bubble.

I main Linux, but you're kidding yourself.

Yeah, I'm closing this before it turns into polemic. Sorry!

inflatebot changed discussion status to closed

Sign up or log in to comment