vllm multi-lora deployment
#3
by
zhongwei
- opened
It is great software and model, I would deploy the Llama-3-8B-Instruct-80K-QLoRA-Merged with vllm.
Is there any solution to deploy this QLora model with base model meta-llama/Meta-Llama-3-8B-Instruct at vllm with multi-lora deployment?
Hi, I'm not familiar with vllm. I think this LoRA model is no different against any other LoRA models. Maybe you can just use the default loading method for it. But remember to set the rope_theta to 200M for the base model (i.e. meta-llama/Meta-Llama-3-8B-Instruct) when using this LoRA.