Serving with vllm-sr
#2
by xfalcox - opened
Why this model doesn't have instruction on how to serve with vllm-sr ?
Thanks for asking. d3-ultra does run on the vLLM Semantic Router model runtime; the card is missing that section. One catch for now: the runtime's built-in entry for d3-ultra still points at a pre-release revision that is no longer on the Hub, so vllm-srun serve vllm-sr/d3-ultra without a revision fails with "Revision Not Found". Until that is fixed, pass the current revision explicitly:
pip install -e "src/model-runtime[multimodal]" flash-linear-attention # in a checkout of vllm-project/semantic-router, main
vllm-srun serve vllm-sr/d3-ultra --revision 52ba9e7a00402f9560c7fba8c039608e7ddee7fb --port 8899 --max-request-bytes 67108864
We will fix the built-in entry and add the serving section to the card.