how to deploy the model?
When I run the command: vllm-sr serve -e vllm-sr/d3-flash
but got:
Usage: vllm-sr serve [OPTIONS]
Try 'vllm-sr serve --help' for help.
Error: No such option '-e'.
The -e / --engine option is only in the development channel so far. The stable release on PyPI (0.4.0) doesn't have it yet, and that version gives exactly this error. Reinstall from the dev channel:
curl -fsSL https://vllm-sr.ai/install.sh | bash -s -- --channel dev --no-launch
export PATH="$HOME/.local/bin:$PATH"
vllm-sr serve --help # should list "-e, --engine"
vllm-sr serve -e vllm-sr/d3-flash
Thanks!
I'd like to know if using vllm-sr is significantly faster than using AutoModel.from_pretrained.
@wanghaikuan currently it is simliar to each other, just for better UX. But for the roadmap, we will have some end-to-end kernal level optimizations.
To add some numbers to this, measured on one AMD Instinct MI300X with d3-flash:
| Request | AutoModel |
AutoModel without flash-linear-attention |
vllm-sr Engine mode¹ |
|---|---|---|---|
| Text, 2 questions | 34.3 ms | 51.8 ms | 40.4 ms |
| Text, 3 questions | 35.1 ms | 51.5 ms | 41.3 ms |
| Image and 2 questions | 161.2 ms | 225.6 ms | 152.2 ms |
Two things make AutoModel feel slower than it is. Without flash-linear-attention, Transformers falls back to its reference PyTorch code for the linear-attention layers, so install it on a GPU. And the first call compiles the kernels (31.7 s here), while vllm-sr does that at startup. Calling model.runtime.warmup() after loading does the same with AutoModel: 61 s once, then the first request takes 35 ms.
Median of 100 requests after a warm-up (30 without flash-linear-attention). ¹ Over HTTP, including the round trip and the router.
Thanks!