sanketrai/qwen-2.5-coder-7b-sql-create-context-qlora
Updated
Sanket Rai
sanketrai
·
AI & ML interests
NLP, CV, RL, Deep Learning, Gen AI, , MLOps
Recent Activity
updated
a model
13 days ago
sanketrai/qwen-2.5-coder-7b-sql-create-context-qlora
published
a model
13 days ago
sanketrai/qwen-2.5-coder-7b-sql-create-context-qlora
updated
a model
25 days ago
sanketrai/starcoder-1b-hf-stack-v1-lora
Organizations
sanketrai's activity
updated
a
model
13 days ago
published
a
model
13 days ago
updated
a
model
25 days ago
published
a
model
25 days ago
reacted to
macadeliccc's
post with 🔥
about 2 months ago
Post
1367
Save money on your compute bill by using LMCache to share prefix KV between 2 different vllm instances. By deploying LMCache backend along with your vLLM containers, you can share a prefix KV Cache between 2 different containers and models. It is very simple to implement into your existing stack.
Step 1: Pull docker images
Step 2: Start vLLM + LMCache
You can add another vLLM instance as long as its on a separate GPU by simply deploying another:
This method supports local, remote or hybrid backends so whichever vLLM deployment method you are already using should work with the LMCache container (excluding BentoML).
LMCache: https://github.com/LMCache/LMCache/tree/dev
vLLM: https://github.com/vllm-project/vllm
Step 1: Pull docker images
docker pull apostacyh/vllm:lmcache-0.1.0Step 2: Start vLLM + LMCache
model=mistralai/Mistral-7B-Instruct-v0.2 # Replace with your model name
sudo docker run --runtime nvidia --gpus '"device=0"' \
-v <Huggingface cache dir on your local machine>:/root/.cache/huggingface \
-p 8000:8000 \
--env "HF_TOKEN=<Your huggingface access token>" \
--ipc=host \
--network=host \
apostacyh/vllm:lmcache-0.1.0 \
--model $model --gpu-memory-utilization 0.6 --port 8000 \
--lmcache-config-file /lmcache/LMCache/examples/example-local.yamlYou can add another vLLM instance as long as its on a separate GPU by simply deploying another:
# The second vLLM instance listens at port 8001
model=mistralai/Mistral-7B-Instruct-v0.2 # Replace with your model name
sudo docker run --runtime nvidia --gpus '"device=1"' \
-v <Huggingface cache dir on your local machine>:/root/.cache/huggingface \
-p 8001:8001 \
--env "HF_TOKEN=<Your huggingface token>" \
--ipc=host \
--network=host \
apostacyh/vllm:lmcache-0.1.0 \
--model $model --gpu-memory-utilization 0.7 --port 8001 \
--lmcache-config-file /lmcache/LMCache/examples/example.yamlThis method supports local, remote or hybrid backends so whichever vLLM deployment method you are already using should work with the LMCache container (excluding BentoML).
LMCache: https://github.com/LMCache/LMCache/tree/dev
vLLM: https://github.com/vllm-project/vllm
updated
2
models
3 months ago
updated
a
dataset
3 months ago
liked
a
dataset
3 months ago
updated
a
model
3 months ago
updated
a
model
4 months ago
upvoted
a
collection
8 months ago
updated
a
model
8 months ago
upvoted
a
collection
8 months ago
liked
a
Space
10 months ago
upvoted
a
collection
11 months ago
upvoted
a
paper
11 months ago