Instructions to use deepseek-ai/DeepSeek-R1-Distill-Llama-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepseek-ai/DeepSeek-R1-Distill-Llama-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepseek-ai/DeepSeek-R1-Distill-Llama-8B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Llama-8B")
model = AutoModelForCausalLM.from_pretrained("deepseek-ai/DeepSeek-R1-Distill-Llama-8B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepseek-ai/DeepSeek-R1-Distill-Llama-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepseek-ai/DeepSeek-R1-Distill-Llama-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B

SGLang

How to use deepseek-ai/DeepSeek-R1-Distill-Llama-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepseek-ai/DeepSeek-R1-Distill-Llama-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepseek-ai/DeepSeek-R1-Distill-Llama-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepseek-ai/DeepSeek-R1-Distill-Llama-8B with Docker Model Runner:
```
docker model run hf.co/deepseek-ai/DeepSeek-R1-Distill-Llama-8B
```

Running DeepSeek-R1-8B with 32k context on a 6GB Laptop GPU via Closed-Form Weight Surgery & In-Situ KV Compaction

#38

by aecetin - opened 3 days ago

Discussion

aecetin

3 days ago

Hi DeepSeek community!

We conducted an empirical benchmark exploring how to run DeepSeek-R1-Distill-Llama-8B with long reasoning chains (32k context) on entry-level consumer hardware (single 6 GB laptop GPU: NVIDIA RTX PRO 500 Blackwell Generation, 32 GB RAM) without Out-Of-Memory (OOM) crashes.

The Two Core Bottlenecks:

FFN Weight Footprint: SwiGLU FFN represents 70.2% of total parameters (5.64B out of 8.03B).
Reasoning KV-Cache Explosion: Multi-thousand-token <think> ... </think> chains consume up to 4.0 GB VRAM for KV-cache alone at 32k context, pushing total VRAM past 9.2 GB.

What We Did:

idempotent-poly: Applied layer-by-layer closed-form algebraic Chebyshev polynomial tensor surgery (K=3) replacing SwiGLU with zero gradient descent:

$C^* = (\Phi(X)^T \Phi(X) + \lambda I)^{-1} \Phi(X)^T Y$

* `idempotent-kv`: Applied zero-copy in-situ idempotent permutation compactor achieving 50% KV memory compaction with 0-byte peak auxiliary allocation.

Empirical Benchmark Results:

Metric	Original SwiGLU (DeepSeek-R1)	Chebyshev PolyFFN ($K=3$)	Delta / Savings
FFN Parameters Per Layer	176,160,768 (176.16M)	67,108,864 (67.11M)	-61.90% Parameters
Total Model FFN Parameters	5,637,144,576 (5.64B)	2,147,483,648 (2.15B)	-3.49 Billion Parameters Removed!
Total Model Parameters	8,030,000,000 (8.03B)	4,540,339,072 (4.54B)	-43.5% Model Total
Full Surgery Time (32 Layers)	Days of GPU fine-tuning	50.74 Seconds (0.85 min)	Instant Zero-Backprop
Layer Forward Latency	100.73 ms	40.43 ms	2.49x Faster Inference! ⚡

KV-Cache at 32,768 Context:

Standard KV Cache: 4.00 GB
Idempotent-KV Cache: 2.00 GB (-50.0%)
In-place Verification: Physical tensor pointer before = 0x3aa15400100, after = 0x3aa15400100 (Zero auxiliary buffers allocated).

Net VRAM Footprint on 6 GB Laptop GPU:

Original 8B @ 32k context: 4.92 GB (Weights) + 4.00 GB (KV) + 0.30 GB (Act) = 9.22 GB ❌ (OOM Crash)
PolySurgery + Idempotent-KV @ 32k context: 2.78 GB (Weights) + 2.00 GB (KV) + 0.20 GB (Act) = 4.98 GB ✅ (Runs Stably!)

Model Cards & Reproducible Code:

🔗 Hugging Face Model: aecetin/DeepSeek-R1-8B-PolySurgery
🔗 Unified 5-Pillar 128k Record: aecetin/DeepSeek-R1-8B-Unified-IdemFormer
🔗 GitHub Repository & Benchmark Script: github.com/aemre-cetin/idempotent-poly
📜 Full Paper: ResearchGate Publication 414060833

Would love to hear feedback and benchmark thoughts from the community!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment