Instructions to use gghfez/DeepSeek-V3-0324-IQ3_KS with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use gghfez/DeepSeek-V3-0324-IQ3_KS with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf gghfez/DeepSeek-V3-0324-IQ3_KS
# Run inference directly in the terminal:
llama cli -hf gghfez/DeepSeek-V3-0324-IQ3_KS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf gghfez/DeepSeek-V3-0324-IQ3_KS
# Run inference directly in the terminal:
llama cli -hf gghfez/DeepSeek-V3-0324-IQ3_KS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf gghfez/DeepSeek-V3-0324-IQ3_KS
# Run inference directly in the terminal:
./llama-cli -hf gghfez/DeepSeek-V3-0324-IQ3_KS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf gghfez/DeepSeek-V3-0324-IQ3_KS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf gghfez/DeepSeek-V3-0324-IQ3_KS

Use Docker

docker model run hf.co/gghfez/DeepSeek-V3-0324-IQ3_KS

LM Studio
Jan

vLLM

How to use gghfez/DeepSeek-V3-0324-IQ3_KS with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "gghfez/DeepSeek-V3-0324-IQ3_KS"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "gghfez/DeepSeek-V3-0324-IQ3_KS",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/gghfez/DeepSeek-V3-0324-IQ3_KS

Ollama
How to use gghfez/DeepSeek-V3-0324-IQ3_KS with Ollama:
```
ollama run hf.co/gghfez/DeepSeek-V3-0324-IQ3_KS
```

Unsloth Studio

How to use gghfez/DeepSeek-V3-0324-IQ3_KS with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for gghfez/DeepSeek-V3-0324-IQ3_KS to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for gghfez/DeepSeek-V3-0324-IQ3_KS to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for gghfez/DeepSeek-V3-0324-IQ3_KS to start chatting

Docker Model Runner
How to use gghfez/DeepSeek-V3-0324-IQ3_KS with Docker Model Runner:
```
docker model run hf.co/gghfez/DeepSeek-V3-0324-IQ3_KS
```

Lemonade

How to use gghfez/DeepSeek-V3-0324-IQ3_KS with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull gghfez/DeepSeek-V3-0324-IQ3_KS

Run and chat with the model

lemonade run user.DeepSeek-V3-0324-IQ3_KS-{{QUANT_TAG}}

List all available models

lemonade list

Atomic Chat

Do you find `ik_llama.cpp` gives you better performance?

by jukofyork - opened Jul 29, 2025

Discussion

jukofyork

Jul 29, 2025

•

edited Jul 29, 2025

Are you running everything on CPU or using a GPU at the same time?

For mainline llama.cpp I can get around 32 tokens/s for prompt processing using Q4_K shared experts and have to keep any batch less than ~2.8K tokens in RAM (I've hacked the ludicrously low 32 token offload threshold).

I had hoped the new ik_llama.cpp4bit quant(s) would speed things up for PP, but it actually was slightly worse than mainline using Q4_K (IIRC ~25 tokens/s for RAM/CPU prompt processing) and everything else was way worse (IIRC token generation was about 50% of mainline and offloaded batches were about 80% of mainline).

It was about a month ago I tried this and I did read the guides ubergarm made and tried a few variations on the 4bit "R" quants and all the different MLA/MOE command line args, but overall it was disappointing :/

Just wondering if I should try again or if the boost people are seeing is from sub-4bit quants or CPU-only inference?

jukofyork

Jul 29, 2025

•

edited Jul 29, 2025

I should add that Q4_K seems to work quite a bit better on mainline llama.cpp usingCascade Lake, Skylake and Broadwell Xeons, and mainline Q4_0 and IQ4_XS get a similar ~25 tokens/s to ik_llama.cpp - making me wonder if there has been some optimisation for Q4_K that is missing in ik_llama.cpp?

gghfez

Owner Jul 30, 2025

Sorry I missed these messages earlier.

Are you running everything on CPU or using a GPU at the same time?

GPU+GPU

Do you find ik_llama.cpp gives you better performance?

So mainline doesn't have -fmoe (fused moe) or -mla3. These features drastically reduce vram usage, allowing me to offload more layers to the GPUs.
If I'm running 100% on GPU, then no, performance is similar (I've read that mainline is actually faster). But prompt processing is a lot quicker for CPU+GPU.

Apparently it's also much faster for CPU-only than mainline, though I haven't tested this myself.

And these ks quants give me much better quality than the equivalent sized quants I could fit on mainline. Eg. with a 280GB IQ_XS or unsloth quant, the model doesn't think for as long, and the creative writing suffers.

If you give it a try, look up Ubergarm's quants, he tests them meticulously. Here's his Kimi-K2 quant: https://huggingface.co/ubergarm/Kimi-K2-Instruct-GGUF

See the IQ3_KS perplexity is actually lower than Unsloth's UD-Q4_K_XL? It's probably a lot better than Q4_0 though I haven't seen a benchmark.

Perplexity certainly isn't everything for creative writing, but anecdotally, I'm finding iq3_ks to be as good as Q4_K with deepseek models, but not the new Qwen3 MoE
I tend to use iq2_ks when I want >200t/s prompt processing, otherwise iq3_ks.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment