Instructions to use jabbatheduck/DeepSeek-v4-flash-mini with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use jabbatheduck/DeepSeek-v4-flash-mini with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jabbatheduck/DeepSeek-v4-flash-mini
# Run inference directly in the terminal:
llama cli -hf jabbatheduck/DeepSeek-v4-flash-mini

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf jabbatheduck/DeepSeek-v4-flash-mini
# Run inference directly in the terminal:
llama cli -hf jabbatheduck/DeepSeek-v4-flash-mini

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf jabbatheduck/DeepSeek-v4-flash-mini
# Run inference directly in the terminal:
./llama-cli -hf jabbatheduck/DeepSeek-v4-flash-mini

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf jabbatheduck/DeepSeek-v4-flash-mini
# Run inference directly in the terminal:
./build/bin/llama-cli -hf jabbatheduck/DeepSeek-v4-flash-mini

Use Docker

docker model run hf.co/jabbatheduck/DeepSeek-v4-flash-mini

LM Studio
Jan

vLLM

How to use jabbatheduck/DeepSeek-v4-flash-mini with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jabbatheduck/DeepSeek-v4-flash-mini"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jabbatheduck/DeepSeek-v4-flash-mini",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jabbatheduck/DeepSeek-v4-flash-mini

Ollama
How to use jabbatheduck/DeepSeek-v4-flash-mini with Ollama:
```
ollama run hf.co/jabbatheduck/DeepSeek-v4-flash-mini
```

Unsloth Studio

How to use jabbatheduck/DeepSeek-v4-flash-mini with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for jabbatheduck/DeepSeek-v4-flash-mini to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for jabbatheduck/DeepSeek-v4-flash-mini to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for jabbatheduck/DeepSeek-v4-flash-mini to start chatting

How to use jabbatheduck/DeepSeek-v4-flash-mini with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jabbatheduck/DeepSeek-v4-flash-mini

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "jabbatheduck/DeepSeek-v4-flash-mini"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use jabbatheduck/DeepSeek-v4-flash-mini with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jabbatheduck/DeepSeek-v4-flash-mini

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "jabbatheduck/DeepSeek-v4-flash-mini" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use jabbatheduck/DeepSeek-v4-flash-mini with Docker Model Runner:
```
docker model run hf.co/jabbatheduck/DeepSeek-v4-flash-mini
```

Lemonade

How to use jabbatheduck/DeepSeek-v4-flash-mini with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull jabbatheduck/DeepSeek-v4-flash-mini

Run and chat with the model

lemonade run user.DeepSeek-v4-flash-mini-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use jabbatheduck/DeepSeek-v4-flash-mini with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf jabbatheduck/DeepSeek-v4-flash-mini

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default jabbatheduck/DeepSeek-v4-flash-mini

Run Hermes

hermes

Atomic Chat

Crashes on dgx spark

by redval - opened 2 days ago

Discussion

redval

2 days ago

Crashes at the first prompt on a dgx spark:

./llama-server -hf jabbatheduck/DeepSeek-v4-flash-mini --temp 0.6 --top-p 1.0 --min-p 0.0 --host 0.0.0.0 --port 8888 -c 262144
0.02.314.688 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.02.315.010 W srv  llama_server: -----------------
0.02.315.011 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.02.315.011 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.02.315.011 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655
0.02.315.011 W srv  llama_server: -----------------
0.02.317.528 I srv    load_model: loading model 'jabbatheduck/DeepSeek-v4-flash-mini'
0.03.837.125 W llama_model_loader: tensor overrides to CPU are used with mmap enabled - consider using --no-mmap for better performance
0.28.702.145 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 262144, kv_unified = 'true'
0.28.719.589 I srv          init: chat template supports preserving reasoning, consider enabling it via --reasoning-preserve
0.28.719.630 I srv  llama_server: model loaded
0.28.719.633 I srv  llama_server: listening on http://0.0.0.0:8888
0.57.169.858 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
0.57.169.886 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
/home/athea/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:106: CUDA error
0.57.615.796 E CUDA error: an illegal memory access was encountered
0.57.615.801 E   current device: 0, in function launch_mul_mat_q at /home/athea/llama.cpp/ggml/src/ggml-cuda/template-instances/../mmq.cuh:1401
0.57.615.801 E   cudaFuncSetAttribute((mul_mat_q<type, J, false>), cudaFuncAttributeMaxDynamicSharedMemorySize, nbytes_shared)
[New LWP 198690]
[New LWP 198689]
[New LWP 198688]
[New LWP 198687]
[New LWP 198686]
[New LWP 198685]
[New LWP 198684]
[New LWP 198683]
[New LWP 198682]
[New LWP 198681]
[New LWP 198680]
[New LWP 198679]
[New LWP 198678]
[New LWP 198677]
[New LWP 198676]
[New LWP 198675]
[New LWP 198674]
[New LWP 198673]
[New LWP 198672]
[New LWP 198670]
[New LWP 198669]
[New LWP 198668]
[New LWP 198667]
[New LWP 198666]
[New LWP 198665]
[New LWP 198664]
[New LWP 198663]
[New LWP 198662]
[New LWP 198661]
[New LWP 198660]
[New LWP 198659]
[New LWP 198658]
[New LWP 198657]
[New LWP 198656]
[New LWP 198655]
[New LWP 198654]
[New LWP 198653]
[New LWP 198652]
[New LWP 198651]
[New LWP 198649]
[New LWP 198648]
[New LWP 198634]
[New LWP 198633]
[New LWP 198632]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ef35febe7a34 in __GI___wait4 (pid=198692, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30	../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0  0x0000ef35febe7a34 in __GI___wait4 (pid=198692, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30	in ../sysdeps/unix/sysv/linux/wait4.c
#1  0x0000b6067647a22c in ggml_print_backtrace ()
#2  0x0000b6067647a3d0 in ggml_abort ()
#3  0x0000b6067592f1cc in ggml_cuda_error(char const*, char const*, char const*, int, char const*) ()
#4  0x0000b60675b199dc in void launch_mul_mat_q<(ggml_type)16, 128, false>(ggml_backend_cuda_context&, mmq_args const&, CUstream_st*) ()
#5  0x0000b60675974878 in ggml_cuda_mul_mat_q(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor const*, ggml_tensor*) ()
#6  0x0000b60675942b84 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) ()
#7  0x0000b6067649c4dc in ggml_backend_sched_compute_splits(ggml_backend_sched*) ()
#8  0x0000b606756c3b1c in llama_context::graph_compute(ggml_cgraph*, bool) ()
#9  0x0000b606756caa60 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) ()
#10 0x0000b606756d0e10 in llama_context::decode(llama_batch const&) ()
#11 0x0000b606756d2920 in llama_decode ()
#12 0x0000b60675320ac4 in server_context_impl::decode(int&, int, llama_batch&) ()
#13 0x0000b60675321eb0 in server_context_impl::update_slots() ()
#14 0x0000b606752caea4 in server_queue::start_loop(long) ()
#15 0x0000b6067526fcd4 in llama_server(common_params&, int, char**) ()
#16 0x0000b60675271644 in llama_server(int, char**) ()
#17 0x0000ef35feb384c4 in __libc_start_call_main (main=main@entry=0xb606751da2c0 <main>, argc=argc@entry=15, argv=argv@entry=0xffffdb5b5f58) at ../sysdeps/nptl/libc_start_call_main.h:58
warning: 58	../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#18 0x0000ef35feb38598 in __libc_start_main_impl (main=0xb606751da2c0 <main>, argc=15, argv=0xffffdb5b5f58, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:360
warning: 360	../csu/libc-start.c: No such file or directory
#19 0x0000b606752677b0 in _start ()
[Inferior 1 (process 198630) detached]
[1]    198630 abort (core dumped)  ./llama-server -hf jabbatheduck/DeepSeek-v4-flash-mini --temp 0.6 --top-p 1.0

jabbatheduck

Owner 2 days ago

Crashes at the first prompt on a dgx spark:

./llama-server -hf jabbatheduck/DeepSeek-v4-flash-mini --temp 0.6 --top-p 1.0 --min-p 0.0 --host 0.0.0.0 --port 8888 -c 262144

[New LWP 198633]
[New LWP 198632]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/aarch64-linux-gnu/libthread_db.so.1".
0x0000ef35febe7a34 in __GI___wait4 (pid=198692, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30 ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#0 0x0000ef35febe7a34 in __GI___wait4 (pid=198692, stat_loc=0x0, options=0, usage=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30
30 in ../sysdeps/unix/sysv/linux/wait4.c
#1 0x0000b6067647a22c in ggml_print_backtrace ()
#2 0x0000b6067647a3d0 in ggml_abort ()
#3 0x0000b6067592f1cc in ggml_cuda_error(char const*, char const*, char const*, int, char const*) ()
#4 0x0000b60675b199dc in void launch_mul_mat_q<(ggml_type)16, 128, false>(ggml_backend_cuda_context&, mmq_args const&, CUstream_st*) ()
#5 0x0000b60675974878 in ggml_cuda_mul_mat_q(ggml_backend_cuda_context&, ggml_tensor const*, ggml_tensor const*, ggml_tensor const*, ggml_tensor*) ()
#6 0x0000b60675942b84 in ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) ()
#7 0x0000b6067649c4dc in ggml_backend_sched_compute_splits(ggml_backend_sched*) ()
#8 0x0000b606756c3b1c in llama_context::graph_compute(ggml_cgraph*, bool) ()
#9 0x0000b606756caa60 in llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) ()
#10 0x0000b606756d0e10 in llama_context::decode(llama_batch const&) ()
#11 0x0000b606756d2920 in llama_decode ()
#12 0x0000b60675320ac4 in server_context_impl::decode(int&, int, llama_batch&) ()
#13 0x0000b60675321eb0 in server_context_impl::update_slots() ()
#14 0x0000b606752caea4 in server_queue::start_loop(long) ()
#15 0x0000b6067526fcd4 in llama_server(common_params&, int, char**) ()
#16 0x0000b60675271644 in llama_server(int, char**) ()
#17 0x0000ef35feb384c4 in __libc_start_call_main (main=main@entry=0xb606751da2c0
, argc=argc@entry=15, argv=argv@entry=0xffffdb5b5f58) at ../sysdeps/nptl/libc_start_call_main.h:58
warning: 58 ../sysdeps/nptl/libc_start_call_main.h: No such file or directory
#18 0x0000ef35feb38598 in __libc_start_main_impl (main=0xb606751da2c0
, argc=15, argv=0xffffdb5b5f58, init=, fini=, rtld_fini=, stack_end=) at ../csu/libc-start.c:360
warning: 360 ../csu/libc-start.c: No such file or directory
#19 0x0000b606752677b0 in _start ()
[Inferior 1 (process 198630) detached]
[1] 198630 abort (core dumped) ./llama-server -hf jabbatheduck/DeepSeek-v4-flash-mini --temp 0.6 --top-p 1.0

/mnt/storage/Projects/llama-cpp-turboquant/build/bin/llama-server \
  -m /mnt/storage/models/deepseek-v4-flash/v0731/full/DeepSeek-V4-Flash-REAP-IQ2XXS-w2Q2K-AProjQ8-OutQ8-chat-v2.gguf \
  -c 512000 -ngl auto --fit on -fa on --numa distribute --jinja \
  -t 8 -Cr 0-7 -Crb 0-7 -b 2048 -ub 1 \
  --temp 1.0 --top-p 0.95 --top-k 20 --parallel 1 \
  --host 0.0.0.0 --port 8095 \
  --repeat-penalty 1.0 --presence-penalty 0.0 \
  --reasoning off --ctx-checkpoints 4 --checkpoint-min-step 16384

try it liket his, the ubatach is giving issues, not 100% sure why. obviously adjust this to your system.

redval

2 days ago

Thanks, that worked. But speed is incredibly slow compared to unsloth IQ3 quant:

3.08.991.045 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =   2048, progress = 0.14, t =  89.52 s / 22.88 tokens per second
4.39.200.848 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =   4096, progress = 0.28, t = 179.73 s / 22.79 tokens per second
6.09.941.158 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =   6144, progress = 0.42, t = 270.47 s / 22.72 tokens per second
7.41.389.580 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =   8192, progress = 0.56, t = 361.92 s / 22.64 tokens per second
9.13.419.144 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =  10240, progress = 0.71, t = 453.95 s / 22.56 tokens per second
10.45.918.754 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =  12288, progress = 0.85, t = 546.44 s / 22.49 tokens per second
12.19.060.577 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =  14336, progress = 0.99, t = 639.59 s / 22.41 tokens per second
12.19.060.985 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =   1300, progress = 1.00, t = 696.42 s / 1.87 tokens per second
12.26.550.110 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =  14496, progress = 1.00, t = 647.08 s / 22.40 tokens per second
12.26.550.530 I slot print_timing: id  3 | task 0 | prompt processing, n_tokens =   1301, progress = 1.00, t = 703.91 s / 1.85 tokens per second
12.26.818.238 I slot print_timing: id  2 | task 2 | prompt processing, n_tokens =  14497, progress = 1.00, t = 647.34 s / 22.39 tokens per second
12.32.338.204 I slot print_timing: id  2 | task 2 | prompt eval time =  647633.34 ms / 14501 tokens (   44.66 ms per token,    22.39 tokens per second)
12.32.338.206 I slot print_timing: id  2 | task 2 |        eval time =    5230.86 ms /    88 tokens (   59.44 ms per token,    16.82 tokens per second)
12.32.338.207 I slot print_timing: id  2 | task 2 |       total time =  652864.21 ms / 14589 tokens
12.32.338.208 I slot print_timing: id  2 | task 2 |    graphs reused =      15620
12.32.338.791 I slot      release: id  2 | task 2 | stop processing: n_tokens = 14588, truncated = 0
12.32.873.409 I slot print_timing: id  3 | task 0 | n_decoded =    100, tg =  16.51 t/s, tg_3s =  16.51 t/s
12.33.600.703 I slot print_timing: id  3 | task 0 | prompt eval time =  704174.56 ms /  1305 tokens (  539.60 ms per token,     1.85 tokens per second)
12.33.600.706 I slot print_timing: id  3 | task 0 |        eval time =    6782.49 ms /   115 tokens (   58.98 ms per token,    16.96 tokens per second)
12.33.600.707 I slot print_timing: id  3 | task 0 |       total time =  710957.05 ms /  1420 tokens

With the unsloth quant I get around 350 tokens per second of pp and 30 t/s for decode

tobyq

2 days ago

Crashes for me too. Best I can get is ub 7 - anything 8 or over crashes. Some of the AI models I asked about the crash seems to indicate some out of bounds thing, something when going over 7 it switches to mmq or something along those lines. Above my knowledge... let's hope someone can make it work without such a small ub. I get sub 200 PP on 3x 3090's.

jabbatheduck

Owner 2 days ago

Yeah, I dont know if its a bug of llama. or just a crap model because of its quant.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment