turboderp

/

DeepSeek-V4-Flash-0731-exl3

like 12

It often comes down to sampler settings, and what your frontend/harness is setting by default. It's really all incredibly messy and hard to figure out exactly what settings TabbyAPI should respond to, where they appear in the request, and so on. Can you say more specifically what you're doing, like what harness are you using, what settings, what are your settings in TabbyAPI (or are you using it outside of TabbyAPI?), etc.

It often comes down to sampler settings, and what your frontend/harness is setting by default. It's really all incredibly messy and hard to figure out exactly what settings TabbyAPI should respond to, where they appear in the request, and so on. Can you say more specifically what you're doing, like what harness are you using, what settings, what are your settings in TabbyAPI (or are you using it outside of TabbyAPI?), etc.

very likely it was due to low quantization, i switched to 2.52 bpw (it loaded fine on the 6000 PRO without dspark), and with exact same settings i got 0 loops so far, but i'm at first hour of benchmarks, i'll use it 1 full day and see

if it's stable i'll try some surgery, maybe switch some layers that don't affect perplexity that much to lower quantization on those and try to make space for dspark, but hey, without dspark i get ~86 tok/s with max reasoning which is already not bad at all, very usable

Do be mindful of sampler settings, but yes, 2 bpw is very aggressive quantization. If you're struggling to fit the DSpark model, in the quants I uploaded all the DSpark modules are quantized to 3 bpw. It may work just as well with the routed experts in 2 bpw (util/convert_mtp.py takes only a single bitrate, which applies to attn and shared experts as well, so while it can convert the MTP layers, you'll want to transplant only the routed experts from the output.)

i will definitely try, btw this is the small/quick shim i'm using for my (pre-existing) benchmarks
(sorry couldn't upload the zip here)

openAI shim and preliminary metrics (still didn't finish running, can't compare quality to vllm-moet yet)
https://limewire.com/d/iV5RL#mM03cjdUkn

With TabbyAPI

Aggregate throughput vs concurrency (4x 256k contexts)

not bad

I added reasoning_budget support to TabbyAPI (currently needs the exllamav3 dev branch to activate) which should help in cases where reasoning ends up completely derailed or stuck in a loop that can't be detected by loop_detect_window. It seems like many (most?) frontends will include this with requests, otherwise you can configure it as a default per model.

I added reasoning_budget support to TabbyAPI (currently needs the exllamav3 dev branch to activate) which should help in cases where reasoning ends up completely derailed or stuck in a loop that can't be detected by loop_detect_window. It seems like many (most?) frontends will include this with requests, otherwise you can configure it as a default per model.

nice

btw i tried as you said requantizing dspark with util/convert_mtp.py (hybrid, partial 2bpw and partial 3bpw) and now it fits, acceptance is still pretty and now i'm at 110+ tok/s (single concurrent, 1x 342K context -> needle retrieval at 0.01, 0.1, 1, 10, 50, 90, 99, 99.9, 99.99, all passed)

i'm currently writing my own calib .utf8 (more coding focused), probably it affects only a tiny bit but...
then i will requantize without doing frankenstains to squeeze the last tiny bit on the 6000 PRO, i will try to see what's the maximum context i can do with needle retrieval and other more stressful tests on the context, then i will readjust the bpw (maybe i will upgrade some experts to 3bpw layers or downgrade the least meaningful ones to 2bpw to fit more context), but more or less 2.5bpw is already on the limit

best model so far i can run on my GPU

I test 3.05bpw version and got around 120 tk/s for tg and ~2200tk/s for pp with my RTX PRO 6000+5090. absolutely no issue with looping behavior in both Hermes and pi.dev up to 200k context. Amazing.

Still running without any looping behavior, even tried to guide it to loop and did not manage to do so at 381k context

pi.dev setting:

{
  "providers": {
    "tabbyapi": {
      "baseUrl": "secret haha",
      "api": "openai-completions",
      "apiKey": "hmmm",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false
      },
      "models": [
        {
          "id": "DeepSeek-V4-Flash-0731-exl3-3.04bpw",
          "name": "DeepSeek V4 Flash (Local 128k)",
          "contextWindow": 381072,
          "maxTokens": 65536,
          "samplingParams": {
            "temperature": 1.0,
            "top_p": 1.0,
            "min_p": 0.05,
            "max_tokens": 65536
          }
        }
      ]
    }
}

tabbyai setting:

network:
  host: 0.0.0.0
  port: 1234
  disable_auth: true
  disable_fetch_requests: false
  send_tracebacks: false
  api_servers: ["OAI"]
  sse_ping_interval: 15

# Options for logging
logging:
  log_prompt: false
  log_generation_params: false
  log_requests: false
  log_chat_completion_requests: false

# Options for model overrides and loading
model:
  tool_format: deepseek_v4
  model_dir: /path/to/models
  model_name: DeepSeek-V4-Flash-0731-exl3-3.04bpw

  inline_model_loading: false
  use_dummy_models: false
  dummy_model_names: ["gpt-3.5-turbo"]
  use_as_default: []

  backend: exllamav3
  max_seq_len: 524288    # 128k context (rock-solid stability on single GPU)
  cache_size: 524544     # 128k + 256 buffer
  cache_mode: FP16         # 8-bit KV cache quantization

  tensor_parallel: false
  tensor_parallel_backend: native

  # gpu_split_auto: true    # <--- Changed to true
  # autosplit_reserve: [96, 96]  # Leaves 96MB buffer per GPU for the OS display
  # gpu_split: []           # <--- Clear the manual array
  gpu_split_auto: false
  autosplit_reserve: [96]
  gpu_split: [90, 28]     # 94 GB on GPU 0, 0 GB on GPU 1 

  cpu_moe_offload_layers: 0
  rope_scale: 1.0
  rope_alpha:
  chunk_size: 2048
  output_chunking: true
  max_batch_size: 4

  prompt_template:
  vision: false
  template_vars_default:
    reasoning_effort: max
    reasoning_budget: 0      # 0 = no limit, or set a high number like 32768
  template_vars_force: {}

  # Reasoning parameters
  reasoning: true
  reasoning_start_token: "<think>"
  reasoning_end_token: "</think>"
  start_in_reasoning: auto
  tool_calls_in_reasoning: true

# Options for draft models (Multi-Token Prediction)
draft_model:
  draft_mode: mtp        # Enables DeepSeek MTP acceleration
  draft_model_dir: /path/to/models
  draft_model_name:
  draft_rope_scale: 1.0
  draft_cache_mode: FP16
  draft_gpu_split: []
  ngram_match_min: 2

# Options for Sampling
sampling:
  override_preset: safe_defaults

lora:
  lora_dir: loras
  loras:

embeddings:
  embedding_model_dir: models
  embeddings_device: cpu
  embedding_model_name:

memory:
  sysmem_recurrent_cache: 4096
  sysmem_kv_cache: 0
  cuda_malloc_async: True

developer:
  unsafe_launch: false
  disable_request_streaming: false
  realtime_process_priority: false
  seqlog: false