unsloth

/

Laguna-S-2.1-GGUF

like 305

Follow

Unsloth AI 32.4k

You are just the best! 🤗

With UD-Q4_K_XL, I'm getting ~30 t/s.
Tools are working as expected.
No loop from limited testing.

System:

• Nvidia RTX 5090 with 32GB VRAM
• Intel Xeon W5-3425
• 256GB of DDR5-4800 RAM (8 channels)
• Ubuntu 24

logs:

prompt eval time =   11s /  1571 tokens (    7.48 ms per token,   133.69 tokens per second)
       eval time =   48s /  1476 tokens (   32.85 ms per token,    30.44 tokens per second)
      total time =   60s /  3047 tokens

my docker-compose.yaml:

services:
  llama-router:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda13-b10143
    container_name: router
    devices:
      - "nvidia.com/gpu=all"
    ports:
      - "8080:8080"
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    volumes:
      - /var/lib/docker/.cache:/root/.cache
      - ./models.ini:/app/models.ini:ro
    entrypoint: ["./llama-server"]
    command: >
      --models-max 1
      --models-preset ./models.ini
      --host 0.0.0.0  --port 8080
      --tools all

my models.ini:

version = 1

[unsloth/Laguna-S-2.1-GGUF:Q4_K_XL]
ctx-size=32768

I see that poolside/Laguna-S-2.1-GGUF:Q4_K_M has DFlash included, but unsloth doesn't. Are you planning to included it in the feature?

@SlavikF you can use the upstream DFlash model right now https://huggingface.co/poolside/Laguna-S-2.1-GGUF/blob/main/laguna-s-2.1-DFlash-BF16.gguf. In my limited testing, it can give me an additional 30tps of decode in the select tasks. without it I get ~35tps on 4x AMD r9700 AI Pro in layer split mode with the rocm backend (Q4_k_XL).

A lower bit version of DFlash would be cool if it speeds things up even more.

@Schimazing ,
Can you please share the command or compose file you're using to start it with DFlash? which version or branch are you using?

When I try, I'm getting this:
> failed to load draft model, '/root/.cache/huggingface/laguna-s-2.1-DFlash-BF16.gguf'

ok, found that because fit doesn't account for DFlash VRAM usage.

It works with fit-target:

services:
  laguna21:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda13-b10143
    container_name: laguna21
    devices:
      - "nvidia.com/gpu=all"
    ports:
      - "8080:8080"
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    volumes:
      - /var/lib/docker/.cache:/root/.cache:ro
    entrypoint: ["./llama-server"]
    command: >
      --host 0.0.0.0  --port 8080
      --tools all
      --ctx-size 65536
      --model /root/.cache/huggingface/hub/models--unsloth--Laguna-S-2.1-GGUF/snapshots/750f92f90cf54159c4d7a610cb7b3e74498e75c6/UD-Q4_K_XL/Laguna-S-2.1-UD-Q4_K_XL-00001-of-00003.gguf
      --spec-type draft-dflash
      --spec-draft-n-max 15
      --spec-draft-model /root/.cache/huggingface/laguna-s-2.1-DFlash-BF16.gguf
      --fit-target 8192

But it slower with DFlash:

prompt eval time =   12s /  1571 tokens (    7.87 ms per token,   127.01 tokens per second)
       eval time =   67s /  1997 tokens (   33.61 ms per token,    29.75 tokens per second)
      total time =   79s /  3568 tokens

I guess it's slower with DFlash in my case because it uses RAM. Probably will faster with DFlash in case I would have enough VRAM

@SlavikF My issue was building poolsideai's llama.cpp fork on the main branch. You must checkout the branch laguna for DFlash to work.
Here is my command:

    ./build/bin/llama-cli \
    -hf unsloth/Laguna-S-2.1-GGUF \
    -hff UD-Q4_K_XL/Laguna-S-2.1-UD-Q4_K_XL-00001-of-00003.gguf \
    --split-mode layer \
    --tensor-split 1,1,1,1 \
    -ngl -1 \
    -c 262144 \
    --cache-type-k q8_0 \
    --cache-type-v q8_0 \
    -cnv \
    --jinja \
    --flash-attn on \
    --no-mmap \
    --fit off \
    --direct-io \
    --chat-template-kwargs '{"enable_thinking":true, "preserve_thinking":true}' \
    --perf \
    -md ~/.cache/huggingface/hub/models--poolside--Laguna-S-2.1-GGUF/snapshots/edd093522473dc7313b0738d8b4116b7f8b9745f/laguna-s-2.1-DFlash-BF16.gguf \
    --spec-type draft-dflash \
    --spec-draft-n-max 6

does the lastest release of llama.cpp (b10176) support dflash now? I still get errors when trying to load it with dflash

With UD-Q4_K_XL, I'm getting ~30 t/s.
Tools are working as expected.
No loop from limited testing.

System:

• Nvidia RTX 5090 with 32GB VRAM
• Intel Xeon W5-3425
• 256GB of DDR5-4800 RAM (8 channels)
• Ubuntu 24

logs:

prompt eval time =   11s /  1571 tokens (    7.48 ms per token,   133.69 tokens per second)
       eval time =   48s /  1476 tokens (   32.85 ms per token,    30.44 tokens per second)
      total time =   60s /  3047 tokens

my docker-compose.yaml:

services:
  llama-router:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda13-b10143
    container_name: router
    devices:
      - "nvidia.com/gpu=all"
    ports:
      - "8080:8080"
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
    volumes:
      - /var/lib/docker/.cache:/root/.cache
      - ./models.ini:/app/models.ini:ro
    entrypoint: ["./llama-server"]
    command: >
      --models-max 1
      --models-preset ./models.ini
      --host 0.0.0.0  --port 8080
      --tools all

my models.ini:

version = 1

[unsloth/Laguna-S-2.1-GGUF:Q4_K_XL]
ctx-size=32768

I see that poolside/Laguna-S-2.1-GGUF:Q4_K_M has DFlash included, but unsloth doesn't. Are you planning to included it in the feature?

with that caed, you can probably get 1000+ tokens/sec prefill if you use a batch and ubatch size of 2048 or 4096. The model doesn't fit in VRAM, so llama.cpp defaults to much smaller sizes. my 2x 5060ti get me 380 Tok/sec pp and 11 gen speed. (DDR4)

if this tip improves your speed, maybe you should buy me DDR5 RAM. lol

So, I'm getting
llama_context: n_ctx_seq (1048576) > n_ctx_train (262144) -- possible training context overflow
srv load_model: the slot context (1048576) exceeds the training context of the model (262144) - capping
with both llama.cpp b10181 and the laguna fork, regardless of yarn and rope settings. Any ideas?

Arguments (laguna fork): --ctx-size 1048576 --rope-scaling yarn --rope-scale 128 --yarn-orig-ctx 8192 --yarn-attn-factor 1.0 --no-mmap -ndio -ctk q8_0 -ctv q8_0 --jinja --fit off

Never mind, seems that it's just this outstanding bug: https://github.com/ggml-org/llama.cpp/issues/17459