poolside

/

Laguna-S-2.1-DFlash

like 9

Follow

Poolside 765

(Filed here because issues are disabled on the poolsideai/llama.cpp fork — this concerns the laguna branch of that fork.)

Restricting the DFlash draft model to a single device with
-devd / --spec-draft-device crashes at draft-context creation when the
target model is split across multiple GPUs:

ggml/src/ggml-backend.cpp:898: pre-allocated tensor (output.weight) in a buffer (ROCm1)
that cannot run the operation (NONE)
  ggml_backend_sched_split_graph
  llama_context::graph_reserve
  llama_context::sched_reserve
  llama_init_from_model
  common_speculative_init_result

Root cause

The Laguna DFlash GGUF ships without token_embd.weight and
output.weight; the dflash graph borrows both from the target model via
ctx_other (src/models/dflash.cpp — "tok_embd from the target model (shared
via ctx_other)" / "lm_head from the target model (shared via ctx_other)").

With --split-mode layer over N GPUs the target's output.weight is resident
on the last device. A draft context restricted to device 0 builds its
scheduler with only {device0, CPU} backends, so the borrowed tensor on the last
device cannot be assigned and ggml_backend_sched_split_graph aborts.

Repro

llama-server -m Laguna-S-2.1-UD-Q3_K_M-*.gguf \
  -md laguna-s-2.1-DFlash-BF16.gguf \
  --spec-type draft-dflash -devd ROCm0 \
  --device ROCm0,ROCm1 --split-mode layer -ngl 999 --jinja -fa on

Commit 04b2b72, 2× AMD MI50 (gfx906), ROCm 7.2.4. Same abort logic should
reproduce on multi-GPU CUDA.

Workaround

Keep the full device list for the draft and pin its own weights instead:

-otd ".*=ROCm0"

This loads and runs correctly (the scheduler keeps a backend for the borrowed
target tensors).

Suggested fix

When constructing the draft context, extend its backend set with the devices
that host any ctx_other-borrowed tensors (or at minimum fail with a
diagnostic naming the flag conflict instead of a scheduler abort).

Additional minor issues found on the same branch

1. Build failure on gcc 13: common/speculative.cpp:1091 uses
   std::isfinite without including <cmath> (introduced in the Metal
   non-finite-features fix, PR #2). One-line fix.
2. Per-request speculative.n_max is silently ignored for
   draft-dflash — only the CLI --spec-draft-n-max takes effect. If
   per-request override is unsupported by design, rejecting the field would be
   less surprising than accepting and ignoring it.