It does the right things, but with typo: "_" instead of "-"
#21
by alextzya - opened
This model is awesome and finds things and suggests on bugs and features the original coding model does not see. However, it has a problem.
The model has been asked to:
Rewrite this script to echo the entire llama-server command we execute:
#!/usr/bin/env bash
# Launch llama-server for BW-Laguna-S-2.1-Q8_0 across 3 GPUs:
# RTX 6000 (Blackwell, 96GB) + 2x RTX 3090 (24GB) = 144 GB total.
#
# MODE: -sm layer (pipeline parallelism). This is the ONLY multi-GPU mode that
# works with your required Q8_0 KV cache:
# -sm tensor (tensor-parallel) requires f16/bf16 KV -> errors with q8_0
# -sm row is deprecated and unsupported by the CUDA backend
# --fit (on by default) distributes layers by free memory and offloads any
# residual MoE to CPU; --fit-ctx 262144 keeps the full context window.
#
# Save on the GPU server, chmod +x, edit MODEL/BIN, then run.
set -euo pipefail
# ---- EDIT THESE -----------------------------------------------------------
MODEL=/intel1.6tb/bb/llama-server/models/BW-Laguna-S-2.1-Q8_0/Laguna-S-2.1-Q8_0-00001-of-00004.gguf
BIN=/home/bb/intel1.6tb/bb/llama-server/llama.cpp.120+86/build/bin
# 1 = RTX 6000 (Blackwell), 2 & 3 = RTX 3090
export CUDA_DEVICE_ORDER=PCI_BUS_ID
#export CUDA_VISIBLE_DEVICES=1,2,3
export CUDA_VISIBLE_DEVICES=3,2,1
# ----------------------------------------------------------------------------
export NCCL_DEBUG=INFO
# Q8_0 weights: leave MMK/MMQ kernels at their default. Do NOT force cuBLAS
# (GGML_CUDA_FORCE_CUBLAS=1) — the docs warn it can overflow on quantized
# models and uses more VRAM.
# Pipeline (layer) split moves activations across GPUs at layer boundaries.
# CUDA peer access lets boundary transfers bypass host RAM. It is opt-in and
# may be unstable on some boards (e.g. with IOMMU). Start without it; if GPU
# boundary transfers look slow, enable it by uncommenting:
# export GGML_CUDA_P2P=1
# export GGML_CUDA_PEER_MAX_BATCH_SIZE=128
cd "$BIN"
echo "==> available CUDA devices + free memory:"
#CUDA_VISIBLE_DEVICES=0,1,2
./llama-server --model "$MODEL" --list-devices 2>/dev/null | grep -iE "cuda|free" || true
echo "==> launching llama-server on 0.0.0.0:8080 (Ctrl-C to stop)"
echo " -sm layer (pipeline); --fit auto-distributes by free memory;"
echo " --fit-ctx 262144 keeps the full context; residual CPU offload"
echo " (if any) is printed because --fit-print is on."
numactl --cpunodebind=0 --membind=0 ./llama-server \
--model "$MODEL" --host 0.0.0.0 --port 8080 \
--split-mode layer \
--ctx-size 262144 --fit-ctx 262144 \
--fit on \
-fa on -ctk q8_0 -ctv q8_0 \
--threads 18 --threads-batch 18 \
--cpu-strict 1 \
--numa isolate --no-mmap \
--jinja
It produces the right script, but with:
MODEL=/intel1.6tb/bb/llama_server/models/BW-Laguna-S-2.1-Q8_0/Laguna-S-2.1-Q8_0-00001-of-00004.gguf
BIN=/home/bb/intel1.6tb/bb/llama_server/llama.cpp.120+86/build/bin
llama_server instead of llama-server.
I do not have HW to test model in full, but Q8 and lower have the same behavior.
Pointed to an exact issue in Pi:
the problem is in with llama_server.
The user is pointing out an issue with the path llama_server. Let me look at the original prompt. In the original, the MODEL path was:
/intel1.6tb/bb/llama_server/models/...
Wait — the original had llama_server but the BIN path had llama_server/llama.cpp.120+86/build/bin. Hmm, but that's inconsistent. Let me look at the original prompt more
carefully.
Actually, in the original prompt, the paths were:
'''
MODEL=/intel1.6tb/bb/llama_server/models/BW-Laguna-S-2.1-Q8_0/Laguna-S-2.1-Q8_0-00001-of-00004.gguf
BIN=/home/bb/intel1.6tb/bb/llama_server/llama.cpp.120+86/build/bin
'''
And it spin in the endless loop of thinking.
alextzya changed discussion status to closed