ggml-org

/

Clef-Flash-GGUF

Like 39

Follow

ggml-org 3.16k

So I ran the BF16 file on a DGX Spark in my home lab to label chat messages. It decides three things about every message: does it ask to misuse confidential or personal data, does it need the company handbook, and does it need Malaysian law. Here's what I measured. Just results from my setup.

Hardware and software

The loader reports 9.08 B parameters (size label 9.1B).

Why b11576: it is b11572 plus the flash-attention fix for Clef on NVIDIA (llama.cpp #30416).

Why BF16: the smaller files shrink the decision head. llama.cpp #30064 reports that flattening the probabilities for the bigger Clef, and a 4-bit run on a DGX Spark was reported to give a different answer about 1 time in 25, on the borderline cases. I had the memory, so I went full size.

Settings

llama-server --model Clef-Flash-BF16.gguf --alias clef-flash-9b --host 0.0.0.0 --port 8085 \
  --n-gpu-layers all --fit off --parallel 1 -c 16384 -b 4096 -ub 4096 --cache-ram 0 -lv 4

What each one is for:

• --parallel 1: Clef runs one request per batch anyway (llama.cpp PR #29831), so extra slots add nothing. Also, #30482 reports HTTP 500 on /v1/systemone with two or more slots.
• -b 4096 -ub 4096: the whole request must fit in one batch, so 4096 tokens is the largest request.
• -c 16384: the context size others had tested with. Clef keeps no KV cache, so it costs almost nothing.
• --cache-ram 0: in this build, Clef requests fill the RAM prompt cache and are never reused (fixed later in #30498).
• No --mmproj: I only send text.

All three questions about a message go in one request, because Clef decides the questions of one request together. Sent one at a time, about 8% of answers were reported to change.

Memory and start-up

Test 1-2: API types (the example from the llama.cpp server docs)

Message: "Customer message: I was charged twice for my order last week and nobody has replied." One request, 354 input tokens, 0.285 s.

Test 3: my real job, three languages

The three yes/no questions, word for word:

• rule_breach: "Does this message ask to share or misuse confidential company information or the personal data of other people (for example IC numbers, home addresses or salaries)?"
• needs_company_handbook: "To answer this message well, do we need to look up the company rules, policies or employee handbook?"
• needs_malaysian_law: "To answer this message well, do we need to look up Malaysian law or government regulations?"

Each message was sent in English, Malay and Chinese. "Yes" means a probability of 0.5 or more. Numbers are English / Malay / Chinese.

That's 17 of 18 as expected in each language, 51 of 54 overall. The only difference was the same one in all three languages: it said the personal-data message also needs Malaysian law. It caught the rule breach every time.

Speed, one request at a time: fastest 0.159 s, middle 0.161 s, slowest 0.295 s.

Test 4: a message that's too long

A message of 7,358 tokens came back as HTTP 500: input (7358 tokens) is too large to process. increase the physical batch size (current batch size: 4096). The server stayed healthy, so cut long messages down before sending.

Test 5: busy box

I sent 10 Clef requests at once while Saluki served 10 chats and the image server made one image, all on the same Spark:

• Clef: all 10 answered in 5.3 s, each taking 0.55-5.31 s. They wait in line, one at a time.
• Same answers: the probabilities matched the same messages sent alone (largest difference 0.0000).
• The other two: Saluki was all HTTP 200, slowest 54.1 s; the image came back HTTP 200 in 49.3 s.
• Memory: lowest free memory on the box was 22.2 GiB.

Reboot

I rebooted the box with all three services enabled. All three came back by themselves, and the same billing request gave exactly the same answer as before the reboot (billing 0.9614). Free memory after the reboot was 36 GiB.

Things that tripped me up

1. Expected start-up warnings:
   • no API key is set and CORS allows all origins
   • model default pooling_type is [-1], but [0] was specified
   • embeddings required but some input tokens were not marked as outputs -> overriding
   • speculative decoding not supported by this context
   • --cache-idle-slots requires --cache-ram, disabling (because of --cache-ram 0)
2. Version text from a shallow clone. A --depth 1 clone prints build 1, commit 358a420, not the release number.
3. The probabilities aren't tuned to your messages. Pick your cut-off after looking at real traffic; I used 0.5 for these tests.

Stuff I hadn't tested: image input, and inputs near the 4096-token limit other than the too-long case.