wigcheng5566/laya-neutron-gguf

Laya GGUF for llama.cpp — encoder and decision head

Laya returns calibrated probabilities over the options of a typed question instead of generating tokens. It is two pieces: a bidirectional encoder, and a decision head that reads the encoder's hidden states at the option-marker positions.

The GGUF conversions already on the Hub are encoder-only — they give you embeddings and leave the head in PyTorch. This repo ships both, so the whole pipeline runs inside llama.cpp:

tokens + marker positions
        -> encoder (llama.cpp)
        -> hidden states at markers
        -> decision head (ggml graph)
        -> calibrated probabilities

The head is built as a ggml graph by laya-score: two pre-norm transformer layers, an option-marker scorer (LayerNorm -> Linear -> GELU -> Linear), a question-type embedding, an act/escalate head, and per-bucket temperatures.

Everything runs on CPU. On an NXP i.MX95 it also runs on the Neutron NPU through a llama.cpp CPU extra buffer type that claims the Q4_0 matmuls — same repo, built with -DGGML_CPU_NEUTRON=ON.

Files

file	size	what
laya-encoder-q4_0.gguf	227 MB	English encoder, ModernBERT-large (1024 hidden, 28 layers, 2624 FFN)
laya-head-f16.gguf	51 MB	English decision head
laya-tokenizer.json	3.4 MB	English tokenizer
laya-tokenizer_config.json	337 B	special-token names for the above
laya-ml-encoder-q4_0.gguf	228 MB	Multilingual encoder, mmBERT-base (768 hidden, 22 layers, 1152 FFN, 100+ languages)
laya-ml-head-f16.gguf	29 MB	Multilingual decision head
laya-ml-tokenizer.json	33 MB	Multilingual tokenizer — required
laya-ml-tokenizer_config.json	524 B	special-token names for the above
tokenize_laya.py	4 KB	builds the input sequence and marker positions

The tokenizer is not optional

The multilingual encoder's vocabulary is BPE with a Metaspace pre-tokenizer. llama.cpp has no Metaspace-from-tokenizer.json path, so the vocab embedded in the GGUF cannot be used to tokenize text — it is there only so the weights load. Token ids must come from the shipped tokenizer.json, which is what tokenize_laya.py does.

Usage

# 1. build laya-score
git clone -b neutron-npu https://github.com/wigcheng/llama-neutron
cmake -B build llama-neutron -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target laya-score

# 2. tokenize (writes tokens + marker positions as JSON)
python3 tokenize_laya.py \
    --tokenizer laya-ml-tokenizer.json \
    --tokenizer-config laya-ml-tokenizer_config.json \
    --type choice --ins "Which subsystem is at fault?" \
    --opt "fan" --opt "power" --state "the fan is noisy"
# {"tokens": "2,6241,2872,235292,12236,113078,603,696,15639,235336,1,4,5683,4,2384,1,573,5683,603,52806,1",
#  "markers": "11,13", "n_tokens": 21, "options": ["fan", "power"], ...}

# 3. score
./build/bin/laya-score -m laya-ml-encoder-q4_0.gguf --head laya-ml-head-f16.gguf -t 6 \
    --type choice --tokens 2,6241,...,1 --markers 11,13
# {
#   "type": "choice",
#   "tokens": 21,
#   "temperature_bucket": "choice:2",
#   "temperature": 1.000000,
#   "choice": "option0",
#   "logits": [1.699239, -0.916494],
#   "probabilities": [0.931867, 0.068133],
#   "options": ["option0", "option1"]
# }

--tokens carries no option labels, so the answer comes back as option0/option1 — positional, in the order the options were given to tokenize_laya.py. Pass --opt to laya-score instead of --tokens and it reports the labels.

--serve prints a compact one-line form instead, one JSON per request.

Loading the encoder takes ~20 s on an A55-class board, so anything interactive should use --serve: the process stays resident and answers one request per line on stdin.

The English encoder's vocabulary can be tokenized by llama.cpp, so it also takes the question directly — no tokenize_laya.py step:

./build/bin/laya-score -m laya-encoder-q4_0.gguf --head laya-head-f16.gguf -t 6 \
    --type choice -i "Which subsystem is at fault?" \
    --opt "fan" --opt "power" -s "the fan is noisy"
# {
#   "type": "choice",
#   "tokens": 21,
#   "temperature_bucket": "choice:2",
#   "temperature": 1.906356,
#   "choice": "fan",
#   "logits": [1.773302, -0.004450],
#   "probabilities": [0.717590, 0.282410],
#   "options": ["fan", "power"]
# }

Provenance

• Encoders and head weights: convaiinnovations/laya and convaiinnovations/laya-multilingual (Apache-2.0), from NandhaKishorM/laya.
• Backbones: ModernBERT-large (English) and mmBERT-base (multilingual).
• This repo adds the GGUF conversion of the decision head and the llama.cpp runtime for it. Encoders are quantized to Q4_0; heads are kept at F16.

Licensed Apache-2.0, following the upstream models.