Qwen3.5-4B-Q8_0-FlashNgram-PLEAdapter-MTP

Experimental research checkpoint. This is not an official Qwen release, not a Qwen 3.8 conversion, and not a claim of general capability improvement.

This model starts with a Qwen 3.5 4B Q8_0 MTP backbone and adds a small, trainable PLE-inspired adapter before transformer block 2. The adapter reads a frozen Qwen 3.8 Flash n-gram lookup table using the current token plus two predecessor token IDs, then adds a learned residual contribution to Qwen 3.5.

What was added

• Frozen Qwen 3.8 Flash per_layer_token_embd.weight n-gram table: IQ4_NL, shape [160, 320001536].
• 16 table heads: 8 bigram and 8 trigram hashes.
• About 13.1M newly trainable adapter parameters:
  • bias-free W_key and W_value projections;
  • key, query, and convolution RMSNorm scales;
  • causal depthwise convolution (kernel 4, dilation 3);
  • one scalar residual scale, alpha.
• One residual injection immediately before Qwen 3.5's zero-based block 2 input RMSNorm.

The original Qwen 3.5 backbone, its 32 transformer layers, MTP tensors, and LM head remain frozen and intact.

Adapter computation

For each position, the current token and two predecessors select 16 rows of the frozen table and produce a 2560-wide lookup vector p:

key   = W_key(p)
value = W_value(p)

gate = sigmoid(signed_sqrt(
    dot(RMSNorm(key), RMSNorm(hidden)) / sqrt(2560)
))

gated_value = gate × value

delta = alpha × (
    gated_value
    + SiLU(CausalDepthwiseConv(RMSNorm(gated_value)))
)

hidden_before_block_2 = base_hidden + delta

At alpha = 0, delta = 0, giving the exact alpha-zero backbone control.

Runtime requirement

This GGUF requires the accompanying PLE-adapter-aware llama.cpp build. A standard upstream llama.cpp binary does not execute this experiment's trainable adapter graph correctly.

.\llama-server.exe `
  -m .\Qwen3.5-4B-Q8_0-FlashNgram-PLEAdapter-MTP.gguf `
  --n-gpu-layers 99 `
  --ctx-size 48192 `
  --flash-attn on `
  --jinja `
  --reasoning on `
  --spec-type draft-mtp `
  --spec-draft-n-max 2 `
  --port 8080

Do not pass --ngram-gate. That option belongs to an earlier direct raw-lookup graft; this model uses the adapter's stored alpha instead.

Training

The adapter was trained in PyTorch with the backbone and n-gram table frozen. The initial pilot used packed WikiText-2 next-token prediction. Training records preserve the two predecessor IDs needed for the PLE hash, and document boundaries use the PLE EOS token configured in the GGUF.

Later experiments used 256-token packed records, held-out validation, periodic checkpoints, and best-checkpoint retention. This is language-modeling training, not HLE-specific instruction tuning.

Evaluation and limitations

The adapter reduced held-out WikiText-2 next-token perplexity relative to the alpha-zero backbone control in development runs. This result is encouraging evidence that the lookup reaches the model and can be optimized, but it is not evidence of general reasoning, factuality, safety, or benchmark improvement.

In particular:

• HLE smoke evaluation did not demonstrate an improvement.
• Development validation was used for checkpoint and hyperparameter choices; it is not a pristine final test set.
• A matched real-table versus per-head-shuffled-table control has not yet been completed.
• PyTorch-to-llama.cpp numerical parity for the exported quantized checkpoint has not yet been established.

Use this model as an experimental artifact, not as a production or safety- critical model.

License and redistribution

This artifact incorporates a Qwen 3.5 base model and a Qwen 3.8 Flash lookup table. Before publishing or redistributing it, verify and comply with the licenses, acceptable-use requirements, and redistribution terms for all base artifacts and every training dataset used. This repository does not grant any additional rights to those components.

Attribution

Please describe this model as a Qwen 3.5 PLE-inspired n-gram adapter experiment, rather than an official Qwen model or a Qwen 3.8 Flash model.