Delta-Ngram

Learned writers for explicit n-gram memory. A writer converts arrived evidence into memory updates; a frozen language model reads the resulting memory when scoring or generating a continuation. Queries do not update memory.

This release contains three writer-only checkpoints and the code/configuration needed to reconstruct their writer modules. Obtain the matching frozen language models and tokenizers separately from their original repositories, using the exact revisions recorded in each package. This repository is not a standalone Qwen model and is not a PEFT/LoRA adapter.

Checkpoints

Directory	Frozen language model	Training endpoint	Stored tensor values	Tensor storage
qwen38-mainline-100m/	Qwen3.8-Flash-Next	99,997,072 loss targets	4,075,697	16,302,788 bytes, FP32
qwen38-memory-100m/	Qwen3.8-Flash-Next	99,997,072 loss targets	4,075,697	16,302,788 bytes, FP32
qwen3-0.6b-m1/	Qwen3-0.6B	99,999,524 loss targets	1,190,465	4,761,860 bytes, FP32

The native mainline writer encodes evidence with the frozen base reader. The native memory writer was continued from the 80M mainline checkpoint with memory-conditioned evidence encoding: each block reads the trajectory's detached pre-write memory. Both native checkpoints use fixed alpha = 1 and RMS-normalized writer evidence features with epsilon = 1e-6. The writer checkpoint and the chosen evidence-encoding mode are distinct settings; record both when comparing them.

The 0.6B M1 checkpoint uses the external-hash D1 implementation and is bound to Qwen/Qwen3-0.6B at revision c1899de289a04d12100db370d81485cdf75e47ca. It is not the earlier Qwen3-0.6B-Base pilot. Its memory readout has no additional trained projection, reader, or normalization weights. The writer files already include their gates and output heads.

The 0.6B reader table

The 0.6B runtime does have an explicit memory table. It starts empty: unwritten rows read as zero, and the writer populates rows from newly supplied evidence. It is not a separately pretrained reader table. The runtime archive includes hash addressing, sparse-table storage, empty-state construction, lookup, and readout. The configuration fixes 16,384 hash buckets, n-gram orders 2 and 3, and injection before decoder layer 1. A saved populated table would represent one particular evidence/session history, not a missing model parameter file.

By contrast, Qwen3.8's native pretrained T0 table is part of its separately downloaded official base model.

The native tensor counts include the 16 saved target-scale buffer values in addition to 4,075,681 writer parameters. The endpoint counts stop at complete trajectory boundaries below the nominal 100M budgets. Safetensors file sizes include a small metadata header in addition to the tensor bytes listed above.

Contents and integrity

Each checkpoint directory includes pure FP32 writer tensors, configuration, source-checkpoint provenance, and CPU export/load validation. Supporting runtime snapshots contain the project's implementation; they do not contain base-model weights. release_manifest.json lists the SHA-256 and size of every release file other than the manifest itself.

No optimizer state, training RNG, corpus, pre-populated session memory, or base-model weights are included. A new session begins with empty explicit memory.

Loading

Use the supplied loading helpers and the matching runtime snapshot. Loading must be strict: all expected tensor names, shapes, and FP32 dtypes must match. Do not load a native writer with the external-hash 0.6B reader, or exchange the two base model families.

From a downloaded copy of this repository, verify the 0.6B writer on CPU without downloading its language model:

python qwen3-0.6b-m1/load_writer.py --verify

Verify each native writer on CPU:

python runtime/load_native_writer.py qwen38-mainline-100m --extract-source
python runtime/load_native_writer.py qwen38-memory-100m --extract-source

Run the 0.6B chat demo, which separately downloads the pinned official language model and tokenizer:

python qwen3-0.6b-m1/load_writer.py --chat --device cuda --max-new-tokens 128

Install the dependencies listed in the accompanying runtime documentation first. The CPU checks validate exported tensors and module construction; they do not load a base model or claim to reproduce full GPU inference numerics.

The original native evaluation runtime uses the project's Megatron/Swift reader with its native n-gram addressing and split PLE precision implementation. It is not equivalent to calling an unmodified Transformers or vLLM generate with these writer tensors. Native A800/vLLM generation integration was still under validation when this weight release was prepared.

The 0.6B demo uses cache-free greedy generation, recomputes the full prefix at each generated token, and keeps queries read-only. Its explicit write commands control when evidence enters memory.

See each package's configuration and the runtime documentation for exact loading commands, base-model identity, and implementation versions.

Evaluation scope

These are research checkpoints, not independently replicated production models. The two native endpoints have completed DEV64 continuation evaluations. Their historical evaluations used different evidence execution layouts. A matched evidence-B1 2-by-2 evaluation separating writer weights and evidence-encoding mode was in progress when this release was prepared; this release does not claim that either native writer is superior based on that unfinished comparison.

Repeated evaluation executions measure execution variability, not variability across independently trained models. A larger donor-minus-self NLL gap alone does not imply better self NLL.

License

The original Delta-Ngram writer weights and project code in this release are provided under Apache-2.0; see LICENSE. Separately downloaded language models, tokenizers, and third-party libraries retain their own licenses. This repository does not redistribute those base models or change their terms.