vtava/Qwen3.5-0.8B-CeNN-Integrated-V1

Qwen3.5-0.8B + TinyCeNN Integrated Memory V1

Experimental text-backbone adapter for Qwen/Qwen3.5-0.8B. The original Qwen3.5 model has 24 text decoder layers: 18 native Gated DeltaNet linear-attention layers and 6 full-attention layers. This experiment leaves all native linear-attention layers untouched and replaces selected full-attention layers with TinyCeNN bounded memory.

Validation-selected candidate: partition_conservative
Replaced full-attention layers: [3, 23]
Remaining original full-attention layers: 4
Base revision: 2fc06364715b967f1860aea9cf38778875588b17
TinyCeNN source commit: 2e4deedc7eddbb276b42ca683e752ad5f684b735

This repository stores an adapter checkpoint plus the exact custom loader/source required to reconstruct the model. It is not a standalone save_pretrained() checkpoint and does not include the original Qwen weights.

Architecture

Qwen3.5-0.8B uses a 3:1 hybrid text stack (Gated DeltaNet linear attention plus periodic full attention). TinyCeNN is applied only to the original full-attention positions. Qwen3.5's Q/K normalization, partial MRoPE, and post-attention output gate are preserved. The TinyCeNN readout remains explicit because it cannot be folded through Qwen3.5's elementwise output gate without changing the computation.

The cenn_partition memory keeps sink/current/previous-block information exact and compresses older history into a bounded recurrent state. Therefore this is not "attention-free" in the strict sense: local exact attention remains inside the replacement, while unbounded global full attention is replaced.

Held-out benchmark

Selection used validation NLL only; held-out test documents were not used for model selection.

Context	Test NLL	PPL	PPL / original	PPL / matched adapted control	Cache / original	Prefill speedup	Decode speedup
128	3.0251	20.595	0.9581	1.0066	1.0139	0.923	0.945
256	2.9552	19.205	0.9693	1.0050	0.9900	0.932	0.938
512	2.8588	17.441	0.9798	1.0112	0.9510	0.944	0.930
1024	2.7747	16.033	0.9953	1.0206	0.8957	0.966	0.950
2048	2.7571	15.754	1.0045	1.0306	0.8315	0.970	0.949

The current implementation is research PyTorch, not a fused production kernel. Cache reduction and quality should be interpreted separately from wall-clock speed.

Load

from load_model import load_model
model, tokenizer = load_model('.')

For generation, use batch size 1 with the custom greedy_generate helper in tinycenn_lm.qwen35_integrated_memory until a standard Transformers cache/generation integration is packaged.

Reproducibility files

manifest.json, selection.json, integrated_report.json, validation/test CSVs, and generation examples are included. The manifest pins the exact base-model revision, dataset revision, source commit, package versions, split hashes, and experiment settings.

Limitations

• Experimental research adapter; not a production model.
• Text backbone only. The original Qwen3.5 vision tower is not modified or packaged here.
• Benchmark confidence intervals are over held-out documents, not multiple independent training seeds.
• Custom TinyCeNN cache currently targets batch-one greedy decoding; beam/batch cache reordering is unsupported.
• Replacing all six full-attention layers removes original quadratic global attention from the text backbone, but native Gated DeltaNet layers and TinyCeNN local exact attention remain.

Licenses

The base Qwen3.5 checkpoint is Apache-2.0. TinyCeNN-LM source is distributed under its repository license; a copy is included when available.