win10/spark-temporal-engram-4b-v3

Spark Temporal Engram V3: QA round three

All eight predeclared acceptance gates pass.

Complete merged checkpoint with a BF16 4B backbone and FP32 Engram. Round three continues all round-two weights with a fresh optimizer. The backbone, tied LM head, and original copy gate remain frozen; no external adapter is required.

Checkpoint	NLL (32 records)	Recall	Order	Natural QA	Paraphrased QA	Temporal QA	Unobserved Unknown	Native BF16/FP32 GPU clean load
Base	1.304515068	0/12	1/12	0/64	0/64	0/64	64/64	Not recorded here
Initial V3	1.336399075	12/12	1/12	Not evaluated	Not evaluated	Not evaluated	Not evaluated	Not recorded here
Step 200	1.036682900	5/12	7/12	Not evaluated	Not evaluated	Not evaluated	Not evaluated	Not recorded here
Step 800	1.019088555	0/12	6/12	Not evaluated	Not evaluated	Not evaluated	Not evaluated	Not recorded here
Recall fix	1.019911441	12/12	6/12	Not evaluated	Not evaluated	Not evaluated	Not evaluated	Not recorded here
Order fix	1.020251819	12/12	12/12	0/64	0/64	0/64	64/64	Not recorded here
QA round 1	1.020251819	12/12	12/12	14/64	14/64	0/64	64/64	Passed
QA round 2	1.020031746	12/12	12/12	22/64	21/64	52/64	64/64	Passed
QA round 3	1.020014452	12/12	12/12	61/64	61/64	49/64	64/64	Passed

The nine-column table retains all nine checkpoint comparisons. Legacy denominators are 32 NLL records / 22,797 supervised tokens, 12 recall cases, and 12 order cases. QA retains the same 256-question regression suite. Missing historical QA results are explicitly marked. The GPU column concerns native mixed-dtype loading of the QA exports.

Acceptance: natural QA 61/64 (required >=50/64); paraphrased QA 61/64 (required >=40/64); temporal QA 49/64 (required >=48/64); Unknown 64/64; recall 12/12; order 12/12; NLL 1.020014452, relative difference 0.001695% from round two (1.020031746). NLL acceptance is <=1.02 or a relative difference <=1%.

Temporal breakdown: first 15/16, last 16/16, before 2/16, after 16/16.

Training configuration

The inference architecture and its 1,526,417 trainable parameters are unchanged from round two: the complete QA reader, original temporal input projection and GRU, and both original read layers. Boundary/date heads, static tables, dynamic projection, sidecars, backbone, and copy gate remain frozen. A new memory namespace isolates the updated evidence geometry; configured context and memory capacities are retained.

Each update accumulates 20 microbatches: two trace QA (rotating over four sources), ten natural QA, four temporal QA, one recall, one order, and two SFT. QA is 80% of microbatches and natural QA is 50%. One eighth of QA microbatches supervise Unknown without observation. Every temporal relation has one slot per update. Recall/order replay weight is 2 and frozen round-two teacher SFT KL weight is 20.

Natural QA uses five source groups per update and two different target fields per source. Source observation can be reused within the update; each question has independent reset reader state, a snapshot of evidence, and fresh KV. Target order is shuffled per document before its deterministic visit schedule, covering every source position during the declared run.

Data include 4,811 retained trace QA records, 7,718 natural questions across 1,200 training documents, and 3,200 temporal training questions. Natural training documents have 2 facts: 51, 4 facts: 178, 6 facts: 432, 8 facts: 539. Six/eight-fact documents account for 80.917% of training documents.

Natural sources include same-noun/different-attribute and same-attribute/different-noun distractors, mixed fields, multiple sentence forms, reordered questions, and 4/5/6-digit identifiers with varying token lengths. Every fact has its own labeled question, with question form randomized independently of source position. Source-text parsing independently verifies answers; token offsets verify full copy spans and all candidate value starts. Hand-authored multi-fact and temporal reference cases validate labels without generator index logic. Wrong-label, wrong-span, wrong-candidate, and split-leak injections are rejected.

The fixed formal schedule contains 5,250 observed natural queries; 4,257 use six/eight facts. All first, middle, and final target positions are supervised. Training/development source groups, source text, and answer values are disjoint. New development question wording uses a held-out template family. All 256 evaluation questions and 448 fixed-suite values are excluded from QA and raw replay sources; the selected SFT replay inputs exclude the 32 NLL fixtures.

Natural source-position CE weight is 1.5 (trace/temporal 0.5). Natural first-token contrastive CE over all observed candidate value starts has weight 1. Original-reader event CE weight is 0.2. First-token and EOS weights are each 3 in vocabulary and source-position CE; an additional final-token CE has weight 0.5. Full source-position supervision trains continuation within the selected value and a distinct EOS target after its last token. Candidate locations, answer spans, event IDs, and reference values enter only loss computation after model scores. Inference receives no labels, field rules, or answer lookup table.

BF16 CUDA AMP; BF16 frozen backbone / FP32 Engram; AdamW, no weight decay, clipping 1, warmup 20, cosine schedule, seed 863033. QA LR 0.0005; original temporal LR 0.00002. Gradient checkpointing is disabled, as in round two. The single formal run ends at step 600 (136.29 minutes); eight smoke updates are excluded from formal initialization. Initial weights and checkpoints at 2, 10, 50, 100, 200, 300, 400, 500, and 600 retain optimizer/scheduler/RNG state. Smoke verifies checkpoint loading, real GPU inference, and gradients in all 39 selected tensors. Its data are archived separately; before formal launch, the final corpus randomized question-form/position assignments while preserving every source token and target span.

Formal examples: {"order": 600, "qa_fable:observed": 300, "qa_hermes:observed": 225, "qa_hermes:unknown": 75, "qa_natural:observed": 5250, "qa_natural:unknown": 750, "qa_ox:observed": 300, "qa_temporal_after:observed": 525, "qa_temporal_after:unknown": 75, "qa_temporal_before:observed": 525, "qa_temporal_before:unknown": 75, "qa_temporal_first:observed": 525, "qa_temporal_first:unknown": 75, "qa_temporal_last:observed": 525, "qa_temporal_last:unknown": 75, "qa_trace:observed": 225, "qa_trace:unknown": 75, "recall": 600, "sft": 1200}. All 21 frozen Engram tensors remain unchanged, and backbone parameters are excluded from the optimizer.

Claude design delegation failed with exit code 1: "You've hit your weekly limit · resets Oct 16, 10pm (Asia/Taipei)". No external review conclusions or changes were adopted. The implementation and checks were completed by the primary agent.

Failure diagnosis

The fixed suite was already exposed for the requested residual diagnosis and measures regression, rather than unseen generalization. No fixed-suite result selects a checkpoint or starts further training. The final step-600 endpoint is evaluated once. Combined data and loss changes do not independently identify their individual causal effects.

Natural QA is 32/32 with four facts and 29/32 with eight facts. Paraphrased QA is 32/32 and 29/32 respectively. Round two was 16/32 and 6/32 for natural QA.

All three remaining natural failures occur in eight-fact documents and confuse backup keys with archive keys; their target facts are the first source sentences. The same three documents fail under paraphrased questions. This is a residual error pattern, not an isolated causal explanation.

Temporal QA declines from 52/64 to 49/64, while meeting the predeclared 48/64 gate. Before declines from 6/16 to 2/16; after improves from 15/16 to 16/16. First and last remain 15/16 and 16/16. One temporal-first output contains an unmatched code and a readout event switch; cross-event fragment errors are not fully eliminated. All 256 outputs generate EOS, with no token-cap hits or malformed digit-length codes.

Candidate membership and output failures:

Category	Correct / total	Wrong observed value among failures	Target event in first-step pool	Target event in first-step selected set	Missing EOS	Token-cap hits
direct	61/64	3/3	64/64	64/64	0	0
paraphrase	61/64	3/3	64/64	64/64	0	0
temporal	49/64	14/15	64/64	64/64	0	0
unobserved	64/64	0/0	0/64	0/64	0	0

There are 21 primary failures, retained individually with inputs, expected answers, raw predictions, and readout diagnostics. Among failures, 1 contain a complete code not present in the source, 0 contain malformed digit lengths, and 1 have a change of readout event. Event switches are diagnostic evidence, not proof of cross-event token splicing. Pool/selected membership means the correct event remains available; it does not establish correct question-to-value binding.

All 256 primary queries preserve stored evidence. Source-recitation flags: 0. Source-in-context controls: 0/16; separate empty-memory controls abstain in 8/8 cases. Controls have separate denominators and do not enter acceptance.

Held-out development

Development category	Correct / total
natural_2_facts	4/4
natural_4_facts	4/4
natural_6_facts	4/4
natural_8_facts	4/4
temporal_after	3/4
temporal_before	1/4
temporal_first	1/4
temporal_last	4/4
trace_fable	7/8
trace_hermes	8/8
trace_ox	5/8
trace_trace	5/8
unobserved	16/16

Development results use separate source groups and a held-out question template family. The 16 natural probes each query the first fact in a different document; they test wording and fact-count transfer without establishing balanced position generalization. They do not select the checkpoint or enter the primary denominator. Underlying trace documents may have appeared in earlier SFT. This experiment does not claim universal QA or calendar extraction; temporal source order agrees with timestamp order.

Export and evaluation

Five merged safetensors shards total 8,237,343,992 bytes (7.671624 GiB). Stored tensors: 290 BF16 backbone tensors (4,112,079,360 parameters) and 60 FP32 Engram tensors (3,286,298 parameters). The complete Engram export is verified against the final checkpoint. Native clean GPU loading checks zero missing/unexpected/mismatched keys, CUDA placement, expected dtypes, tied embedding/LM-head storage, and finite forward outputs.

Fixed-suite generation retains greedy decoding, max_new_tokens=96, EOS 1, pad 2, and the native non-thinking chat template. Each source is observed once with sentence-end annotations only; source KV is discarded before the question. Question and answer memory writes and feedback are disabled. Complete-value scoring, competing-source-value rejection, Unknown scoring, and all denominators are unchanged.

Evidence: fixed plan, data/replay/sampling/baseline audits, raw smoke and formal logs, optimizer/RNG checkpoints, development outputs, clean-load audit, full predictions, failures, and residual diagnosis are retained under the round-three run. The export includes QA_ACCEPTANCE.json and FAILURE_DIAGNOSIS.json. Source licenses and notices are preserved.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "Spark-X2.5-4B-v3-qa-round3-merged-bf16"
model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda").eval()
tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
assert all(p.dtype == torch.float32 for p in model.model.engram.parameters())