amx-moe-eda-w2048-bugfixinit

A 30M-parameter block-routed MoE (3.3M active per token) fine-tuned to write C functions for a small EDA-tooling corpus. Trained entirely on one Intel AMX CPU core.

This release differs from amx-moe-eda-p250x4 in two ways, both measured rather than assumed:

1. Attention window 2048 instead of 256.
2. Initialised from a bug-fix curriculum on real C rather than from the base pretrain.

What it can and cannot do

Read this before using it. On 198 held-out EDA blocks -- programs it never trained on -- it reaches 1.2269 nats and 0% exact match. Of 61 held-out blocks compiled after substitution into the real program, 13.1% compile and 8.2% both compile and reference anything outside their own signature. It does not write correct C from a specification. It is a research artifact for studying what a CPU-trained model of this size does, not a coding assistant.

Measured results

Held-out EDA blocks, 198 targets, each arm at its own best checkpoint:

init	window	held-out CE
4-day base	256	1.3358
4-day base	2048	1.3054
bug-fix curriculum	2048	1.2269

Paired against the 4-day init at window 2048, on identical blocks:

-0.0443 nats, t = -3.02, significant; wins on 122/198 blocks (62%)

Compile rate on 61 held-out blocks, against the 4-day init:

	4-day init	this model	paired p
compiles	8.2%	13.1%	0.508
compiles and is not a stub or hollow	6.6%	8.2%	1.000
hollow	6.6%	14.8%	0.125

The compile advantage is not established, and its extra compiles are disproportionately hollow -- bodies that reference nothing outside their own signature. The cross-entropy gain is the real one.

Comparison with the previous release

The prior model is gdiamos/amx-moe-eda-p250x4. Both are the same architecture and the same 250-program corpus; this one adds window 2048 and the bug-fix init.

The two releases were selected by different criteria, and that matters more than the architecture difference. p250x4 published its step-20,631 checkpoint deliberately -- its card calls it "the fitted end of that trade", chosen because fit buys compile rate even as held-out loss gets worse. This release publishes step 4,125, chosen by held-out cross-entropy. So the honest comparison is two tables, not one.

As published: what you get if you download each

All numbers re-measured here on the same 61-block held-out set, so they are comparable to each other (they differ slightly from the p250x4 card, which used n=67 and its own eval run).

	p250x4 (step 20,631)	this model (step 4,125)
held-out cross-entropy	1.6892	1.2269
blocks that compile	8.2%	13.1%
compile and not stub/hollow	6.6%	8.2%
hollow	1.6%	14.8%
exact match	0%	0%

Better on loss and on raw compile rate, and noticeably worse on hollow output -- nearly one in seven of this model's held-out generations compiles while referencing nothing outside its own signature, against one in sixty for p250x4. If you are counting compiles as successes without the guards, this model will flatter itself more than the old one did.

Controlled: both at step 4,125, same eval

Strip out the checkpoint-selection difference and the picture narrows:

	p250x4 arm	this model
held-out cross-entropy	1.3358	1.2269
blocks that compile	14.8%	13.1%
compile and not stub/hollow	9.8%	8.2%

Matched checkpoint for checkpoint, the gain is cross-entropy only. On compile rate the older configuration is nominally ahead, and a paired test over the same blocks puts the difference at p=1.000 -- i.e. no difference either way. Neither window 2048 nor the bug-fix init is shown to produce more compilable C on this task.

What is actually established

• Cross-entropy: −0.0443 nats against the same-architecture 4-day init, t = −3.02, winning on 122/198 blocks. Significant and broad.
• Compile rate: nothing. Six paired tests across these arms, none reaching p < 0.05 at n≈61 with 5–13 events per arm.
• The reason the window helps less than its mechanism suggests: only 34.5% of EDA target lines appear verbatim in the prompt, against 96% on the bug-fix task. Retrieval pays only where there is something to retrieve.

Why the window is 2048

The copy circuit in this model family is hard-capped at the training window. Asked to repeat a literal span, the window-256 model scores 74.9% token accuracy at distance 259 and 0.2% at 367 -- a step function, not a decay. Held-out EDA block prompts have a median length of 882 tokens, so at window 256 roughly 72% of the skeleton a block must copy identifiers from was outside the model's reach.

Widening it helps less here than the mechanism suggests, because only 34.5% of EDA target lines appear verbatim in the prompt (against 96% on the bug-fix task). Retrieval only pays where there is something to retrieve.

Why a bug-fix init

The bug-fix curriculum -- 24,539 distinct C functions with one injected bug each -- was built to install a copy circuit and failed at that goal: the copy probe was unchanged at every checkpoint. It still transfers usefully here, most likely as 15k functions of real C rather than as a circuit.

Training

parameters	30,029,429 total / 3,315,744 active per token
architecture	6 layers, [lin, swa, swa, lin, swa, lin], d_model 256
MoE	64 experts, k=4, 2 shared, route_block 256
window	2048, chunked
hardware	one Intel AMX CPU core
init	bug-fix window-2048 model, itself from a 1.75B-token pretrain
fine-tune	84.5M tokens; best checkpoint is step 4,125, not the final

The released checkpoint is step 4,125. The final step 20,631 checkpoint scores 1.5251 -- 24% worse -- because 200 passes over 250 programs is far past the optimum. Anyone reproducing this should early-stop.

Known limitations

• 0% exact match on held-out blocks. Generation derails over long outputs: per-token accuracy on copyable spans is about 52%, which compounds.
• Hollow outputs. 14.8% of held-out generations compile while computing nothing. Check outputs against the stub/hollow guards before counting a compile as a success.
• Trained on one narrow corpus. It speaks this EDA dialect and its nrl runtime, not C in general.
• Greedy decoding derails; sampling and repetition penalties were swept and none produced a significant improvement.

License

Apache 2.0.