i64 Systems PRO

good questions. the receipt has the machine at 27.4 GiB guest RAM, 18.1 GiB RSS, 7–9 GiB cache, and 4–6 GiB swap; nothing dropped caches between chunks. of the 2401 s wall, reads were 1025 s at 1.12 GB/s, verify 484 s at 2.38 GB/s, and madvise 79 s. so yes: page cache materially contributed--the 1.12 GB/s exceeds the 0.55 GB/s SATA device--but the cache was only 7–9 GiB against a ~43 GiB cold set, so the split was not measured. the NVMe rerun records device bytes beside engine reads and adds an O_DIRECT control

i also agree 1024 is the lever: our 120B traces put static top-32 at ~75% of picks and reselecting every 512 tokens at ~80–84%; the planner optimizes avoided stall, not hit rate. on verification, 121,952 × 3 + 4,608 is three tensors per expert—gate/up/down--each hashed once, not one slice hashed three times. we measured verification as the bottleneck and the next row moves SHA-256 onto the CPU SHA extensions with identical digests. distinct resident experts/chunk and the 380-vs-384 pick deficit are not receipt-backed yet, so i wont guess--they need instrumentation

this is the right order: read the failures before you believe the number, then go looking for the passes that could be quietly wrong. the two placeholder credentials are the finding of the post - asked about a value it never saw, the model produced something value-shaped twice in 1200. we hit the same class from the other side: our runtime refuses that by name (the answer must say the value was never given) rather than hoping the model does.

one tip from our side: freeze and digest the judge before the run and journal its version with every score, so 1189 -> 1200 is a versioned re-score on the same samples, not a moving target. and make the "quietly wrong pass" search a standing falsifier in the prereg, not a one-off. stay frosty🩵🥰

you know, i'll actually test all of this. i can run some experiments comparing:

Architecture Mamba : Attention
Mamba-only ∞:1
Mostly Mamba 7:1
Current model 3:1
More attention 1:1
Transformer-only 0:1

Then we can actually see which ratio performs best at this scale.

As far as I know, I don't think the attention layers were dominating the gradients, but I'll measure that when I run these experiments.

When I get to them, the models will probably be released under https://huggingface.co/basically-experimental

heck yes! ill keep an eye out🩵

youre right that identity was never the half that could break; the number that moves is misses per token. thank you for re-running the checks!!🩵 nice to meet you!

the qwen arm, K=32 of 128, code, 32,768 tokens: 12,451,840 picks, 9,954,590 misses (79.94%), so ~304 of 380 picks per token missed and the resident mass was 20.06%- below the 25% a uniform-random 32 would give. the pool is 14,495,514,624 B (your 14.50 GB is exact). lease events 121,952, verifications 370,464, zero mismatches. 1.15 TB of verified reads in the run. leased arm 13.65 tok/s eval vs 24.59 fully resident: 1.80x the wall, 67% of it read + sha256

two caveats: the row is batch-shaped (teacher-forced over 1024-token chunks), so a layer's misses lease once per chunk and amortise: 35 MB per token effective. single-token decode with a static set leases per token, ~2.9 GB, which is the regime your 0.28 tok/s describes - your order of magnitude is right there. and the fixed set was the preregistered one, not tuned on the run's trace; on the 120b a whole-run top-32 carries ~75% of uses and a set re-chosen every 512 tokens about 80 to 84%, while a prose-chosen set is near random on code. finer routing did not concentrate harder by itself; which 32 is the lever

one more measured thing: the lease tier turned out to be a SATA SSD (546 MB/s direct read), which is your 616 MB/s. it is on NVMe now at 2.8 GB/s on the same read; the 120b work runs on that tier next. the verify is the other half of the lease and is next. i was running tests with the 120b and conflated the two at print which is admittedly pretty embarrassing. but yes, those were SATA speeds

at the end of the day we are trying to bring 120b/235b grade local models to the market, using integer pipelines and deterministic reasoning- i saw you do AI infra, and hope you too can see the value in a deterministic local agent:) the first big use will be for context continuity; particularly around agentic implementation fleets with lots of compactions and handoffs- particularly design and critical automation apparatus. keep an eye on openbob coming out!

correction, 2026-09-03: the card says "96 on NVMe". the lease tier in this run was a SATA SSD (measured 546 MB/s direct read); the card is re-cut with the right word. the identity claim is unchanged - the same bytes on any bus - and the card carried no speed number. the store is on NVMe now (2.8 GB/s on the same read). runs were completed on NVMe with our 120b work- hence the confusion. im ready to be done with 30b experiments lol

@Hoglet-33 did you have attention layers dominating the gradient? im guessing an all ratio sweep at like 100% mamba 7:1 3:1 and 1:1 are already planned for a wide curve. seeing where the attention layers sit too

Nice work getting a 3:1 Mamba2/attention hybrid trained at this size:) On the saturation question above: it's cheap to settle at 10M- a loss curve, and the same curve for a transformer-only run at matched tokens, would show whether the extra tokens kept buying anything. I'd wanna read both. I'm curious how you landed on 3:1 rather than 2:1 or 4:1- factor of three seems to be a pattern at this scale.

keeping an eye on this!