NoTokenLM-Gen-3.5-Focused

A 9M-parameter, byte-level, tokenizer-free language model -- fine-tuned specifically to fix its predecessor's biggest weakness: long-range coherence.

Part of the NoTokenLM family. If you haven't seen NoTokenLM-Gen-3.5 yet, read that card first -- this model is a direct, short, targeted continuation of it, not a new architecture.

This card was last auto-updated from checkpoint step 45,600 (session #9) on 2026-08-19.

---

What changed, and why

Gen-3.5's own model card was honest about its biggest weakness: "Long-range coherence (3+ sentences) degrades noticeably. [...] the model starts losing track of who the subject is." We measured this directly: ~38-42% of generations were fully coherent on a 100-prompt manual evaluation.

Focused is a continuation training run on top of the Gen-3.5 checkpoint, using a narrower, simpler-narrative dataset (TinyStories-only, after an earlier c4/math/synth mix was found to inject a "word-problem" register into unrelated prompts) chosen specifically to reinforce subject tracking and short-story structure, rather than adding new capability or new knowledge.

Result: on the same style of evaluation (200 generations, 10 held-out prompts, temperature 0.6), coherence rose to ~86%.

This is not a new model bolted onto the old one. It's the same 9M architecture (RoPE, RMSNorm, SwiGLU, byte-level vocabulary, weight tying), same weights carried forward, just pushed further in the one direction its predecessor was weakest at.

---

Real examples, unedited (from the 200-generation evaluation)

Subject tracking across 2-3 sentences, which was Gen-3.5's main failure point:

"She walked into the room and saw a big machine. She was curious and wanted to see what i[t was]"
"The door opened and a little girl came out to see what was inside. She saw the [...]"
"He looked at the sky and saw a big cloud. He saw the sun shining in the sky with the [...]"
"The little girl was so excited to go on an adventure. She wore her favorite [...]"
"He said to his mom, 'I have a surprise for you!' His Mom was very e[xcited]"

Note the consistent pronoun tracking (she/she, he/he, his/his) across multiple clauses -- this is exactly the failure mode Gen-3.5's card documented ("a 'she' quietly becomes a 'he' a few sentences later") and it's substantially reduced here.

We're not hiding the misses either. Out of 200 generations, roughly 1 in 7 still drifts or contradicts itself:

"The old man was a girl. She was very popular."          <- contradicts its own subject mid-sentence
"The little girl was a little boy who liked to walk..."   <- same failure mode, different prompt
"They walked into a big forest with a long direction."    <- grammatically fine, semantically empty

That failure rate (~14%) is real and we're reporting it, not the ~86% alone.

---

What this model is and isn't

• It is: the same base capability as Gen-3.5 (real word morphology, dialogue formatting, short-range grammar), with meaningfully better multi-sentence subject consistency.
• It is not: a chat or instruction-following model. It has not seen QA-formatted or instruction data. Prompt it with a sentence start, not a question expecting a direct answer.
• It still has no reliable arithmetic ability and no reliable factual/world knowledge. An earlier checkpoint (still trained on a c4/math/synth mix) showed partial operation-type recognition on word problems (correctly picking + or *) but consistently wrong arithmetic; the current TinyStories-only checkpoint doesn't attempt arithmetic at all anymore and treats math-shaped prompts as story openers instead. Neither is a reliable math capability.
• Style has shifted toward simpler, shorter-sentence narrative (closer to children's-story structure) compared to Gen-3.5's more 19th-century-novel register, as a side effect of the dataset used for this focused pass. If you want the more literary tone, use base Gen-3.5; if you want more reliable multi-sentence coherence, use this one.

---

How to run it

Recommended temperature: 0.5-0.6 (0.6 was used for the evaluation above).

Use the "Open in Colab" button in this repo's action menu (top of this page) for a zero-setup run -- no token needed.

From Python directly

import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "omurberaisik/NoTokenLM-Gen-3.5-Focused", trust_remote_code=True
)
model.eval()

prompt = "She walked into the room and"
ids = torch.tensor([list(prompt.encode("utf-8"))], dtype=torch.long)
out = model.generate(ids, max_new_tokens=100, do_sample=True, temperature=0.6, top_k=40)
text = bytes(x for x in out[0].tolist() if 0 <= x <= 255).decode("utf-8", errors="replace")
print(text)

No tokenizer object needed -- this is a byte-level model (256-value vocabulary, the raw UTF-8 bytes), so encode/decode is just str.encode("utf-8") / bytes.decode("utf-8").

Architecture details

Parameters	9,055,440 (9,053,520 in this repo's safetensors -- the difference is the deep-supervision aux heads used only during training, dropped here since they're not needed for inference)
Layers	13
d_model	240
Attention heads	4
Feedforward dim	640
Vocabulary	256 (raw bytes, no tokenizer)
Context length	1024 bytes
Position encoding	RoPE
Normalization	RMSNorm
Feedforward	SwiGLU
Output layer	Weight-tied to input embedding
Base checkpoint	NoTokenLM-Gen-3.5
This stage's training	TinyStories-only continuation run, narrative-consistency-focused
Training progress	step 45,600, 9 session(s), best val_loss 0.8514
Format	safetensors, trust_remote_code=True (custom architecture, standard AutoModelForCausalLM.generate() API)

What's next

Whether future continued training holds onto this checkpoint's coherence gains while adding new capability (broader world knowledge, more reliable reasoning) without regressing subject-tracking is an open, testable question -- we'll report the real numbers when we have them, same as here.

---

Part of the NoTokenLM family -- small models, built and evaluated honestly.