NoTokenLM-Gen-3.6

A 13.9-million-parameter, byte-level, tokenizer-free language model trained on an English + French text mix. No subword vocabulary, no BPE — just raw UTF-8 bytes in, raw UTF-8 bytes out.

This is part of the NoTokenLM family: a series of small models built around one guiding question — how much can a genuinely small model do, if the architecture and training are done carefully, without leaning on scale to cover for weak design?

If you're looking for a model that answers questions, does math, or holds a conversation — this isn't that, and this card will tell you exactly why not. If you're curious what a ~14M-parameter byte-level transformer does when it's trained on a broad mix of educational web text, encyclopedic text and simple stories — keep reading.

---

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO_ID = "omurberaisik/NoTokenLM-Gen-3.6"
model = AutoModelForCausalLM.from_pretrained(REPO_ID, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(REPO_ID, trust_remote_code=True)

text = model.generate_bytes("Once upon a time, there was a little girl named Mia.",
                             max_new_tokens=200, temperature=0.5, top_k=40)
print(text)

trust_remote_code=True is required — this model ships its own architecture code (byte-level I/O, RoPE, RMSNorm, SwiGLU, plus a short input convolution, QK-norm, value residual and per-head output gates) alongside the weights. There is no subword tokenizer; AutoTokenizer here is a thin byte<->id mapping provided purely so the model loads the standard Hugging Face way.

An interactive Colab/Kaggle notebook with a prompt box and sampling controls is in this repo as notebook.ipynb.

Recommended sampling settings are in How to actually run this thing below.

What is this checkpoint, actually?

Gen-3.6 is a from-scratch pretraining run of a 13.87M-parameter byte-level transformer (12 layers, d_model=320). This card describes step 13,000, and every number in it comes from that checkpoint's EMA (exponential moving average) weights, which is the standard choice for evaluation at this scale.

Training volume: 13,000 steps × (batch 16 × gradient accumulation 4 × 1,024 bytes) ≈ 0.85 GB of text. The best validation loss recorded during the run was 0.8915 (nats per byte, ≈1.29 bits per byte, on a held-out split of the training mix); the released weights are the final step-13,000 EMA weights. Peak learning rate 1e-3, 500 warmup steps, weight decay 0.1. Pretraining only — no instruction tuning.

Unlike Gen-4.5, which was trained on narrow children's-story text only, Gen-3.6 was trained on a broad mix: educational web text, Wikipedia, simple stories and textbook-style text, in English and French (details in Training data). That difference drives most of what you'll see below — including where it is better than Gen-4.5 and where it is worse.

---

How well does it actually write? (1,000-prompt evaluation)

We generated 1,000 completions — 1,000 distinct prompts, one generation each, temperature 0.5, top-k 40, 35 new bytes — and every one of the 1,000 outputs was read individually and sorted into one of three categories. The prompt set, prompt order, ids and generation settings are identical to those used for Gen-4.5's evaluation. No filtering, no cherry-picking; everything below is the real distribution.

Grading criteria (same strict standard as prior NoTokenLM evaluations — "grammatically fine but says nothing coherent" does not count as a win):

Category	Definition	Count	Share
Fully coherent	Correct grammar and the sentence actually makes sense — characters, objects, and pronouns tracked correctly throughout.	780	78.0%
Grammar OK, meaning breaks down	Reads like well-formed English sentence-by-sentence, but drifts into a pronoun/gender mismatch, a non-sequitur detail, or an object described inconsistently with the scene.	215	21.5%
Grammar breaks down	The sentence structure itself collapses — stuck repetition, a garbled or malformed clause.	5	0.5%

Result: 78.0% fully coherent, 99.5% grammatically correct overall, on a test set with no repeated prompts and no simplification.

Roughly half of the middle category (112 of 215) is a pronoun/gender mismatch, covered in detail in the next section; the rest is fluent text whose content contradicts the scene.

The labels were assigned by a single annotator. The pronoun check and the benchmarks below do not depend on them.

The label for every one of the 1,000 outputs is in gen36_13000_1000_test_outputs.json under the category field.

Real examples, unedited, that landed in "fully coherent":

"On a quiet evening, Jack spent the whole morning at the park."
  -> " He had a big smile on his face and"

"One special day, Rosie spent the whole morning at the campsite."
  -> " She was so excited to see the firs[t]"

"One special day, Ben was playing in the attic when something strange happened."
  -> " He got scared and ran away. He ran"

"One sunny morning, Anna stopped to rest in the village."
  -> " She saw a big puddle on the ground"

Real examples that landed in "grammar OK, meaning breaks down" (also unedited):

"One sunny morning, Silas was looking for a bag of apples in the backyard."
  -> " She saw a big bag of apples and sh[e]"
  <- pronoun doesn't match the character

"On a rainy Tuesday, a gentle horse saw a wooden box lying in the attic."
  -> " The box was a big dog! The dog was"
  <- grammatically fine, but the box turns into a dog

"Lily was playing in the attic when something strange happened."
  -> " The hill was a little boy who was "
  <- fluent, but nothing in the scene supports it

"Without any warning, Tom was walking on the farm road and started to sing."
  -> "\n\n\"Wow, look at all the warning!\" T"
  <- picks up a word from the prompt and turns it into nonsense

A pronoun/gender mismatch a sentence or clause after a character is introduced is the single most common failure inside this category.

Real examples that landed in "grammar breaks down" (all five, unedited):

"During the summer, Nova was cleaning up in the treehouse after a long day."
  -> " After the summer were ready to sit"         <- subject/verb agreement collapses

"Not far from home, Tom had never seen a wooden spoon before."
  -> " He had far from his house, and he "        <- broken verb phrase

"Just before sunset, Grace was making a plan to find a wooden box."
  -> " All of his living room were not cl"        <- agreement collapses, then trails off

"Just before sunset, a small bird was walking in the kitchen and started to sing."
  -> " Just as Just before sunset, Just b"        <- repeats the prompt's opening phrase

"Without any warning, a gentle horse saw a paper boat lying by the lake."
  -> " The paper leaves the paper and the"        <- word-salad repetition

---

Pronoun agreement, measured automatically

A generation can be fluent and still give a named character the wrong pronoun. To measure this directly, we took every prompt in the 1,000-prompt set that has a single named character with a clearly gendered name, and checked the first he/she pronoun in the completion against that name. 522 of the 1,000 completions contained such a pronoun.

	Pronoun matches the name
All names	77.2% (403 / 522)
Male names	92.7% (241 / 260)
Female names	60.4% (142 / 235)

The female-name number hides a sharp split by name:

Female names	Pronoun matches
Hazel, Nora, Willow, Poppy, June, Zoe	8.5% (7 / 82) — the model almost always writes "he"
Anna, Ellie, Lily, Ivy, Luna, Mia, Sadie, Daisy, Rosie, Grace, Aria, Ruby	88.2% (135 / 153)

So the model has learned the gender of common names well, and defaults to "he" for less common female names. Each name appears roughly 15–30 times in the set. (Willow and June can also be unisex; even without them the pattern holds — Hazel 1/17, Nora 1/14, Poppy 1/12, Zoe 2/15.) Ambiguous names (Wren) and prompts with two named characters are excluded, because a pronoun can legitimately refer to either character.

---

Real-world benchmark results

The coherence grading above is our own methodology. To see how the model does on external benchmarks, we ran it on two established suites. Neither involves code or math. Sample sizes are noted, and the sampling error is not small — read differences of a few points as noise.

BLiMP (Benchmark of Linguistic Minimal Pairs)

BLiMP scores whether a model assigns higher likelihood to a grammatical sentence than to a minimally-different ungrammatical one, across 67 phenomena. We ran all 67 paradigms with 30 randomly sampled pairs each (2,010 pairs, fixed seed), and list the baselines from the BLiMP paper for reference.

Model	Params	BLiMP avg.
5-gram	—	61.2%
NoTokenLM-Gen-4.5 (same 2,010 pairs)	20M	65.4%
NoTokenLM-Gen-3.6	13.9M	68.9% (95% CI 66.8–70.9)
Transformer-XL	—	69.6%
LSTM	—	69.8%
GPT-2-large	774M	83.0%

Gen-3.6 lands in the LSTM / Transformer-XL range. Against Gen-4.5 on the same pairs it is ahead on 34 paradigms, behind on 23, and tied on 10 — a gap that is within sampling error overall. (Gen-4.5's own card reports 66.8% on a different sample of pairs.)

Strengths and weaknesses follow the same shape as the rest of this card: 19 of 67 paradigms score 90% or higher, mostly local agreement and licensing (determiner_noun_agreement_2, principle_A_case_1, sentential_negation_npi_licensor_present, wh_vs_that_no_gap_long_distance all 100%). Four paradigms score 20% or below, all requiring long-range or structurally complex dependencies: wh_vs_that_with_gap_long_distance (3.3%), coordinate_structure_constraint_complex_left_branch (10.0%), existential_there_quantifiers_2 (10.0%), matrix_question_npi_licensor_present (16.7%).

LAMBADA (long-context final-word prediction)

LAMBADA gives a paragraph and asks for the final word. We scored 500 randomly sampled passages from the LAMBADA test split (fixed seed) with greedy decoding: the first word of the continuation, punctuation stripped, lowercase, exact match.

Model	Params	LAMBADA accuracy
NoTokenLM-Gen-3.6	13.9M	11.4% (57 / 500, 95% CI 8.9–14.5)
NoTokenLM-Gen-4.5 (same 500 passages, same scoring)	20M	8.0% (40 / 500)
GPT-2-small	124M	~46%
GPT-2-large	774M	~59%
GPT-3 (zero-shot)	175B	76.2%

On identical passages Gen-3.6 gets 29 right that Gen-4.5 misses and misses 12 that Gen-4.5 gets right (sign test p ≈ 0.01). The likely reason is data, not architecture: LAMBADA passages are fiction, and Gen-3.6 saw web and encyclopedic text as well as stories. (Gen-4.5's card reports 12.0% on a 100-passage sample under its own scoring; the two numbers are not directly comparable.) In absolute terms this is still very far below even GPT-2-small.

HellaSwag was not evaluated for this checkpoint.

Bits per byte on held-out text

A model-agnostic measure of how well it predicts raw text (first 1,000 bytes of each passage; lower is better; not comparable across tokenizers or other corpora):

Text	Bits per byte
200 fiction passages (LAMBADA validation)	1.891
201 short grammatical sentences (BLiMP)	1.967

---

What it knows, and what it doesn't

We probed the model directly with short factual and pattern prompts (temperature 0.5, top-k 40, 4 seeds each):

Probe	Result
Capital cities ("The capital of France is", French and English phrasings, 8 prompts × 4 seeds)	0 / 32
Arithmetic (10 prompts × 4 seeds)	2 / 40 — both are coincidences of a leading digit; the model writes a number and keeps going
Mixed facts, opposites, sequences, code, context reuse (25 prompts × 4 seeds)	23 / 100 — and 12 of those hits come from the common-idiom prompts ("Once upon a", "Thank you very", "Good morning! How are")

The model does capture frequent surface patterns. Measured directly as next-item probability (no sampling):

Prompt	Target	Probability
1, 2, 3, 4,	5	35.6%
2, 4, 6, 8,	10	20.7%
10, 20, 30, 40,	50	17.6%
5, 10, 15, 20,	25	3.5%
Monday, Tuesday, Wednesday,	Thursday	0.0%

Common counting patterns are picked up; the rule behind them is not (compare 2, 4, 6, 8 with 5, 10, 15, 20). Anything that needs a fact, a rule, or a step of reasoning is out of reach.

---

Language behavior

The training mix targets 58% English and 42% French by byte share, and the data pipeline has no language-ID filter. What that looks like in practice, from informal testing (a handful of prompts, not a benchmark):

• English prompts stay in English and in a narrative or expository register.
• French prompts produce French, but in a news/encyclopedic register rather than a story register, with fluent-looking sentences that don't hold together. French is not usable for story generation at this checkpoint.
• Prompts in other languages (Spanish, Portuguese, Italian, German, Dutch, Turkish, Hausa) are not reliably continued in the prompt's language: the model drifts into English, French, Italian-like text, or falls into repetition loops. Earlier in training the same prompts could produce fluent Hausa, so non-target-language text is clearly present in the training data and the model picks it up.
• Broken byte sequences ( , a UTF-8 double-encoding artifact) appear in 2 of the 1,000 evaluation outputs, which suggests some source text contains encoding errors.

---

What it's actually good at

• Well-formed English at the sentence level. Grammar is largely intact, and local syntactic agreement is strong (19 of 67 BLiMP paradigms at 90%+).
• Staying in a narrative register on story-style prompts. Given a concrete character and action, continuations are usually on-topic for a sentence or two.
• Predicting general text. At 1.89 bits per byte on held-out fiction, it models everyday English prose far better than a narrow story-only model would.
• Occasional multi-sentence stretches. In informal 240-byte generations it stays on topic for a couple of sentences before repeating or drifting; this has not been measured at scale, so treat it as a directional observation.

What it's not good at, and why

• No usable world knowledge. Despite seeing Wikipedia and educational web text, ~0.85 GB of training text at 13.9M parameters is far too little to store facts. It writes plausible-sounding text that is wrong.
• Pronoun and gender consistency. The most common failure, and very name-dependent: reliable for common names, close to random for less common female names (see above).
• Long-range structure. LAMBADA at 11.4% and near-chance BLiMP paradigms on long-distance dependencies point at the same gap.
• Semantic drift. Fluent continuations that contradict the scene: objects turning into other objects, inanimate things acting as people, characters addressing themselves.
• Register drift. Occasional jumps out of story mode into encyclopedic or expository text.
• Non-English output. See above; French in particular is not reliable.
• Rare malformed output. Repeating the prompt's opening phrase, broken verb inflection, or garbled bytes.

---

How to actually run this thing

Recommended: temperature 0.5, top-k 40. The 1,000-prompt evaluation and the probes above were produced at these settings (BLiMP is likelihood-based and LAMBADA is greedy). Greedy decoding on open-ended prompts tends to fall into repetition loops; much higher temperatures increase drift and malformed output.

---

Architecture details

Parameters	~13.87M
Layers	12
d_model	320
Attention heads	5 (head dim 64)
Feedforward dim	768 (SwiGLU)
Vocabulary	256 (raw bytes, no tokenizer)
Context length	1024 bytes
Position encoding	RoPE (base 10000)
Normalization	RMSNorm
Attention extras	QK-norm, value residual, per-head sigmoid output gate
Input stage	Causal depthwise convolution (kernel 4) over the byte embeddings
Output layer	Weight-tied to the input embedding
Training	Byte-level next-token prediction, from scratch (pretraining only — no instruction tuning)

Training data

Target byte shares (the loader normalizes and balances toward these):

Language	Source	Share
English (58%)	FineWeb-Edu (sample-100BT)	33%
	TinyStories	10%
	Wikipedia (20231101.en)	6%
	Cosmopedia — stories	5%
	Cosmopedia — Khan Academy-style text	4%
French (42%)	FineWeb2-HQ (fra_Latn)	24%
	Wikipedia (20231101.fr)	8%
	Wikibooks (Luciole training set)	6%
	Vikidia (Luciole training set)	4%

Documents are kept if they are 200–20,000 bytes, at least half alphabetic characters, and under a non-ASCII byte cap (8% for English sources, 25% for French). There is no language-ID filter, no dedicated code or math corpus, and no instruction-following data.

---

Part of the NoTokenLM family — small models, built and evaluated honestly.

---

License

This project is licensed under the Apache License 2.0. Please refer to the LICENSE file for the full text of the license.