jfrick/hnet-720m

H-Net 720M (byte-level, 23.8 GB): a research checkpoint

A byte-level language model with one stage of dynamic chunking (H-Net) wrapped around a 16-layer Kimi-Delta-Attention / full-attention hybrid trunk. It was trained from random initialization on 23.8 GB of UTF-8 text to compare it with the same trunk trained on GLM-4.7 tokens over the same data and compute. It is a base model (no instruction tuning) and a small research artifact. Its text is fluent but often wrong, and it repeats itself.

Result in one line: after 23.8 GB at matched FLOPs per byte, this H-Net is 2.7–4.5% worse in held-out bits per byte than the GLM-tokenized model. Cartesia's improved H-Net (October 2026) is not released; this is the published 2025 design.

Architecture

Input	UTF-8 bytes 0–255, document end = 256 (vocabulary 257)
Encoder	4 blocks at width 1,024: KDA mixer (8 heads x 128) + SwiGLU (2,816)
Routing	p_t = (1 - cos(W_q x_(t-1), W_k x_t)) / 2; a byte with p > 0.5 starts a chunk; the first byte of each document always does
Main network	16 layers at width 1,536, 4 x [KDA, KDA, KDA, full attention]; full attention: 12 query / 4 KV heads, head dim 128, QK-norm, RoPE (base 1e6), attention sink, sigmoid output gate; SwiGLU 4,224
Dechunk	EMA over chunk outputs z_t = p_t h_t + (1 - p_t) z_(t-1), repeated over each chunk's bytes; times a straight-through estimate of p, plus a zero-initialized projection of the encoder output
Decoder	4 blocks like the encoder, then an untied 1,024 x 257 output layer
Training context	16,384 bytes; every stage restarts at each document end
Parameters	580,052,736 (main network 469.2M; encoder + decoder 107.1M; routing, residual projection, byte embedding and output layer 3.7M)

Measured compression after training: 4.38 bytes per chunk on ClimbMix, 4.13 on code and 3.88 on math. The GLM-4.7 tokenizer gives 4.83, 3.99 and 3.62 bytes per token on the same text.

Training

• Data: 70% ClimbMix, 20% permissively licensed code from github-code-clean and 10% FineMath, by tokens. Documents sharing a 13-gram with common benchmark test sets were removed. The model saw the exact documents, in the same order, that the GLM-token model saw in its first 5,000 steps, decoded back to bytes.
• Budget: 5,043 steps x 288 x 16,384 bytes = 23,795,859,456 bytes (about 5.2B GLM tokens). The learning rate was still constant at the end (no cooldown).
• Optimizer: Muon (LR 0.02) for hidden matrices; AdamW (LR 3e-4, betas 0.9/0.95, weight decay 0.1) for embeddings, output layer, norms and the routing projections. Warmup over 2.27 GB; ratio-loss weight 0.3 with target 4.5 bytes per chunk; z-loss 1e-4; gradient clipping 1.0; bf16 autocast.
• Compute: 8 x H100 for about 8.7 hours, at 850–890K bytes/s and 14.8% MFU. Training FLOPs per byte (1.29G) match the GLM-token model (1.31G) within 2%.

Results

Held-out bits per byte on the same documents. Both models were at a constant learning rate; H-Net steps carry 1% more bytes than GLM steps.

Step	Bytes	GLM ClimbMix	H-Net ClimbMix	GLM code	H-Net code	GLM math	H-Net math
500	2.4 GB	1.089	1.284	0.802	0.955	1.108	1.368
1,000	4.7 GB	0.974	1.078	0.635	0.669	0.970	1.080
2,000	9.4 GB	0.926	0.998	0.578	0.614	0.909	0.983
3,000	14.2 GB	0.902	0.951	0.532	0.540	0.876	0.919
4,000	18.9 GB	0.894	0.937	0.528	0.559	0.867	0.917
5,000	23.6 GB	0.885	0.922	0.510	0.527	0.853	0.892
5,043 (this model)	23.8 GB		0.920		0.524		0.891

• The gap fell from about 20% to about 4% by 14 GB and then barely moved.
• Per byte, the H-Net took about 2.2x the GLM model's wall-clock time, because its byte-level layers ran at about 10% MFU.
• With Muon on the routing projections, or the paper's ratio-loss weight of 0.03, the chunk rate collapsed and then overshot to about 2 bytes per chunk, and training ran out of memory. AdamW routing with weight 0.3 kept it near 4.3.

Samples

The prompts are the 20-prompt fluency set of the repository's sampling toolkit. Decoding was greedy, plus temperature 0.8 / top-p 0.95 with two seeds. Generations were about 320 bytes from this model and 72 tokens (about 270 characters) from the GLM-token model at its 5.0B-token checkpoint (22.7 GB of the same data). Below is the first sampled seed, cut to about 200 characters; nothing was cherry-picked.

Prompt	GLM-token hybrid (5.0B tokens)	This H-Net (23.8 GB)
The capital of France is	the capital of France. The capital of France is the capital of France. French and English are the two most-used languages in France…	Paris, with a population of over 30 million people. The country's culture is diverse, being characterized by religious origins, famous figures…
Photosynthesis is the process by which	plants convert sunlight, carbon dioxide, water and carbon dioxide into energy. Photosynthesis takes place in plants, plants and animals…	plants absorb solar energy to convert carbon dioxide into sugars and water. Plants are found in a variety of places in the world…
Once upon a time,	there was a little bunny named Alice who loved to sew. She loved exploring the world around her…	there was a little creature named Benny who lived in a big place called Hangzhou…
How to make a cup of tea:\n1.	Boil water. Put the tea pot on a stove, when it will boil slowly. 2. With the water, mix the tea bags with tea…	Add a flavorful ingredient and soak it in cold water for a few minutes to soften it. 2. Strain and discard the waste contents…
The French Revolution began in	1532 as the struggle for France to become a country…	1930. After the French Revolution, France adopted a policy of revolutionary technological advancements…
def factorial(n):\n	return n / def prime(n): return n	if n == 0: return 0 else: return n * 10 // n * 10 * 10 …
Question: A train travels 60 miles in 2 hours. What is its average speed?\nAnswer:	Speed of train= 120 mph. Average speed of train= 60 mph… Time = 120 ÷ 2. Time = 48 hours.	A speed of 60 miles per hour is the average speed of the train…

Both models write fluent, loosely factual text and get arithmetic wrong. Repetition over the first 128 characters (share of repeated 16-character windows) is similar: greedy 19.1% (GLM) vs 18.5% (H-Net), sampled 1.9% vs 2.2%. Over 256 characters the H-Net repeats more under greedy decoding (40.8% vs 26.0%). Because the H-Net reads raw bytes, a prompt that ends in spaces or mid-word needs no special handling; the GLM-token model sometimes stumbles on those token boundaries (def factorial(n):\n followed by a space-prefixed return).

Files

• tokens-023795859456/model.safetensors: bf16 weights.
• tokens-023795859456/config.json: HNetConfig fields (main holds the trunk's HybridConfig fields).
• tokens-023795859456/progress.json: step, bytes and training seconds.
• metrics.jsonl: the full training log, including held-out evaluations and samples.

Use

The model code is the model_hybrid package in koogle/lm-exploration (hnet.py, plus model.py, kda.py and their imports). The CUDA path needs fla-core==0.6.0 and torch 2.14 with FlexAttention; on CPU the model runs pure-PyTorch reference attention and KDA.

import json, torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from model_hybrid import hnet

folder = "tokens-023795859456"
config = json.load(open(hf_hub_download("jfrick/hnet-720m", f"{folder}/config.json")))
fields = {k: v for k, v in config.items() if k not in ("architecture", "tokenizer")}
lm = hnet.HNetLM(hnet.HNetConfig(**fields))
weights = load_file(hf_hub_download("jfrick/hnet-720m", f"{folder}/model.safetensors"))
lm.load_state_dict({k: v.float() for k, v in weights.items()})
lm = lm.cuda().eval()

ids = list("The history of the printing press".encode())
with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16):
    for _ in range(200):
        ids.append(int(lm(torch.tensor([ids], device="cuda"))[0, -1].argmax()))
print(hnet.ByteTokenizer().decode(ids))

Limitations

• Not a fair final comparison. Both models are mid-training checkpoints without a learning-rate cooldown. The GLM-token model gained 11–21% in bits per byte from its cooldown at 40B tokens.
• Text quality. Generations are fluent but factually unreliable and repetitive, and the model has not been evaluated on downstream benchmarks.
• Decoding speed. Decoding recomputes the whole sequence for every byte; there is no incremental (cached) decoding path.
• Data. English-centric web text, code and math; no safety tuning.