urls-250m โ a URL language model for lossless compression
246,087,680 parameters: 12 layers ร 1280 wide, 20 heads, SwiGLU MLP,
tied 8,192-entry byte-level BPE vocabulary (structural pre-split, hosts stored TLD-first),
learned absolute positions reset at every <eos>, so each URL is its own context.
Shipped as a 124.8 MiB int4-g64 artifact (.nurl v3: 34 bytes per 64 weights, f16 scales)
that the browser codec streams with exact integer dot products, making encode and decode bit-identical.
Objective: bits per character on held-out URLs (the compression ratio), decoded on the client in under a second.
Training
| data | ks46/urls-tokenized shards 0โ39, 19.9B tokens โ 561M URLs, one pass, seeded order |
| optimizer | AdamW ฮฒ(0.9, 0.95), weight decay 0.01, peak LR 0.0006, warmup 1000, linear decay over the last 10% |
| batch | 64 windows ร 512 tokens ร 8 GPUs = 262,144 tokens per iteration, 76,000 iterations |
| tail | 7,600 quantisation-aware iterations on the exact int4-g64 grid at LR 6e-05 |
| hardware | 8ร RTX 5090 (PCIe, no NVLink), bf16 autocast, ~680k tokens/s |
| code | linklet training/ at git unknown |
Checkpoints
checkpoints/ckpt-i<iter>.pt are immutable snapshots every 4,000 iterations (1.05B tokens), stable-phase
weights with optimizer state, loadable by training/train.py (run.init=) for decay or fine-tuning.
ckpt-fp.pt is the end of the full-precision phase; ckpt.pt is the final model after the QAT tail.
Eval is bits/char on a fixed 2,000-URL held-out slice (shard 512, even-stride sample).
| file | tokens | eval bits/char |
|---|---|---|
| ckpt-i004000.pt | 1.05B | nan |
| ckpt-i020000.pt | 5.24B | nan |
| ckpt-i040000.pt | 10.49B | nan |
| ckpt-i060000.pt | 15.73B | nan |
| ckpt-i068000.pt | 17.83B | nan |
| ckpt-i072000.pt | 18.87B | nan |
| ckpt-i080000.pt | 20.97B | nan |
Final evaluation
fp32: 1.2629 bits/char over 99926 URLs (74 skipped)
kernel_sim(int4): 1.26372 bits/char
kernel: 1.2952 bits/char (rust/nurlcheck) over 500 URLs, artifact 124.8 MiB
model/linklet-target-base.nurl is the packed artifact; model/tokenizer/ the tokenizer.
Every number above is copied from manifest.json, the run's provenance record.