Flat byte baseline (pretrained)

Flat 22-layer byte-level transformer, no segmentation and no hierarchy. Every byte costs one forward pass.

From Dynamic Multi-Byte Prediction With Hierarchical Language Models. Code: skai-research/lca-multibyte.


Parameters	374M
Vocabulary	261 (256 bytes + `<pad>`/`</s>`/`<unk>` + `<en>`, `<eot>`)
Context	4096 bytes
`model_config`	`[22, (0,), 0, 0]`
`attn_type`	`None`
Precision	fp32

Results

Metric	Value
Byte-level BPC (validation)	0.888
Training steps	48,186

Usage

This is not a transformers architecture — load it with the code from the paper repo:

git clone https://github.com/skai-research/lca-multibyte && cd lca-multibyte
uv sync && export PYTHONPATH=$(pwd)

from huggingface_hub import snapshot_download
from src.eval.model_loader import load_fxt_model

path = snapshot_download("skai-research/bytes-base")
model, tokenizer, config = load_fxt_model(path, device="cuda")

Or generate directly:

python src/eval/generate.py --model_path $(python -c \
  "from huggingface_hub import snapshot_download; print(snapshot_download('skai-research/bytes-base'))") \
  --prompt "The capital of France is" --mode cached

--show_tokenization prints the learned segment boundaries.

Training

Pretrained on FineWeb-Edu sample-100BT. Config: configs/train/modern_fxt_baseline_btyes_256_scale_bp_dual.yaml.

Citation

@article{owodunni2026lca,
  title   = {Dynamic Multi-Byte Prediction With Hierarchical Language Models},
  author  = {Owodunni, Abraham Toluwase and Okocha, Chibuzor and Grant, Christan
             and Limisiewicz, Tomasz and Kumar, Sachin},
  year    = {2026},
  journal = {arXiv preprint arXiv:2608.15454},
  url     = {https://arxiv.org/abs/2608.15454}
}