byte-llm-pretrain-v1

A small Qwen3-architecture causal LM (31.6M params) trained from scratch with a byte-level tokenizer — 256 raw UTF-8 byte tokens plus <|pad|>/<|bos|>/<|eos|>/[S1]/[S2]/3 reserved slots (264 total vocab). Because any Unicode string decomposes losslessly into UTF-8 bytes, this tokenizer covers any language/script with no fitting step and no <unk>. [S1]/[S2] are greedily matched as single tokens whenever their literal string appears in the input text.

Trained on the targets column of humairmunirawn/xp3x-urdu.

See training_config.json in this repo for the full architecture and optimisation config, and the runs/ folder for TensorBoard logs.

Usage

from transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained("zuhri025/byte-llm-pretrain-v1")
# tokenizer: see byte_tokenizer.py in this repo, or infer.py for a full
# generation script with sampling, streaming, and an interactive mode.