XAbys-tiny β pipeline-proof checkpoint (NOT the flagship)
Honest summary first: this is a 1.45M-parameter demo model trained from random initialization on 9,226 tokens of sample text. It proves the XAbys training pipeline works end to end (tokenizer β data β training β checkpoint β eval β generate). It cannot reason, code, or answer questions. Its generations are gibberish. The real flagship XAbys will come from fine-tuning an open-weight base model (Qwen 3 / Llama 4 / Nemotron) on Colab with Unsloth β that work has not happened yet. Do not mistake this checkpoint for that model.
Model details
- Architecture: decoder-only Transformer, written from scratch (RMSNorm, RoPE, SwiGLU, causal attention, tied embeddings)
- Config: 2 layers, 4 heads, d_model 128, vocab 8000, block size 512
- Parameters: 1,450,624 (train.py count)
- Tokenizer: ByteLevel BPE, trained on the sample corpus (actual vocab 1,788)
- Training: from random initialization, AdamW, cosine LR, 3 epochs, 51 steps, batch 4, block 128, CPU only, ~17.5 seconds
- Data: 573 raw lines β 550 kept, 9,226 tokens (train 8,765 / val 461) β original sample text (English, Roman Hinglish, Python, maths)
Measured results (14 Nov 2026, CPU)
- Train loss: 8.99 (uniform over 8000 vocab) β 7.79 at step 50
- Eval loss (best checkpoint, step 40): 7.8109, perplexity 2467.39
- Held-out perplexity (sample_math.txt): 2814.04
- Generation exact-match: 0 / 4 correct (0.0%)
- Sample generation, prompt "Dukaan subah khulti hai" β "TheTheTheTheThe."
Files
config.jsonβ model configtokenizer.jsonβ BPE tokenizerxabys-tiny-model.ptβ model weights only (5.8 MB, state dict)xabys-tiny-best.ptβ full training checkpoint (17.4 MB: model + optimizer + scheduler + config + step + best loss)
How to load
import json
import torch
from xabys.model import XAbysConfig, XAbysForCausalLM
from xabys.tokenizer import load_tokenizer
cfg = XAbysConfig(**json.load(open("config.json")))
model = XAbysForCausalLM(cfg)
model.load_state_dict(torch.load("xabys-tiny-model.pt", map_location="cpu"))
model.eval()
tok = load_tokenizer("tokenizer.json")
Full source, training scripts, tests and docs: https://github.com/hexray1 β see the XAbys project (workspace/xabys).
Limitations
- 1.5M params on 9k tokens cannot learn language, reasoning or code.
- No vision, audio, video, tools or agent capability in this checkpoint.
- Generation repeats tokens ("AAAA.", "TheTheThe...").
- The 7B config in the source is config-only and was never trained here.
- Downloads last month
- 37
Inference Providers NEW
This model isn't deployed by any Inference Provider. π Ask for provider support