justinsimpsad/la-vel

LA VEL

LA VEL is a from-scratch, open decoder-only transformer built to be grown in public. This repo carries the language core; sibling repos and Kaggle kernels grow vision, music, 3D-mesh, and market-analysis modules.

Language core (this repo)

Architecture	pre-LN GPT-style, SDPA causal attention, GELU MLP, tied embeddings
Position encoding	learned (seed) or rope (long-context) — selectable in config.json
Seed model	6L / 6H / d384 / ctx512 / 16k ByteLevel-BPE ≈ 17.1M params
Scale-up target	12L / 12H / d768 / ctx2048 RoPE ≈ 69.3M params (Kaggle growth runs)
Formats	model.safetensors + la-vel.gguf (llama.cpp-compatible, arch=gpt2)
Training data	TinyStories + Gutenberg mix (~16M tokens per seed pass)
License	MIT

Files

• model.py — standalone implementation (no transformers dependency)
• train.py — pretraining; --resume-from local dir or HF repo id, optional --push-to-hf
• generate.py — sampling CLI
• gguf_export.py — safetensors -> GGUF converter (embeds BPE vocab + merges)
• kaggle_la_vel_growth_kernel.py — GPU kernel that resumes from this repo, optionally scales up (LAVEL_SCALE=large), trains, and pushes new weights back
• KAGGLE_SETUP.md — how to run/grow on Kaggle

Usage (PyTorch)

import torch
from model import LaVelForCausalLM
from tokenizers import Tokenizer

model = LaVelForCausalLM.from_pretrained("justinsimpsad/la-vel")
tok = Tokenizer.from_file("tokenizer.json")
ids = tok.encode("Once upon a time").ids
out = model.generate(torch.tensor([[1] + ids]), max_new_tokens=120)
print(tok.decode(out[0].tolist()[1:]))

Usage (llama.cpp)

# la-vel.gguf embeds its own tokenizer - just run it:
llama-cli -m la-vel.gguf -p "Once upon a time" -n 200

Growth fleet (one kernel per domain)

Module	Kernel	Repo
Text core	justinsimpsad/la-vel-growth	this repo
Vision (ViT)	justinvibesrise/la-vel-vision	la-vel-vision
Music LM	justinvibesrise/la-vel-music	la-vel-music
3D meshes	justinvibesrise/la-vel-mesh3d	la-vel-mesh3d
Markets (research)	justinsimpsad/la-vel-markets	la-vel-markets

Each kernel resumes the latest checkpoint from its repo, trains on a free Kaggle GPU, and commits updated weights back. Run repeatedly to grow.

Honest status

The text core is an early seed trained briefly (val ppl ~50-60 and dropping); it produces toy-level English. The vision/music/mesh/markets modules are capability seeds trained on synthetic data. Roadmap stages land here as they are built: real-image pretraining, audio codecs, face-sequence meshes, walk-forward market validation.