sumitv461/si-lm-code

SI-LM v1 — 3CS AI Innovation Superintelligence Language Model

A from-scratch decoder-only Transformer, trained on Kaggle in two autonomous sessions (10h + 8h), grown incrementally across versions, deployed publicly on HuggingFace Spaces. Designed against the 25 training instructions in superintelligent_model_training_instructions.txt and the 26 build-discipline instructions in 26 instructions_for_appointed engineer.txt.

Growth ladder

Stage	Active params	Purpose
v0.1	0.10B	Sanity: end-to-end pipeline works on laptop
v0.3	0.30B	Pilot: real pretraining at small scale
v0.7	0.70B	Public-min: first HF release, gather feedback
v1.0	1.5B	Public: full Kaggle budget deployed
v2.0	3.0B	Frontier-mini
v3.0	7.0B	Frontier

Growth is done via width+depth expansion of the same checkpoint (no re-training from scratch). See train/growth.py and the master plan.

Three role variants

Variant	Role
SI-LM-Reason	Main public model: reasoning, math, code, tool-use
SI-LM-Ground	Grounded conversation, retrieval, "I don't know" calibration
SI-LM-Orchestrator	Routes queries to Reason / Ground / tools

Layout

new_language_model_SI/
├── data_engine/      dataset download, clean, SFT curation, token packing
├── train/
│   ├── model/        transformer, RoPE, GQA, SwiGLU, configs
│   ├── data/         dataset, packing
│   ├── optim/        AdamW, cosine schedule
│   ├── utils/        checkpoint, resume, logging
│   ├── growth.py     width+depth expansion
│   ├── rewards.py    GRPO reward functions
│   ├── pretrain.py   Stage A: causal LM pretrain
│   ├── sft.py        Stage B: supervised fine-tune
│   ├── rl_grpo.py    Stage C: GRPO with verifiable rewards
│   ├── eval.py       eval entry point
│   └── smoke_v01.py  v0.1 smoke test
├── kaggle/           session_a.ipynb, session_b.ipynb
├── space/            Gradio public chat
├── eval/             MMLU / GSM8K / HumanEval / IFEval runners
├── configs/          JSON configs for every variant + growth stage
├── reports/          training logs, eval reports, time audit
├── tests/            CPU unit tests
├── data_set_conver/  raw conversation logs (input to data engine)
├── .env.example      template for secrets
└── .gitignore        keeps .env, .kaggle/, weights out of git

Reproduce

1. cp .env.example .env && nano .env (add real tokens, never commit)
2. pip install -r requirements.txt
3. python -m data_engine.train_tokenizer (one-time, ~30 min)
4. python -m data_engine.build_corpus (downloads + cleans, idempotent)
5. python -m train.smoke_v01 (validates the whole pipeline at 1/1000th scale)
6. Upload kaggle/session_a.ipynb to Kaggle, run
7. Upload kaggle/session_b.ipynb to Kaggle, run
8. huggingface-cli push the Space

Honest limits

• 0.7B–1.5B trained on 40B tokens will not beat GPT-4 on broad knowledge.
• It can beat much-larger models on: Hinglish chat, tool-use task completion, grounded Q&A, math/code with verifiable answers, calibration of refusal. Those are the benchmarks we optimize for and ship.
• No live world sense during training; tool use is trained SFT-style on static traces; real-world grounding comes from the public Space connecting to the live web.
• Single-account Kaggle = single point of failure. Mitigated by fail-safe + HF-pushed intermediate state every 30 minutes.