snapgate-surge-v2

A bilingual language model (Indonesian + English) trained from scratch — no base model was used at any stage of training.

Model Specifications

Specification	Detail
Parameters	~10 Million
Architecture	GPT-style Transformer (CausalLM)
Vocab Size	8,000
Hidden Dim	384
Layers	8
Attention Heads	8
Context Length	512 tokens
Languages	Bahasa Indonesia 🇮🇩, English 🇺🇸
License	MIT

Training Progress

This model was trained across multiple sessions due to interruptions. The chart below reflects the continued pre-training — loss resumes from a checkpoint and continues to converge.

Checkpoint	Train Loss	Val Loss	Notes
Step 0	~8.5	~8.5	Training start / resume
Step 5,000	~3.6	~3.6	Rapid descent phase
Step 10,000	~3.2	~3.2	Stabilizing
Step 15,000	~3.1	~3.1	Continued convergence
Step 20,000	~3.05	~3.0	Near plateau
Step 25,000	~3.0	~2.95	Val loss slightly below train
Step 30,000	~3.0	~2.95	Final checkpoint

Train and validation loss closely track each other throughout training, indicating no overfitting. The model converged smoothly from ~8.5 → ~3.0 over 30,000 steps across continued pre-training sessions.

Notes

This is a pure base model — it has not been fine-tuned for instruction following or conversational tasks. It serves as the foundation for further supervised fine-tuning (SFT) within the Snapgate AI ecosystem.

---

Developed by the Snapgate AI team | snapgate.tech