faridahmed-devhub/FaridLLM-Tiny

FaridLLM

A modular large language model research project by Farid Ahmed.

This is a tiny experimental model built to validate the full training pipeline (dataset -> tokenizer -> dataloader -> forward -> loss -> backward -> optimizer -> scheduler -> checkpoint -> resume -> evaluation -> generation). It does not represent frontier-level performance.

Model Description

FaridLLM is a modular transformer with:

• Grouped Query Attention (GQA)
• Rotary Position Embeddings (RoPE)
• SwiGLU feed-forward layers
• RMSNorm
• Optional Sparse Mixture-of-Experts (off in this model)

This exported model is dense, trained on synthetic numeric sequences ('toy arith numeric sequences, pattern=arith, numeric vocab 32'), trained for 300 steps.

Architecture (from the actual checkpoint)

Parameter	Value
Parameters	2,492,672
Hidden size	256
Layers	4
Attention heads	4
KV heads	2
Head dim	64
Intermediate size	512
FFN type	swiglu
Attention type	gqa
Context length	512
RoPE theta	10000.0
RMS norm eps	1e-06
Vocab size	256
Tie word embeddings	False
MoE	no
Dtype	float32

Tokenizer

ByteLevel BPE (Hugging Face tokenizers backend). Vocabulary size and contents match the checkpoint used during training. Special tokens: <pad>, <unk>, <bos>, <eos>, <mask>.

Intended Use

• Research and experimentation
• Validating training/inference pipelines
• Educational demonstrations of a small transformer

Limitations

• Tiny model; results do not represent frontier-level performance.
• Experimental: unvalidated outside the toy benchmark below.
• Benchmark coverage is limited (single tiny perplexity estimate, initially random, then trained for a few hundred steps on synthetic data).
• Outputs may be incorrect or nonsensical.
• Not a production system. Do not treat as such without further validation.

Training

Field	Value
Dataset	toy arithmetic-progression numeric sequences (values 0..31)
Steps	300
Seed	2024
Batch size	8
Optimizer	AdamW (lr 0.0003)
Hardware	CPU (14 threads)
Dtype	float32

Evaluation (measured, from the tiny experiment)

• Validation perplexity on 64 held-out sequences: 3.0788 (baseline at init: 275.6866)
• Train perplexity: 1.5325
• Greedy generation exact-match on 6 held-out prompts (16/16 tokens): 0.8333
• Overfit sanity passed: final loss 0.0207 on the 8-sequence fixed corpus; exact match 100%
• Checkpoint/resume equivalence passed (max loss diff 0.0 vs continuous run)

License

Apache-2.0