Lad1-0.2B-Instruct

Introduction 🥀

Lad1-0.2B-Instruct is a compact 235M parameter instruction-tuned causal language model developed by lloid-labs.

Lad1 uses a custom Transformer architecture with Grouped Query Attention (GQA), RMSNorm, SiLU feed-forward layers, and sinusoidal positional embeddings.

The model was first pretrained from scratch on approximately 1.2B tokens from FineWeb-Edu, followed by full-parameter supervised fine-tuning on a filtered version of SmolTalk.

The model is designed for lightweight instruction following, conversational generation, experimentation, and research with small language models.

Model Details

• Model type: Causal Language Model (custom architecture)
• Parameters: ~235M
• Layers: 12
• Hidden size: 1024
• Attention heads: 16
• Key/Value heads: 4
• Attention: Grouped Query Attention
• Normalization: RMSNorm
• Activation: SiLU
• MLP ratio: 4
• Position encoding: Sinusoidal
• Vocabulary: 50,257 tokens
• Tokenizer: GPT-2 tokenizer
• Context length: 512 tokens

Training

Pretraining

The base model was trained from scratch on:

• Dataset: HuggingFaceFW/fineweb-edu
• Configuration: sample-10BT
• Tokens: ~1.2B
• Steps: 36,000
• Learning rate: 3e-4
• Batch size: 8
• Gradient accumulation: 8
• Effective batch size: 64
• Sequence length: 512
• Precision: FP16 mixed precision
• Gradient clipping: 1.0

The dataset was streamed and shuffled with a buffer size of 10,000 and seed 42.

Instruction Tuning

Lad1-0.2B-Instruct was subsequently fully fine-tuned using supervised fine-tuning (SFT) on SmolTalk.

The following subsets were intentionally excluded, since the base model has no pretraining exposure to these domains:

• apigen-80k — function calling
• metamathqa-50k — mathematics
• numina-cot-100k — mathematics
• self-oss-instruct — code

The remaining SmolTalk data was used to improve general instruction following and conversational behavior.

Usage

Lad1 uses a custom architecture, so it's loaded with plain PyTorch rather than AutoModelForCausalLM:

import math
import torch as t
import torch.nn as nn
import torch.nn.functional as F
from transformers import AutoTokenizer
from huggingface_hub import hf_hub_download

REPO_ID = "lloid-labs/Lad1-0.2B-Instruct"
CKPT_FILENAME = "checkpoint.pt"  # adjust to actual filename in the repo
DEVICE = "cuda" if t.cuda.is_available() else "cpu"

class Config:
    vocab_size = 50257
    d_model = 1024
    n_heads = 16
    n_layers = 12
    seq_length = 512
    mlp_ratio = 4

# --- Model definition (GQA, RMSNorm pre-norm blocks, SiLU FFN, sinusoidal PE) ---
# See the project repo for the full Lad model class definition (GQA, TransformerBlock,
# PositionalEncoding, Lad). Import or paste it here before loading the checkpoint.

def load_model():
    ckpt_file = hf_hub_download(repo_id=REPO_ID, filename=CKPT_FILENAME)
    ckpt = t.load(ckpt_file, map_location=DEVICE)
    model = Lad(
        vocab_size=Config.vocab_size,
        d_model=Config.d_model,
        n_heads=Config.n_heads,
        n_layers=Config.n_layers,
        mlp_ratio=Config.mlp_ratio,
    ).to(DEVICE)
    model.load_state_dict(ckpt["model"])
    model.eval()
    return model

@t.no_grad()
def generate(model, tokenizer, prompt, max_new_tokens=128, temperature=0.7, top_k=40):
    ids = tokenizer.encode(prompt)
    x = t.tensor([ids], dtype=t.long, device=DEVICE)
    for _ in range(max_new_tokens):
        x_cond = x[:, -Config.seq_length:]
        logits = model(x_cond)[:, -1, :] / temperature
        if top_k is not None:
            v, _ = t.topk(logits, top_k)
            logits[logits < v[:, [-1]]] = float("-inf")
        probs = F.softmax(logits, dim=-1)
        next_id = t.multinomial(probs, num_samples=1)
        x = t.cat([x, next_id], dim=1)
        if next_id.item() == tokenizer.eos_token_id:
            break
    return tokenizer.decode(x[0].tolist())

tokenizer = AutoTokenizer.from_pretrained("gpt2")
tokenizer.pad_token = tokenizer.eos_token

model = load_model()
print(generate(model, tokenizer, "Explain machine learning in simple terms."))

No chat_template is provided since the model was fine-tuned on plain prompt/response pairs rather than a native transformers chat format. Format multi-turn input manually to match the SFT data structure.

Limitations

Lad1-0.2B-Instruct is a small language model and should not be expected to match the capabilities of larger models.

The model may produce incorrect, misleading, repetitive, or incoherent responses.

It has a relatively short 512-token context window and has not been specifically optimized for mathematics, function calling, or tool use.

The model should not be used as the sole source of information for high-stakes applications.

License

Lad1-0.2B-Instruct is released under the Apache-2.0 license.

Citation

If you use Lad1-0.2B-Instruct in your research or project, please cite:

@misc{lad1_0.2b_instruct,
  title        = {Lad1-0.2B-Instruct},
  author       = {lloid-labs},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/lloid-labs/Lad1-0.2B-Instruct}
}

Acknowledgements

Thanks to the teams behind FineWeb-Edu, SmolTalk, and the Hugging Face ecosystem for providing resources that made this project possible.