nayeshdaggula/sutradhar-3b-moe-base-backup-before-v02

Sutradhar 3B MoE Base v0.1

Sutradhar 3B MoE Base is the first public checkpoint in the Sutradhar model family — an India-inspired, from-scratch Mixture-of-Experts language model.

This release is a 300-step validation checkpoint, not a finished base model.

It is useful for verifying:

• model architecture
• tokenizer loading
• MoE routing
• checkpoint saving
• Hugging Face loading
• continued training workflow

The name Sutradhar comes from Sanskrit, meaning “thread-holder” or “narrator/orchestrator.”
This name represents the model’s MoE architecture, where a router connects different experts to generate the final response.

Important: This is an early validation checkpoint trained for architecture, tokenizer, dataset, checkpointing, and upload-flow testing. It is not production-ready and should not be used as a final chat or coding model yet.

---

Model Details

Field	Value
Model name	Sutradhar 3B MoE Base
Version	v0.1 / 300-step checkpoint
Architecture	Mixtral-style decoder-only Mixture-of-Experts
Model type	Causal Language Model
Training type	From scratch
Total parameters	~3.283B
Tokenizer vocabulary	64,000
Context length used	1,024 tokens
Experts	8
Experts per token	2
Hidden size	2,048
Layers	24
Attention heads	16
KV heads	8
Precision	bfloat16
Training hardware	2× NVIDIA A100 80GB
Training steps completed	300
Approx training tokens	~9.83M

---

What This Checkpoint Proves

This checkpoint validates that the full model-building pipeline works:

• Custom tokenizer loading
• From-scratch MoE model initialization
• Mixtral-style expert routing
• Dataset processing
• Distributed/large-model training flow
• Checkpoint saving
• Safetensors export
• Hugging Face upload and reload testing

This is the first public base checkpoint in the Sutradhar model family.

---

Intended Use

This checkpoint is mainly intended for:

• Research and experimentation
• MoE architecture validation
• Continued pretraining
• Tokenizer and dataset testing
• Building future SFT/instruct versions
• Testing loading, generation, and quantization workflows

It is not yet intended for:

• Production chatbots
• Real customer-facing applications
• Code generation reliability
• Factual question answering
• Safety-critical use cases

---

Model Family Plan

Future releases may include:

Model	Purpose
Sutradhar 3B MoE Base	Base pretraining checkpoint
Sutradhar 3B MoE Instruct	Chat/instruction-tuned model
Sutradhar 3B MoE Agent	Agentic/tool-use model
Sutradhar 3B MoE Coder	Coding-focused model
Sutradhar 7B Base	Larger dense/base model
Sutradhar 7B Instruct	Higher-quality instruction model

---

Quick Start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "nayeshdaggula/sutradhar-3b-moe-base"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

prompt = "India is known for"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))