ztor2/jerboa-sft

Jerboa

Lightweight Language & Multimodal Model (in active development)

Jerboa is an ultra-lightweight language and multimodal model (~221M parameters) featuring a 28-layer Deep & Thin architecture optimized for Apple Silicon (MPS / Metal) and edge deployments.

---

Model Specifications

Attribute	Specification
Language Backbone	JerboaForCausalLM (~221M parameters)
Multimodal Model	JerboaVLForConditionalGeneration
Layers & Hidden Dim	28 Layers, $d_{model}=768$, $d_{ffn}=2048$ (SwiGLU)
Attention	GQA (12 Query : 4 KV heads), QK-Norm, Interleaved SWA (window 2048, global every 4 layers)
Vocabulary & RoPE	49,152 BPE Vocab, RoPE $ heta=500,000$
Pre-training Volume	> 1.23 Billion tokens (75% FineWeb-Edu, 25% Python Code)
Hardware Target	Apple Silicon (MPS / Metal) & Edge Devices

---

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ztor2/jerboa-sft"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

prompt = "Hello, what are you?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))