ztor2/jerboa-base

Jerboa

Lightweight Language & Multimodal Model (in active development)

Jerboa is an ultra-lightweight language and multimodal model (~138M–146M parameters) optimized for Apple Silicon (MPS / Metal) and edge deployments.

---

Model Specifications

Attribute	Specification
Language Backbone	JerboaForCausalLM (~138.4M base / ~145.9M with MTP)
Multimodal Model	JerboaVLForConditionalGeneration (~145.2M)
Layers & Hidden Dim	16 Layers, $d_{model}=768$, $d_{ffn}=2048$ (SwiGLU)
Attention	GQA (12 Query : 4 KV heads), QK-Norm, Interleaved SWA
Context Length	4,096 tokens (extensible to 16K via YaRN)
Hardware Target	Apple Silicon (MPS / Metal) & Edge Devices

---

Quickstart

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ztor2/jerboa-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

prompt = "Hello, what are you?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))