HobbyLM-Omni (500M MoE, text + image + audio)

HobbyLM-Omni is the multimodal core: one 500M MoE model that handles text, image, video, audio, and speech — plus tool use, OCR, and UI grounding — folded into a single checkpoint across 18 training paths (TinyLLaVA-style projectors over frozen SigLIP2 / Whisper / CLAP front-ends). The headline isn't any single score; it's the breadth in one small model.

It's part of the HobbyLM family — a 500M sparse-MoE model (and its variants) built from scratch on a hobby budget: FineWeb, a handful of Modal H100 hours, a lot of ablations, and a from-scratch Rust engine (hobby-rs) to run it on a laptop CPU.

Intended use

Vision-language and audio-language tasks: captioning, visual QA, OCR, sound/speech understanding, spoken-question answering, and tool calling. Image/audio/speech features are projected and spliced at the [IMAGE]/[AUDIO]/[SPEECH] sentinel tokens (ids 50257–50262).

Architecture

Every HobbyLM variant shares one core: a sparse Mixture-of-Experts (MoE) decoder in the modern small-MoE style (DeepSeek-V3 / OLMoE lineage), where each design choice was picked by ablation rather than by guesswork.

Component	Value
Total parameters	~500M (only a fraction is active per token)
Hidden size / layers	768 / 16 (first FFN dense, the rest MoE)
Routed experts / active	36 / top-6 (+ 1 always-on shared expert)
Attention	GQA, 12 query / 3 KV heads, decoupled head-dim 128, per-head QK-norm
Router	sigmoid gating, DeepSeek-V3 aux-loss-free load balancing, no top-k renorm
Positional	RoPE (θ up to 1e6 for the 8k-context checkpoints)
Tokenizer	GPT-2 byte-level BPE (50,304 vocab, sentinel-padded)
Optimizer	Muon on the 2-D + per-expert matrices, AdamW on everything else

The full ablation log (QK-norm is the single biggest lever; aux-loss-free beats classic aux-loss; ≥32 experts and top-6 help; embedding-scaling hurt) lives in the project's architecture notes.

Multimodal use

This repo also ships the projector weights — vision_projector.safetensors (SigLIP2 → LLM) and speech_projector.safetensors (Whisper-mel → LLM), plus melfilters.bytes. The frozen front-ends encode the raw image/audio, the projectors map those features into the LLM embedding space, and they're spliced in at the modality sentinel tokens.

Benchmarks

Visual QA is scored with containment (the model is chat-trained and answers in full sentences, so strict single-word exact-match badly under-scores it):

Task	Score
VQAv2 (val)	47.0
GQA	39.2
POPE — accuracy / F1	50.0 / 66.7
Tool calling — Needle (JSON-parse / Name-F1 / param-halluc)	93.8 / 77.7 / 0.0
BFCL (forced-call: simple / multiple)	21.7 / 18.3
Text — lm-eval 9-task avg	0.432

POPE at 50/66.7 is a real ceiling — object-presence hallucination ("yes" to everything) is the known small-VLM weakness, quantified. On function calling, Omni can call as well as the dedicated tool model (forced-call simple 21.7 ≈ the specialist's 22.7); left to itself it prefers to abstain (irrelevance 86.7), a safer agent failure mode. Speech does spoken-QA and commands rather than verbatim transcription.

How these were measured. All language-model scores are 0-shot through our own port of EleutherAI's lm-evaluation-harness (a custom MoELMWrapper that runs log-likelihood scoring over the HobbyLM MoE + GPT-2 tokenizer). Reference models in the comparison table were run through the identical harness and task set, so the numbers are apples-to-apples with ours — they are not copied from other model cards. We validated the harness against published cards (e.g. TinyLlama 52.75 vs card 52.99). These are small research models: read the numbers in context, not as leaderboard claims.

Usage

Python (PyTorch reference implementation)

HobbyLM is a custom sparse-MoE architecture — there's no transformers AutoModel for it, so load it with the small reference implementation from the GitHub repo:

# HobbyLM is a CUSTOM sparse-MoE architecture, so load it with the reference implementation —
# NOT transformers.AutoModelForCausalLM (there is no AutoModel mapping for this arch).
# pip install torch safetensors tiktoken huggingface_hub
# git clone https://github.com/harishsg993010/HobbyLM && cd HobbyLM

import json, torch, tiktoken
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from hobbylm.config import ModelConfig
from hobbylm.model import MoETransformer
from hobbylm.generate import generate

repo = "rootxhacker/HobbyLM-Omni"
cfg = ModelConfig(**{k: v for k, v in json.load(open(hf_hub_download(repo, "config.json"))).items() if k != "preset"})
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
cfg.expert_backend = "grouped" if device.type == "cuda" else "bmm"

model = MoETransformer(cfg).to(device).eval()
model.load_state_dict(load_file(hf_hub_download(repo, "model.safetensors")))

enc = tiktoken.get_encoding("gpt2")
prompt = "USER: Explain a mixture-of-experts model in one sentence.\nASSISTANT:"
ids = torch.tensor([enc.encode_ordinary(prompt)], device=device)
out = generate(model, ids, max_new_tokens=64, temperature=0.7, top_k=0, device=device,
               repetition_penalty=1.3)               # temperature=0.0 for greedy
print(enc.decode(out[0].tolist()))

The snippet above is the text path. For image / audio / speech, encode the input with the (frozen) SigLIP2 / Whisper / CLAP front-end, project it with the bundled projectors, and splice it at the modality sentinel token — see hobbylm/multimodal.py, or just pass --image / --speech to hobby-rs.

GGUF + hobby-rs (CPU)

GGUF builds (architecture hobbylm) live in rootxhacker/HobbyLM-gguf. They load directly in the from-scratch hobby-rs CPU engine — stock llama.cpp won't load them without registering the hobbylm architecture first.

hobby-rs --model HobbyLM-Omni.gguf --prompt "..." --n 64

Training

Built in stages on the context-extended (8k, θ 1e6) backbone with a 512px SigLIP2 vision tower: projector alignment → multimodal SFT → a joint 18-path co-training cycle (image / video / audio / speech / text / tools / OCR / UI-grounding) that keeps every modality from drifting.

Limitations

• Breadth over depth: strong in-distribution (VQA, JSON tool calls with 0 hallucination, OCR, grounding) but below specialist sub-1B models on hard text reasoning (GSM8K, multi-hop QA).
• Object-presence hallucination on POPE-style probes.
• Verbose by default — ask for short answers explicitly, or score with containment, not exact-match.

License

Apache-2.0. Weights aren't a substitute for judgement — this is a research / hobby model at the 500M scale, not a production system.