SWARM-v1: Deterministic Reasoning Model (35.8M Parameters)

SWARM-v1 is a compact 35.8-million parameter decoder-only transformer trained from scratch on algorithmic and logical reasoning problems generated by the SWARM autonomous protocol on Solana.

Unlike marketing-heavy releases, SWARM-v1 does not claim to replace frontier foundation models. It is a purpose-built, lightweight, 100% open-weights reasoning model designed to run with sub-200ms latency on commodity CPU hardware with zero API dependencies.

---

Model Architecture

SWARM-v1 uses a modern LLaMA-style decoder-only transformer architecture with SwiGLU activations, RMSNorm, and Rotary Position Embeddings (RoPE), configured with an independent custom Byte-Level BPE tokenizer.

Parameter	Value	Note
Total Parameters	35,756,544 (~35.8M)	Exact non-embedding + embedding weights
Layers (Transformer Blocks)	6	Decoder-only self-attention
Hidden Size ($d_{\text{model}}$)	512	Representation dimension
Intermediate Size	1,376	SwiGLU feed-forward projection
Attention Heads	8	64 dimensions per head
Vocabulary Size	16,384	Custom Byte-Level BPE tokenizer
Context Window	1,024 tokens	Trained on sequence length 96
Position Embeddings	RoPE	Rotary position embeddings
Normalization	RMSNorm	$\epsilon = 10^{-5}$
Weight Format	SafeTensors	143 MB (model.safetensors, float32)

---

Autonomous Reasoning Ascent & Intelligence Convergence (Generation 1)

The model was trained from scratch with randomly initialized weights (no pre-trained foundation base weights were used).

• Training Dataset: octoswarm/swarm-judgements (train.jsonl, 1,600 verified reasoning examples across 8 task types and 10 difficulty tiers).
• Random Baseline Loss: 9.7041 ($\ln(16384)$)
• Final Training Loss: 1.0608
• Final Validation Loss: 1.2210
• Test Set Loss: 1.8986 (Perplexity: 6.68)
• Test Accuracy: 6.00% exact match across held-out difficulty 1–10 problems
• CPU Inference Latency: ~180.4 ms / query

---

Quickstart & Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

repo_id = "octoswarm/swarm-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id)

prompt = "Question: What is 7 * 8?\nAnswer: <answer>"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=16,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id or 0,
        eos_token_id=tokenizer.eos_token_id or 2
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=False))