🐻🔥 BareTorch-2.5B-Base

🔗 Official GitHub Repository & Engine Source Code: https://github.com/martin-kbcc/baretorch

BareTorch-2.5B-Base is a 2.53-billion-parameter hybrid language model designed from the ground up for fast, long-context inference across cloud systolic arrays (Google TPUs), consumer GPUs (NVIDIA CUDA), and edge devices (Apple Silicon MLX)—without requiring custom hardware kernels.

🌍 Why BareTorch? (Built for Any Chip — From Cloud TPUs to Edge NPUs)

Most modern subquadratic architectures (such as Mamba-2, RWKV, and Gated DeltaNet) achieve fast linear scaling on NVIDIA datacenter GPUs by relying on custom CUDA, Triton, or platform-locked kernels. When developers try to deploy those models across Google Cloud TPUs (OpenXLA), Apple Silicon, mobile NPUs, Vulkan, or WebGPU, they hit a severe performance wall wherever custom hardware kernels are missing and models fall back to sequential loops. Meanwhile, standard Transformers suffer from quadratic $O(L^2)$ slowdowns and expanding KV-cache memory as context grows.

BareTorch solves this hardware lock-in. Its core sequence engine—the **Chunk-Segmented Low-Rank Associative Delta Engine (CS-LRAD)**—is formulated 100% out of standard matrix multiplications (GEMMs) for parallel chunk prefill and an exact $O(1)$ recurrent associative state update for single-token decoding. Because every AI processor on earth (Google TPU MXU systolic arrays, Apple M-series GPUs/ANEs, mobile NPUs, AMD/Intel GPUs, and NVIDIA Tensor Cores) already accelerates matrix multiplication natively, BareTorch delivers subquadratic long-context speed and compact memory usage out-of-the-box on any chip.

---

🚀 Key Highlights

• Universal Subquadratic Hybrid Architecture (24 Layers): Combines 18 CS-LRAD recurrent blocks and 6 Grouped-Query Attention (GQA) blocks in a repeating 3:1 pattern ([CS-LRAD, CS-LRAD, CS-LRAD, Transformer] x 6). Because 75% of the network uses a fixed-size low-rank recurrent state (rank = 16) instead of a growing KV cache, long prompts process rapidly with minimal memory overhead.
• 13.37x Faster Long-Context Prefill & 2.33x Faster Decoding on Google Cloud TPU (v5e-1 — BF16): Because Qwen3.5-2B relies on Triton/CUDA kernels (flash-linear-attention and causal_conv1d) unavailable on systolic-array TPUs, it falls back to sequential reference loops (17.51s TTFT / 1,871.3 tok/s at 32k). BareTorch's pure-GEMM CS-LRAD lowers natively onto the TPU MXU systolic array, processing 32,768 tokens in 1.31 seconds (25,018.2 tok/s — peaking at 26,168.6 tok/s) while decoding up to 2.33x faster (49.02 tok/s vs 21.07 tok/s) and compiling 2.56x faster.
• Edge & Workstation Acceleration (Apple Silicon MLX & NVIDIA CUDA): Under strict 4-bit native Apple MLX benchmarking on a MacBook Pro M1, BareTorch delivers ~34% faster prefill throughput (peaking at 1,204.1 tok/s and sustaining 1,028.0 tok/s vs 769.8 tok/s at 32k context), faster decoding (99.27 tok/s at 1k, 75.54 tok/s vs 71.28 tok/s at 32k), and 30% lower peak RAM at 32,768 tokens (3.52 GB vs 5.00 GB)—plus 44% faster decoding (65.4 tok/s vs 45.6 tok/s) on local NVIDIA CUDA hardware.
• Distilled from Qwen/Qwen3.5-9B on 75B Tokens: Pre-trained from scratch on 75 Billion tokens (142,000 steps) using a 50/50 multi-task objective (50% Hard Cross-Entropy + 50% Top-4 Logit KL Divergence from Qwen/Qwen3.5-9B) with a Warmup-Stable-Decay (WSD) curriculum schedule.
• Outperforms 300B-Token Baselines: Beats Pythia-2.8B and Mamba-1.4B (both trained on 4x more tokens) across all 7 standard zero-shot benchmarks, and outperforms Qwen3.5-2B-Base on physical commonsense (PIQA: 74.81%) and multi-step fact QA (OpenBookQA: 39.40%).

---

📊 Zero-Shot Downstream Benchmarks (lm-evaluation-harness)

All models were evaluated zero-shot (0-shot, sequence length 2,048, bfloat16) using the standard EleutherAI lm-evaluation-harness:

Benchmark Task	Metric	BareTorch-2.5B-Base (75B Tokens)	Pythia-2.8B (300B Tokens)	Mamba-1.4B (300B Tokens)	Qwen3.5-2B-Base
BoolQ (Reading Comprehension)	Accuracy	66.12%	64.10%	63.00%	69.72%
PIQA (Physical Commonsense)	Acc (Norm)	74.81%	74.10%	73.90%	74.65%
HellaSwag (Commonsense Reasoning)	Acc (Norm)	59.86%	59.40%	59.00%	65.90%
WinoGrande (Coreference Resolution)	Accuracy	60.54%	58.20%	61.40%	65.51%
ARC-Easy (Science QA)	Acc (Norm)	65.91%	59.10%	65.50%	71.30%
ARC-Challenge (Hard Science QA)	Acc (Norm)	38.48%	32.90%	32.90%	45.48%
OpenBookQA (Multi-Step Fact QA)	Acc (Norm)	39.40%	36.80%	37.00%	38.20%
7-Task Zero-Shot Average	Mean	57.87%	54.94%	56.10%	61.54%

---

⚡ Hardware Speed, TTFT & Memory Profiling: Sub-Quadratic vs. Sub-Quadratic (512 to 32,768 Tokens)

Why Sub-Quadratic vs. Sub-Quadratic (CS-LRAD vs. Gated DeltaNet)?
Comparing a sub-quadratic hybrid against a standard $O(L^2)$ full-attention Transformer at 32k+ context is not a fair competition, as sub-quadratic models hold an inherent algorithmic and KV-cache advantage over full attention. To establish a rigorous, apples-to-apples benchmark, we evaluate BareTorch-2.5B-Base (18 CS-LRAD + 6 GQA layers) directly against Qwen3.5-2B (18 Gated DeltaNet + 6 GQA layers)—a state-of-the-art sub-quadratic hybrid sharing the exact same 24-layer 3:1 hybrid topology.

Note on Context Length: BareTorch-2.5B-Base was pre-trained at a foundational sequence length of 2,048 tokens (rope_theta = 10,000). The hardware benchmarks below measure raw architectural speed, Time-To-First-Token (TTFT), prefill throughput, and memory scaling up to 32,768 tokens. Long-context sequence adaptation (16k–32k) and Supervised Fine-Tuning (SFT) will follow in upcoming releases.

1. Google Cloud TPU Benchmark (TPU v5e-1 16GB HBM — Native TorchTPU BF16)

Evaluated head-to-head on a Google Cloud TPU v5e-1 (16GB HBM) in bfloat16 (64 generated tokens per trial, last-logit-only prefill optimization enabled for both models). Because BareTorch's CS-LRAD is built from standard matrix multiplications and an $O(1)$ recurrent associative state update rather than Triton/CUDA-specific kernels, it lowers directly onto the TPU's $128 imes 128$ MXU systolic array—delivering up to 13.37x faster prefill (1.31s vs 17.51s at 32k), 2.06x–2.33x faster compiled decoding (49.02 tok/s peak, and up to +44.4% faster in uncompiled eager mode), and 2.56x faster XLA compilation:

Context Length	Model (Backend)	TTFT (ms)	Prefill Throughput	Decode Speed	Warmup / XLA Compile
512	BareTorch-2.5B (TorchTPU)	97.0 ms	5,276.7 tok/s	42.40 tok/s (26.80 eager)	54.4s
512	Qwen3.5-2B (TorchTPU)	188.9 ms	2,710.3 tok/s	20.56 tok/s	61.7s
1,024	BareTorch-2.5B (TorchTPU)	108.2 ms	9,462.5 tok/s	43.48 tok/s (27.59 eager)	63.8s
1,024	Qwen3.5-2B (TorchTPU)	298.7 ms	3,428.2 tok/s	22.20 tok/s	92.2s
2,048	BareTorch-2.5B (TorchTPU)	129.9 ms	15,770.6 tok/s	49.02 tok/s (24.83 eager)	65.0s
2,048	Qwen3.5-2B (TorchTPU)	513.9 ms	3,984.9 tok/s	21.07 tok/s	105.6s
4,096	BareTorch-2.5B (TorchTPU)	186.8 ms	21,923.1 tok/s	42.71 tok/s (24.33 eager)	84.3s
4,096	Qwen3.5-2B (TorchTPU)	897.6 ms	4,563.4 tok/s	21.27 tok/s	118.7s
8,192	BareTorch-2.5B (TorchTPU)	332.5 ms	24,640.6 tok/s	42.89 tok/s (29.33 eager)	92.4s
8,192	Qwen3.5-2B (TorchTPU)	2,272.2 ms	3,605.2 tok/s	20.31 tok/s	160.3s
16,384	BareTorch-2.5B (TorchTPU)	626.1 ms	26,168.6 tok/s	41.17 tok/s (25.09 eager)	143.6s
16,384	Qwen3.5-2B (TorchTPU)	5,733.5 ms	2,857.6 tok/s	20.63 tok/s	286.5s
32,768	BareTorch-2.5B (TorchTPU)	1,309.8 ms	25,018.2 tok/s	43.38 tok/s (27.97 eager)	148.6s
32,768	Qwen3.5-2B (TorchTPU)	17,510.6 ms	1,871.3 tok/s	21.07 tok/s	380.7s

2. Apple Silicon Edge Benchmark (MacBook Pro M1 — Native MLX 4-Bit)

Both models were evaluated head-to-head in native Apple MLX using identical 4-bit quantization (mlx.nn.quantize, bits=4, group_size=64). Even with zero custom Metal shaders, BareTorch-2.5B achieves ~34% faster prefill across all context lengths, 99.3 tok/s short-context / 75.5 tok/s 32k-context decode speed, and cuts 32k peak unified memory by 1.48 GB (3.52 GB vs 5.00 GB):

Context Length	Model (Backend)	TTFT (ms)	Prefill Throughput	Decode Speed	Peak RAM
512	BareTorch-2.5B (MLX 4-bit)	437.9 ms	1,169.3 tok/s	99.06 tok/s	1.99 GB
512	Qwen3.5-2B (MLX 4-bit)	576.4 ms	888.2 tok/s	98.82 tok/s	1.84 GB
1,024	BareTorch-2.5B (MLX 4-bit)	855.5 ms	1,197.0 tok/s	99.27 tok/s	2.28 GB
1,024	Qwen3.5-2B (MLX 4-bit)	1,140.1 ms	898.2 tok/s	99.07 tok/s	2.10 GB
2,048	BareTorch-2.5B (MLX 4-bit)	1,700.8 ms	1,204.1 tok/s	95.85 tok/s	2.41 GB
2,048	Qwen3.5-2B (MLX 4-bit)	2,271.7 ms	901.5 tok/s	96.63 tok/s	2.32 GB
4,096	BareTorch-2.5B (MLX 4-bit)	3,438.4 ms	1,191.2 tok/s	93.63 tok/s	2.69 GB
4,096	Qwen3.5-2B (MLX 4-bit)	4,617.7 ms	887.0 tok/s	94.39 tok/s	2.76 GB
8,192	BareTorch-2.5B (MLX 4-bit)	6,963.2 ms	1,176.5 tok/s	90.85 tok/s	2.69 GB
8,192	Qwen3.5-2B (MLX 4-bit)	9,386.1 ms	872.8 tok/s	90.65 tok/s	3.18 GB
16,384	BareTorch-2.5B (MLX 4-bit)	14,549.3 ms	1,126.1 tok/s	85.05 tok/s	3.33 GB
16,384	Qwen3.5-2B (MLX 4-bit)	19,503.7 ms	840.0 tok/s	83.17 tok/s	3.75 GB
32,768	BareTorch-2.5B (MLX 4-bit)	31,876.9 ms	1,028.0 tok/s	75.54 tok/s	3.52 GB
32,768	Qwen3.5-2B (MLX 4-bit)	42,567.7 ms	769.8 tok/s	71.28 tok/s	5.00 GB

3. Local NVIDIA GPU Workstation Benchmark (CUDA — BF16)

Measured in strict apples-to-apples mode (benchmark_context.py, 64 generated tokens per trial, last-logit-only prefill optimization enabled for both models). BareTorch delivers 44% faster token generation (65.4 tok/s vs 45.6 tok/s) across all context lengths:

Context Length	Model	TTFT (ms)	Prefill Throughput	Decode Speed	Peak VRAM
512	BareTorch-2.5B (Compiled)	18.22 ms	28,108.3 tok/s	66.17 tok/s	4.77 GB
512	Qwen3.5-2B-Base	39.93 ms	12,821.2 tok/s	45.53 tok/s	4.52 GB
1,024	BareTorch-2.5B (Compiled)	27.95 ms	36,631.8 tok/s	65.45 tok/s	4.81 GB
1,024	Qwen3.5-2B-Base	38.75 ms	26,425.4 tok/s	45.29 tok/s	4.56 GB
2,048	BareTorch-2.5B (Compiled)	50.46 ms	40,583.0 tok/s	65.41 tok/s	4.88 GB
2,048	Qwen3.5-2B-Base	49.93 ms	41,020.3 tok/s	45.69 tok/s	4.64 GB
4,096	BareTorch-2.5B (Compiled)	103.03 ms	39,756.7 tok/s	65.39 tok/s	5.03 GB
4,096	Qwen3.5-2B-Base	103.54 ms	39,558.0 tok/s	45.70 tok/s	4.81 GB
8,192	BareTorch-2.5B (Compiled)	219.34 ms	37,348.6 tok/s	65.36 tok/s	5.34 GB
8,192	Qwen3.5-2B-Base	231.62 ms	35,368.6 tok/s	45.57 tok/s	5.14 GB
16,384	BareTorch-2.5B (Compiled)	461.37 ms	35,511.6 tok/s	66.15 tok/s	5.94 GB
16,384	Qwen3.5-2B-Base	487.32 ms	33,620.6 tok/s	46.17 tok/s	5.80 GB
32,768	BareTorch-2.5B (Compiled)	995.40 ms	32,919.4 tok/s	63.79 tok/s	7.14 GB
32,768	Qwen3.5-2B-Base	1,044.72 ms	31,365.4 tok/s	45.82 tok/s	7.12 GB

---

📐 Model Architecture & Pre-Training Specifications

Specification	Configuration
Source Code Repository	https://github.com/martin-kbcc/baretorch
Total Parameters	2.53 Billion (Untied Input/Output Embeddings)
Total Layers	24 Layers (18 CS-LRAD Recurrent Layers + 6 Transformer GQA Layers)
Interleaved Layer Pattern	[cs_lrad, cs_lrad, cs_lrad, transformer] x 6 (3:1 Hybrid Ratio)
Hidden Dimension (d_model)	2,048
Attention Heads	16 Query Heads / 4 KV Heads (GQA 4:1, head_dim = 128) with Per-Head QK-Norm
CS-LRAD Chunk Size & Rank	chunk_size = 32, Low-Rank State Subspace rank = 16
Pre-Training Sequence Length	2,048 tokens (rope_theta = 10,000.0)
Tokenizer	Qwen/Qwen3.5-9B (vocab_size = 248,077)
Pre-Training Token Volume	75 Billion Tokens (142,000 steps @ 524,288 tokens/step)
Distillation Objective	Teacher: Qwen/Qwen3.5-9B (0.5 * CrossEntropy + 0.5 * Top4_KL, T = 1.0)
Optimizer & WSD Schedule	AdamW (lr = 3e-4, weight_decay = 0.1, warmup = 1,000 steps, 90% Stable / 10% Cosine Decay)
Training Hardware	8x NVIDIA H100 SXM5 (80GB) with PyTorch FSDP (ZeRO-2) & torch.compile
Training Corpus Mix	General Phase (90% / 67.5B tokens): FineWeb-Edu, The Stack Dedup, DCLM, FinePDFs WSD Annealing Phase (10% / 7.5B tokens): FineMath-4+, Cosmopedia-v2, OpenR1-Math

---

💻 Quickstart Usage

1. PyTorch via Hugging Face transformers (Zero Installation Required)

You can load BareTorch-2.5B-Base directly using standard Hugging Face AutoModelForCausalLM with trust_remote_code=True—no custom packages or repositories required:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "model-rampage/BareTorch-2.5B-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    trust_remote_code=True,
)
model.eval()

# Optional: Compile CS-LRAD recurrent blocks for peak CUDA speed (~65.6 tok/s)
model.compile_lrad()

prompt = "The biggest advantage of running a pure-GEMM subquadratic language model on edge devices is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=64,
        do_sample=False,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

2. Apple Silicon (Native MLX — 4-Bit Quantized or FP16)

For native Apple Silicon MLX execution, install the official BareTorch repository from GitHub (https://github.com/martin-kbcc/baretorch):

git clone https://github.com/martin-kbcc/baretorch.git
cd baretorch
pip install -e .

import mlx.core as mx
import mlx.nn as nn
from transformers import AutoTokenizer
from baretorch.integration_mlx.modeling_baretorch import (
    BareTorchForCausalLMMLX,
    extract_baretorch_cache_arrays,
)

model_path = "./BareTorch-2.5B-Base"  # Local clone of model-rampage/BareTorch-2.5B-Base
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = BareTorchForCausalLMMLX.from_pretrained(model_path, dtype=mx.float16, fuse=True)

# Optional: Quantize in-place to 4-bit (~99 tok/s decode, 3.52 GB RAM @ 32k context)
nn.quantize(
    model,
    group_size=64,
    bits=4,
    class_predicate=lambda p, m: hasattr(m, "weight") and m.weight.ndim == 2 and m.weight.shape[-1] % 64 == 0,
)
mx.eval(model.parameters())

prompt = "Edge AI applications require models that can process large documents locally because"
input_ids = mx.array([tokenizer.encode(prompt)])

logits, past_kv = model(input_ids, use_cache=True, num_logits_to_keep=1)
next_token = mx.argmax(logits[:, -1:, :], axis=-1)
mx.eval(next_token, extract_baretorch_cache_arrays(past_kv))

---

🗺️ Release Roadmap

• Phase 1 — Foundational Pre-Training & Distillation: BareTorch-2.5B-Base trained on 75B tokens at 2,048 context length.
• Phase 2 — Long-Context Extension (In Progress): Context window expansion from 2,048 to 16,384 / 32,768 tokens.
• Phase 3 — Supervised Fine-Tuning (In Progress): BareTorch-2.5B-Instruct tuned for long-context document QA, coding, and reasoning (<think>).

---

📜 Citation

@article{kovacevic2026baretorch,
  title={BareTorch: Challenging State-of-The-Art Sequence Mixing Topologies via Kernel-Free, Pure GEMM-Compliant Architectures},
  author={Kovacevic Buvinic, Martin Ignacio},
  journal={BareTorch Framework Laboratory Technical Report},
  year={2026},
  url={https://github.com/martin-kbcc/baretorch}
}