🐻🔥 BareTorch-500M-Base
BareTorch-500M Base is a foundational sub-quadratic language model built under a pure GEMM-compliant, kernel-free paradigm. The model combines CS-LRAD (Chunk-Segmented Low-Rank Associative Delta Engine) recurrent layers with standard Transformer multi-head self-attention in a 3:1 interleaved hybrid topology ($3\times\text{CS-LRAD} \to 1\times\text{Transformer}$).
By structuring sub-quadratic state updates into block-parallel chunk segments ($C=32$), BareTorch bypasses the compilation and hardware lock-in of custom CUDA or Triton kernels, running with $O(N)$ execution and memory efficiency natively across NVIDIA CUDA, Apple Silicon MLX, WebGPU, and TPUs.
📐 Model Architecture Specifications
- Parameters: ~500M (498.2M active parameters)
- Hidden Dimension ($d_{model}$): 1,152
- Total Layers: 24 (Interleaved 18x CS-LRAD + 6x Transformer)
- Attention Heads: 16 ($d_{head} = 72$)
- CS-LRAD Subspace Rank ($r$): 8
- Chunk Size ($C$): 32 tokens
- Tokenizer:
HuggingFaceTB/SmolLM2-360M (Vocab size: 49,152)
- Context Window: Up to 32,768 tokens
🏋️ Pre-Training Runway Specs & Loss Convergence
- Training Runway: 100 Billion Tokens ($190,735$ optimization steps)
- Hardware Cluster: $4\times$ NVIDIA H100 SXM (80GB VRAM)
- Global Batch Size: $524,288$ tokens/step ($256$ sequences of length $2048$)
- Optimizer & LR: AdamW ($ ext{LR}_{peak} = 6 \times 10^{-4}$, weight decay $0.1$, cosine decay scheduler with $2,000$ warmup steps)
Pre-Training Evaluation Loss Milestones
| Optimization Step |
Tokens Processed |
Train Loss |
Eval Loss |
| Step 1,000 |
~0.52 Billion |
5.2876 |
-- |
| Step 10,000 |
~5.24 Billion |
2.7632 |
2.6943 |
| Step 50,000 |
~26.21 Billion |
2.5275 |
2.4836 |
| Step 100,000 |
~52.43 Billion |
2.4324 |
2.3965 |
| Step 150,000 |
~78.64 Billion |
2.3355 |
2.3054 |
| Step 190,735 (Final) |
100.0 Billion |
2.2901 |
2.2690 |
📊 Zero-Shot Downstream Benchmarks
| Task / Benchmark |
Metric |
Score |
| HellaSwag |
Acc (Norm) |
43.69% |
| ARC Easy |
Acc (Norm) |
53.58% |
| ARC Challenge |
Acc (Norm) |
28.92% |
| WinoGrande |
Accuracy |
51.30% |
| MMLU (Overall 57-Subject Average) |
Accuracy |
24.70% |
| ** ├─ MMLU STEM** |
Accuracy |
23.53% |
| ** ├─ MMLU Humanities** |
Accuracy |
24.87% |
| ** ├─ MMLU Social Sciences** |
Accuracy |
24.86% |
| ** └─ MMLU Other** |
Accuracy |
25.46% |
⚡ Long-Context Inference Hardware Scaling (32,768 Context)
BareTorch replaces context-dependent KV-caches with constant-sized $O(1)$ recurrent state updates, eliminating memory bus bottlenecks and out-of-memory crashes on long-context workloads.
1. Discrete CUDA GPU (NVIDIA RTX 4090 - 24GB)
| Baseline Pair |
Context |
Prefill Latency |
Local GPU Decode |
Peak VRAM |
Advantage |
| Qwen3 0.6B Match |
32,768 |
314.89 ms vs 4,343.26 ms |
164.49 tok/s vs 13.15 tok/s |
1.66 GB vs 8.58 GB |
12.51x Faster (-80.6% VRAM) |
| SmolLM2 1.7B Match |
32,768 |
1,134.20 ms vs 4,292.54 ms |
98.92 tok/s vs 15.79 tok/s |
4.39 GB vs 15.57 GB |
6.26x Faster (-71.8% VRAM) |
| Llama 3.2 1B Match |
32,768 |
598.36 ms vs 2,960.03 ms |
169.24 tok/s vs 24.44 tok/s |
3.02 GB vs 4.67 GB |
6.92x Faster (-35.4% VRAM) |
| Gemma 2 2B Match |
32,768 |
1,049.72 ms (Baseline: 💥 OOM) |
101.51 tok/s (Baseline: 💥 OOM) |
5.96 GB (Baseline: 💥 OOM) |
Prevents OOM Crashes |
2. Apple Silicon Unified Memory (M1 MacBook Pro 16GB - Native MLX)
| Baseline Pair |
Context |
Prefill Latency |
Local GPU Decode |
Peak VRAM |
Advantage |
| SmolLM2 1.7B Match |
32,768 |
29.56 s vs 59.03 s |
29.69 tok/s vs 0.66 tok/s |
3.78 GB vs 9.97 GB |
44.98x Faster (-62.1% VRAM) |
| Llama 3.2 1B Match |
32,768 |
18.69 s vs 41.06 s |
28.42 tok/s vs 3.29 tok/s |
2.81 GB vs 3.71 GB |
8.64x Faster (-24.4% VRAM) |
| Qwen3 0.6B Match |
32,768 |
7.06 s (Baseline: 💥 OOM) |
79.55 tok/s (Baseline: 💥 OOM) |
1.37 GB (Baseline: 💥 OOM) |
Prevents OOM Crashes |
💻 Usage & Code Example
import torch
from transformers import AutoTokenizer
from baretorch.integration.configuration_baretorch import BareTorchConfig
from baretorch.integration.modeling_baretorch import BareTorchForCausalLM
model_id = "model-rampage/BareTorch-500M-Base"
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-360M")
model = BareTorchForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).cuda()
prompt = "The key innovation of pure GEMM sub-quadratic architectures is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
with torch.no_grad():
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
📜 Citation
@article{kovacevic2026baretorch,
title={BareTorch: Challenging State-of-The-Art Sequence Mixing Topologies via Kernel-Free, Pure GEMM-Compliant Architectures},
author={Kovacevic Buvinic, Martin Ignacio},
journal={BareTorch Framework Laboratory Technical Report},
year={2026}
}
