SGI-Preview
BF16 | NVFP4 | Base Model | DFlash2 | Launch Blog | License: Apache 2.0
We introduce SGI-Preview, a 70B parameter dense Transformer model pretrained on 12 trillion tokens with support for up to 1M context length. SGI-Preview delivers strong results on agentic coding and reasoning, scoring 80.52 on TerminalBench 2.1 and surpassing significantly larger open-weight models including Reflection Beam 501B (80.1), Meta Muse Spark Open 816B (76.4), Thinking Machines Inkling 975B (63.8), and NVIDIA Nemotron 3 Ultra 550B (56.4).
Our Preview release includes an instruction-tuned model in BF16 and NVFP4 numerics, along with a pretrain-only BF16 base model. This repository contains the BF16 instruction-tuned model.
Starting from an initial Olmo 3 baseline, we iteratively improved all aspects of the training recipe including scaling laws, architecture, data curation, and the post-training stack. Our final model ladder improves Compute Efficiency (CE) on code by 16x relative to the baseline ladder, with advantages reaching up to 55x at full scale.
SGI-Preview was codesigned for high-efficiency training and inference on NVIDIA hardware. A core objective was reducing infrastructure complexity while maximizing reasoning and agentic performance. We selected a modernized dense Transformer architecture over Mixture-of-Experts (MoE), enabling training from scratch with a lean team while maintaining a consistent ~40% Model Flops Utilization (MFU) across pretraining and posttraining using standard sharding (BF16), without custom kernels or early quantization.
Architectural Highlights
- Aspect Ratio: Thin-and-deep design with a narrow
d_model(5120) and 80 total layers. - Backbone Stability: Transitioned from an Olmo 3-style setup to a Gemma 3N-like architecture for enhanced large-scale stability.
- Attention Design: Overprovisioned attention heads paired with an increased SWA-to-global layer ratio and compact SWA window sizes.
- Stabilization Features: FFN activation clipping, gated attention, global NoPE, and value residuals.
At 70B dense parameters, the model is compact enough to run on workstation setups (e.g., NVIDIA DGX Spark via the NVFP4 checkpoint) while sustaining generation throughputs of ~240 tok/s on datacenter nodes (2x B200, batch=8).
Training Overview
- Pre-training: Trained from scratch on a 12T token text and code corpus (predominantly English) at an 8K sequence length.
- Post-training: Multi-stage pipeline covering midtraining, context window extension, SFT, long-context SFT, and mixed-domain RL.
| Stage | Tokens | Steps | Seqlen | Batch Size |
|---|---|---|---|---|
| Pretrain | 12T | 877,500 | 8K | 1,408 โ 3,072 |
| Midtrain | 200B | 8,000 | 64K | 384 |
| Long Context | 6.5B | 389 | 2M | 8 |
| SFT | 183B | 7,274 | 128K | 192 |
| LC SFT | 13.5B | 802 | 2M | 8 |
| RL | N/A | 700 | 128K | 2,048 |
Instruct Model Benchmarks
Evaluated at full BF16 precision with maximum effort levels. Official model card results are shown where available; values in italics denote in-house benchmark reproductions. Higher is better across all metrics.
| Benchmark | SGI-Preview (70B Dense) |
Nemotron 3 Ultra 550B |
Inkling 975B |
Muse Spark Open 816B |
Reflection Beam 501B |
|---|---|---|---|---|---|
| Architecture | Dense | MoE | MoE | MoE | MoE |
| Total Params | 70B | 550B | 975B | 816B | 501B |
| Active Params | 70B | 55B | 41B | 42B | 23B |
| Pretrain Tokens | 12T | 20T | 45T | N/A | 23.8T |
| Knowledge | |||||
| MMLU | 89.6 | 92.0 | 91.3 | 92.6 | 92.4 |
| MMLU-Pro | 84.2 | 86.8 | 87.3 | 87.9 | 76.5 |
| GPQA Diamond | 85.2 | 87.0 | 87.2 | 89.0 | 90.6 |
| Math | |||||
| AIME 2024 | 97.1 | 94.2 | 97.5 | 97.9 | 96.7 |
| AIME 2025 | 98.3 | 94.2 | 91.7 | 100.0 | 99.6 |
| AIME 2026 | 95.0 | 95.4 | 97.1 | 98.3 | 97.7 |
| HMMT Feb 2026 | 90.5 | 80.3 | 86.3 | 93.9 | 93.9 |
| Coding & Agentic | |||||
| Terminal-Bench 2.1 (Self-reported) | 80.5 | 56.4 | 63.8 | 76.4 | 80.1 |
| Terminal-Bench 2.1 (Terminus 2) | 80.5 | 55.1 | 56.2 | 78.7 | 70.8 |
| SWE-bench Verified | 76.5 | 71.9 | 77.6 | 78.7 | 81.6 |
| SWE-bench Pro v1.0 | 61.4 | 46.9 | 54.3 | 58.2 | 66.1 |
| SWE-bench Pro v2.0 Hard | 79.1 | 30.7 | 2.0 | 52.3 | 76.5 |
| DeepSWE 1.1 | 44.3 | 0.2 | 1.8 | 60.0 | 43.3 |
| SciCode (Inspect AI) | 39.4 | 44.6 | 46.1 | 58.5 | 49.2 |
| ฯยณ-bench Banking | 31.3 | 22.6 | 23.7 | 14.4 | 36.1 |
| AutomationBench (Public) | 36.4 | 8.4 | 14.6 | 42.4 | 39.3 |
| MCP Atlas | 71.7 | 62.2 | 76.0 | 89.2 | 80.8 |
| Safety & Alignment | |||||
| FORTRESS Adversarial | 83.8 | 79.3 | 78.0 | 94.2 | 92.1 |
| FORTRESS Benign | 95.5 | 89.1 | 95.9 | 82.6 | 86.4 |
| StrongREJECT | 97.5 | 98.2 | 98.6 | 94.7 | 95.8 |
| Instruction & Context | |||||
| IFBench | 79.2 | 81.7 | 79.8 | 78.6 | 79.6 |
| AA-LCR (Long Context) | 75.0 | 65.4 | 78.3 | 80.7 | 78.2 |
Base Model Benchmarks
Evaluation results for SGI-Base-Preview (pretrain-only checkpoint) against representative open-weight base models using standardized multi-shot setups:
| Benchmark (Metric) | Shots | SGI-Base-Preview | Laguna M.1 225B | DeepSeek V3.1 671B | DeepSeek V4 Flash 284B | Nemotron 3 Ultra 550B | | :--- | :---: | :---: | :---: | :---: | :---: | | Architecture | โ | Dense | MoE | MoE | MoE | MoE | | Total Params | โ | 70B | 225B | 671B | 284B | 550B | | Active Params | โ | 70B | 23B | 37B | 13B | 55B | | Tokens (T) | โ | 12T | 30T+ | 15.6T | 32T+ | 20T | | World Knowledge | | | | | | | MMLU (MC acc) | 5 | 83.7 | 76.6 | 88.3 | 88.7 | 89.1 | | MMLU-Pro (CoT EM) | 5 | 61.5 | 52.2 | 57.8 | 68.3 | 79.1 | | AGIEval-En (acc) | 3โ5 | 74.4 | 66.2 | 81.9 | 83.9 | 78.7 | | GPQA Main (CoT EM) | 5 | 37.7 | 31.7 | 30.8 | 44.0 | 49.6 | | GPQA Diamond (CoT EM) | 5 | 38.4 | 27.8 | 37.4 | 44.4 | 53.0 | | C-Eval (MC acc) | 5 | 69.6 | 59.3 | 90.7 | 92.1 | 82.2 | | SuperGPQA (MC acc) | 5 | 34.2 | 24.5 | 43.1 | 46.5 | 52.9 | | SimpleQA-Verified (EM) | 25 | 10.1 | 10.8 | 17.3 | 30.1 | 27.1 | | Language & Reasoning | | | | | | | BBH (CoT EM) | 3 | 85.6 | 79.6 | 87.2 | 86.9 | 88.0 | | BBEH (EM) | 1 | 17.9 | 14.2 | 18.4 | 25.4 | 21.2 | | DROP (F1) | 1 | 77.8 | 66.8 | 81.2 | 88.6 | 87.6 | | HellaSwag (acc_norm) | 0 | 84.6 | 86.2 | 86.8 | 85.7 | 89.1 | | HellaSwag (acc_norm) | 10 | 87.2 | 88.1 | 89.4 | 88.8 | 90.5 | | ARC-Challenge (MC acc) | 25 | 95.7 | 90.8 | 95.1 | 95.8 | 97.4 | | OpenBookQA (acc_norm) | 0 | 47.4 | 48.6 | 50.4 | 48.0 | 48.6 | | PIQA (acc_norm) | 0 | 83.6 | 84.8 | 85.2 | 84.7 | 83.8 | | WinoGrande (acc) | 5 | 83.4 | 83.2 | 83.2 | 81.4 | 79.3 | | Code & Math | | | | | | | HumanEval (pass@1) | 0 | 70.9 | 83.4 | 69.8 | 69.5 | 83.8 | | MBPP Sanitized (pass@1) | 3 | 81.5 | 81.1 | 75.2 | 74.6 | 86.0 | | BigCodeBench (pass@1) | 3 | 49.2 | 50.5 | 59.9 | 56.8 | 55.6 | | GSM8K (CoT EM) | 8 | 88.5 | 80.4 | 79.6 | 90.8 | 88.1 | | GSM1K (CoT EM) | 8 | 88.4 | 79.7 | 81.8 | 83.2 | 90.2 | | MATH (flex EM) | 4 | 59.1 | 52.5 | 59.7 | 57.4 | 82.0 | | MGSM (11-lang avg) | 8 | 78.2 | 65.4 | 81.7 | 85.7 | 87.7 |
Citation
@misc{sgi2026preview,
title={SGI-Preview: Pushing the Frontier of Dense Model Efficiency},
author={SGI Team},
year={2026},
}
- Downloads last month
- -