~350M model pretrained on ~12.9B tokens of DCLM-Edu, UltraData-Math, and other assorted datasets, then further trained on an extra ~4B tokens of a higher quality filtered mix of real and synthetic data designed to bring out better benchmarks while not entirely destroying general language modelling capabilities.

Tasks	Version	Filter	Metric		Value		Stderr
arc_challenge	1	none	acc	↑	0.2594	±	0.0128
		none	acc_norm	↑	0.2850	±	0.0132
arc_easy	1	none	acc	↑	0.5509	±	0.0102
		none	acc_norm	↑	0.4971	±	0.0103
hellaswag	1	none	acc	↑	0.3260	±	0.0047
		none	acc_norm	↑	0.3749	±	0.0048
piqa	1	none	acc	↑	0.6627	±	0.0110
		none	acc_norm	↑	0.6589	±	0.0111

Comparison

Metric	Rambley-V2.1-350M	Rambley-V2-350M-Preview	GPT-2 Medium	OPT-350m	Pythia 410M	GPT-3 Medium (closed source, reported)
Hellaswag	37.49	35.95	39.38	36.82	40.63	43.6
Arc-E	49.71	47.22	43.60	40.19	45.92	46.5
Arc-C	28.50	26.37	25.00	23.72	24.49	29.5
PIQA	65.89	64.42	66.38	64.36	67.30	70.2
Arithmark 3.0	36.10%	37.80%	33.80%	38.30%	36.90%	N/A
Training Tokens	~16B	4B	Unk. (~20B)	180B	~300B	~300B

Bold indicates the best open source model, and italics indicate the best overall model when it is closed source. Bolded italics indicate the best overall model when it is open source.

Architecture Details

Natively trained at FP8
2.5x intermediate SwiGLU MLPs
Headwise gated attention (headwise chosen instead of elementwise to save params for the smaller model families)
Learned attention sinks on every attention head
SWA:SWA:GQA attention pattern, with the final layer forced to GQA
128 sliding window size
2048 native context length
HoPE positional embeddings (on both SWA and GQA layers)
Patch-level pretraining for 60% of the pretrained ~13B token budget
Optimizers:
- Muon at 0.02 LR for matrices with Hyperball weight norm constraints
- AdamW at 0.002 LR for embeddings and vectors, norms, etc. with no weight decay or Hyperball

Notes on Data

Only clusters 2, 6, 12, and 15 of ClimbMix were used; notably this excludes the mathematics cluster and most of the STEM clusters.

Code

Pretraining codebase: https://code.allura.moe/fizz/OLMo-core

Other Benchmarks

====================================================================
  Loading glossia/Rambley-V2.1-350M...
====================================================================
Loading weights:
  348,583,480 parameters (torch.bfloat16)
  arithmark-3: raw 36.10% (361/1000)  normalized 36.10% (361/1000)
  speed: 648.4 examples/s  (tokenize 0.25s, evaluate 1.54s)

  Category                                                             N       Raw   Normalized
  ----------------------------------------------------------------------------------------------
  elementary_school_math_continuation::addition::grades_1_2::easy    128    24.22%       24.22%
  elementary_school_math_continuation::comparison::grades_2_3::medium    44    27.27%       27.27%
  elementary_school_math_continuation::comparison_difference::grades_2_3::medium    48    27.08%       27.08%
  elementary_school_math_continuation::data::grades_2_3::easy         43    32.56%       32.56%
  elementary_school_math_continuation::division::grades_3_4::medium    54    35.19%       35.19%
  elementary_school_math_continuation::fractions_counting::grades_3_4::medium    50    18.00%       20.00%
  elementary_school_math_continuation::geometry_area::grades_4_5::medium    52    51.92%       53.85%
  elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium    45    48.89%       48.89%
  elementary_school_math_continuation::measurement::grades_2_3::easy    76    32.89%       32.89%
  elementary_school_math_continuation::money::grades_3_4::medium      64    34.38%       34.38%
  elementary_school_math_continuation::multiplication::grades_3_4::medium    74    47.30%       47.30%
  elementary_school_math_continuation::patterns::grades_3_4::medium    53    22.64%       22.64%
  elementary_school_math_continuation::subtraction::grades_1_2::easy   117    34.19%       33.33%
  elementary_school_math_continuation::time::grades_2_3::easy         55   100.00%      100.00%
  elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium    46    19.57%       19.57%
  elementary_school_math_continuation::two_step_addition::grades_2_3::medium    19    26.32%       26.32%
  elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium    32    34.38%       31.25%

====================================================================
  glossia/Rambley-V2.1-350M (348,583,480 params) RESULTS
====================================================================
  Raw continuation accuracy        36.10%
  Length-normalized accuracy       36.10%
  Primary (acc_norm)           36.10%
====================================================================
Results saved to benchmark_cache/glossia_Rambley-V2.1-350M_arithmark-3_results.json