glossia/Rambley-V2.2-64M

~64M model pretrained on ~12.9B tokens of DCLM-Edu, UltraData-Math, and other assorted datasets, then further trained on an extra ~4B tokens of a higher quality filtered mix of real and synthetic data designed to bring out better benchmarks while not entirely destroying general language modelling capabilities.

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
arc_challenge	1	none	0	acc	↑	0.1980	±	0.0116
		none	0	acc_norm	↑	0.2474	±	0.0126
arc_easy	1	none	0	acc	↑	0.4903	±	0.0103
		none	0	acc_norm	↑	0.4390	±	0.0102
hellaswag	1	none	0	acc	↑	0.2883	±	0.0045
		none	0	acc_norm	↑	0.3061	±	0.0046
piqa	1	none	0	acc	↑	0.6289	±	0.0113
		none	0	acc_norm	↑	0.6208	±	0.0113

Architecture Details

• Natively trained at FP8
• 2.5x intermediate SwiGLU MLPs
• Headwise gated attention (headwise chosen instead of elementwise to save params for the smaller model families; Qiu et al., 2025)
• Learned attention sinks on every attention head (various, but mainly OpenAI, 2026, LLM-Core Xiaomi, 2026)
• SWA:SWA:GQA attention pattern, with the final layer forced to GQA
• 128 sliding window size (Qiao et al., 2026)
• 2048 native context length
• HoPE positional embeddings (on both SWA and GQA layers; Chen et al., 2024)
• Patch-level pretraining for 60% of the pretrained ~13B token budget (Shao et al., 2025; Peng et al., 2026)
• Optimizers:
  • Muon at 0.02 LR for matrices with Hyperball weight norm constraints (Wen et al., 2026)
  • AdamW at 0.002 LR for embeddings and vectors, norms, etc. with no weight decay or Hyperball
  • AIR LR scheduler (Yu et al., 2026)

Notes on Data

Only clusters 2, 6, 12, and 15 of ClimbMix were used; notably this excludes the mathematics cluster and most of the STEM clusters.

Code

Pretraining codebase: https://code.allura.moe/fizz/lobotomy

Other Benchmarks

====================================================================                                                                                                       
  Loading ../exported_hf...                                                                                                                                                
====================================================================                                                                                                       
/marimo/lobotomy/.venv/lib/python3.13/site-packages/transformers/configuration_utils.py:720: FutureWarning: `rope_scaling` is deprecated and will be removed in a future ve
rsion. Set `rope_parameters` instead, with `rope_type` and `rope_theta` included.                                                                                          
  warnings.warn(                                                                                                                                                           
Skipping import of cpp extensions due to incompatible torch version 2.14.1+cu130 for torchao version 0.15.0             Please see https://github.com/pytorch/ao/issues/291
9 for more info                                                                                                                                                            
Loading weights: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 236/236 [00:00<00:00, 6554.82it/s] 
/marimo/lobotomy/.venv/lib/python3.13/site-packages/transformers/configuration_utils.py:720: FutureWarning: `rope_scaling` is deprecated and will be removed in a future ve
rsion. Set `rope_parameters` instead, with `rope_type` and `rope_theta` included.                                                                                          
  warnings.warn(                                                                                                                                                           
  63,874,704 parameters (torch.bfloat16)                                                                                                                                   
  arithmark-3: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:01<00:00, 29.82it/s] 
  arithmark-3: raw 33.10% (331/1000)  normalized 33.20% (332/1000)                                                                                                         
  speed: 931.5 examples/s  (tokenize 0.24s, evaluate 1.07s)                                                                                                                
                                                                                                                                                                           
  Category                                                             N       Raw   Normalized                                                                            
  ----------------------------------------------------------------------------------------------                                                                           
  elementary_school_math_continuation::addition::grades_1_2::easy    128    23.44%       23.44%                                                                            
  elementary_school_math_continuation::comparison::grades_2_3::medium    44    15.91%       15.91%
  elementary_school_math_continuation::comparison_difference::grades_2_3::medium    48    27.08%       27.08%
  elementary_school_math_continuation::data::grades_2_3::easy         43    27.91%       27.91%
  elementary_school_math_continuation::division::grades_3_4::medium    54    29.63%       29.63%
  elementary_school_math_continuation::fractions_counting::grades_3_4::medium    50    22.00%       22.00%
  elementary_school_math_continuation::geometry_area::grades_4_5::medium    52    44.23%       46.15%
  elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium    45    48.89%       48.89%
  elementary_school_math_continuation::measurement::grades_2_3::easy    76    34.21%       34.21%
  elementary_school_math_continuation::money::grades_3_4::medium      64    25.00%       25.00%
  elementary_school_math_continuation::multiplication::grades_3_4::medium    74    48.65%       48.65%
  elementary_school_math_continuation::patterns::grades_3_4::medium    53    30.19%       30.19%
  elementary_school_math_continuation::subtraction::grades_1_2::easy   117    23.08%       23.08%
  elementary_school_math_continuation::time::grades_2_3::easy         55   100.00%      100.00%
  elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium    46    17.39%       17.39%
  elementary_school_math_continuation::two_step_addition::grades_2_3::medium    19    26.32%       26.32%
  elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium    32    25.00%       25.00%

====================================================================
  ../exported_hf (63,874,704 params) RESULTS
====================================================================
  Raw continuation accuracy        33.10%
  Length-normalized accuracy       33.20%
  Primary (acc_norm)           33.20%
====================================================================
Results saved to benchmark_cache/.._exported_hf_arithmark-3_results.json