glossia/Rambley-V2-64M

~64M (active) model trained on ~17.1B tokens of Fineweb-Edu, DCLM-Edu, and UltraData-Math.

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
arc_challenge	1	none	0	acc	↑	0.1988	±	0.0117
		none	0	acc_norm	↑	0.2381	±	0.0124
arc_easy	1	none	0	acc	↑	0.4566	±	0.0102
		none	0	acc_norm	↑	0.4242	±	0.0101
hellaswag	1	none	0	acc	↑	0.2806	±	0.0045
		none	0	acc_norm	↑	0.3060	±	0.0046
piqa	1	none	0	acc	↑	0.6034	±	0.0114
		none	0	acc_norm	↑	0.6039	±	0.0114

Architecture Details

• 2.5x intermediate SwiGLU MLPs
• Headwise gated attention (headwise chosen instead of elementwise to save params for the smaller model families)
• Learned attention sinks on every attention head
• SWA:SWA:GQA attention pattern, with the final layer forced to GQA
• 512 sliding window size
• 2048 native context length
• HoPE positional embeddings (on both SWA and GQA layers)
• One MTP head for auxiliary CE loss at 0.3x multiplier (currently unused at inference)
• Optimizers:
  • Muon at 0.02 LR for matrices with Hyperball weight norm constraints
  • AdamW at 0.002 LR for embeddings and vectors, norms, etc. with no weight decay or Hyperball

Training Data

Full mix available here, but it's around 17B tokens of mostly fineweb-edu and dclm-edu with some ultradata-math mixed in

Code

Pretraining codebase: https://code.allura.moe/fizz/OLMo-core

Other Benchmarks

====================================================================                                                                                                       
  Loading ../exported_hf...                                                                                                                                                
====================================================================                                                                                                       
Loading weights:
  67,369,432 parameters (torch.bfloat16)
  arithmark-3: raw 33.90% (339/1000)  normalized 34.30% (343/1000)                                                                                                         
  speed: 819.5 examples/s  (tokenize 0.24s, evaluate 1.22s)                                                                                                                

  Category                                                             N       Raw   Normalized
  ----------------------------------------------------------------------------------------------
  elementary_school_math_continuation::addition::grades_1_2::easy    128    21.88%       21.88%
  elementary_school_math_continuation::comparison::grades_2_3::medium    44    20.45%       20.45%
  elementary_school_math_continuation::comparison_difference::grades_2_3::medium    48    14.58%       14.58%
  elementary_school_math_continuation::data::grades_2_3::easy         43    32.56%       32.56%
  elementary_school_math_continuation::division::grades_3_4::medium    54    35.19%       37.04%
  elementary_school_math_continuation::fractions_counting::grades_3_4::medium    50    18.00%       20.00%
  elementary_school_math_continuation::geometry_area::grades_4_5::medium    52    61.54%       61.54%
  elementary_school_math_continuation::geometry_perimeter::grades_4_5::medium    45    55.56%       55.56%
  elementary_school_math_continuation::measurement::grades_2_3::easy    76    39.47%       39.47%
  elementary_school_math_continuation::money::grades_3_4::medium      64    28.12%       29.69%
  elementary_school_math_continuation::multiplication::grades_3_4::medium    74    40.54%       40.54%
  elementary_school_math_continuation::patterns::grades_3_4::medium    53    26.42%       26.42%
  elementary_school_math_continuation::subtraction::grades_1_2::easy   117    19.66%       21.37%
  elementary_school_math_continuation::time::grades_2_3::easy         55   100.00%      100.00%
  elementary_school_math_continuation::two_step_add_subtract::grades_2_3::medium    46    26.09%       26.09%
  elementary_school_math_continuation::two_step_addition::grades_2_3::medium    19    21.05%       21.05%
  elementary_school_math_continuation::two_step_subtraction::grades_2_3::medium    32    31.25%       28.12%

====================================================================
  ../exported_hf (67,369,432 params) RESULTS
====================================================================
  Raw continuation accuracy        33.90%
  Length-normalized accuracy       34.30%
  Primary (acc_norm)           34.30%
====================================================================