muon-gradientMatching

Qwen2.5-0.5B models pretrained from scratch on openwebtext-100k, from the Mu-GREATS subset-selection / Muon gradient-matching experiments. Each run is a subfolder; load one with:

from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "efficiency-learning/muon-gradientMatching"
sub = "full_muon_lr0.001"  # any folder below
model = AutoModelForCausalLM.from_pretrained(repo, subfolder=sub)
tokenizer = AutoTokenizer.from_pretrained(repo, subfolder=sub)

Subfolder	Optimizer	LR	Mode	Step	dtype
`full_adamw_lr0.0001`	adamw	0.0001	full	6317	?
`full_adamw_lr0.0005`	adamw	0.0005	full	6317	?
`full_adamw_lr0.001`	adamw	0.001	full	6317	?
`full_adamw_lr0.005`	adamw	0.005	full	6317	?
`full_adamw_lr5e-05`	adamw	5e-05	full	6317	?
`full_muon_lr0.0001`	muon	0.0001	full	6317	?
`full_muon_lr0.0002`	muon	0.0002	full	6317	?
`full_muon_lr0.0005`	muon	0.0005	full	6317	?
`full_muon_lr0.001`	muon	0.001	full	6317	?
`full_muon_lr0.005`	muon	0.005	full	6317	?
`full_muon_lr5e-05`	muon	5e-05	full	6317	?
`random_muon_lr0.001`	muon	0.001	random	6317	?

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for efficiency-learning/muon-gradientMatching

Base model

Qwen/Qwen2.5-0.5B

Finetuned

(708)

this model