SN38 Round 12 — all CPT checkpoints
Continue-pretrain of anacoluthe89/chrono-2020 @633eb832 on kepton0117/sn38-train-2021.
On-chain pin is not this repo. UID 141 uses kepton0117/chrono-2021-sn38 (step-4000 only).
Each revision is one optimizer checkpoint (1e-3 stage, init = 1e-4 step-4000):
| Revision | Cosine vs UID 131 | Loss ~ |
|---|---|---|
step-2000 |
0.9538 (fail copy) | 2.8 |
step-4000 |
0.9294 | 2.66 |
step-6000 |
0.9144 | — |
step-8000 |
0.9080 | 2.54–2.61 |
step-10000 |
0.9061 | — |
step-12000 |
0.9055 | — |
step-14000 |
0.9055 | — |
step-15000 |
0.9055 | 2.50 |
from transformers import AutoModelForCausalLM
m = AutoModelForCausalLM.from_pretrained(
"kepton0117/chrono-2021-ckpts", revision="step-8000", trust_remote_code=False,
)