Chess-SFT-Models

SFT (thinking) chess models from the compute-allocation study. One subfolder per model, named model_{size}_{pretraining_tokens} (tokens in billions, 2 s.f.). Each subfolder has its own model card with a pass@1 / pass@16 figure (score vs pretraining tokens, pretraining base vs SFT-thinking).

Loading

Models ship a custom tokenizer (tokenizer.py), so trust_remote_code=True is required. The remote-code loader ignores subfolder=, so snapshot the model folder locally first and load from that path:

from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "model_50m_17B"   # see the table below
path = snapshot_download("pavelslab-nyu/Chess-SFT-Models", allow_patterns=f"{name}/*") + f"/{name}"
model = AutoModelForCausalLM.from_pretrained(path, trust_remote_code=True)
tok   = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

Generating a rollout

generate.py at the repo root ships a default prompt (a test_B0 position):

python generate.py --model model_50m_17B          # uses the default prompt
python generate.py --model model_680m_6.4B --prompt "1. d4 Nf6 2. c4 g6 <T>"

Note: append the <T> (think) token to the end of your prompt — it's the cue for the model to start its rollout. A prompt without a trailing <T> will not generate as intended.

Models & evaluation

pass@k on the multi-turn chess benchmark (test_B0 family), SFT-thinking model.

model	size	pretrain tokens	pass@1	pass@16
model_20m_0.53B	20m	0.53B	3.0%	21.4%
model_20m_1.6B	20m	1.6B	4.5%	29.6%
model_20m_2.6B	20m	2.6B	4.8%	31.7%
model_20m_5.3B	20m	5.3B	7.0%	37.3%
model_20m_11B	20m	11B	7.9%	39.3%
model_20m_16B	20m	16B	9.1%	42.4%
model_20m_21B	20m	21B	9.8%	44.2%
model_20m_32B	20m	32B	10.1%	44.4%
model_20m_40B	20m	40B	10.0%	46.4%
model_20m_53B	20m	53B	10.6%	49.8%
model_50m_0.23B	50m	0.23B	1.2%	9.4%
model_50m_0.69B	50m	0.69B	7.6%	37.4%
model_50m_1.1B	50m	1.1B	10.5%	44.6%
model_50m_2.3B	50m	2.3B	14.4%	52.5%
model_50m_4.6B	50m	4.6B	18.4%	60.9%
model_50m_9.2B	50m	9.2B	18.9%	63.1%
model_50m_17B	50m	17B	20.6%	66.8%
model_50m_23B	50m	23B	20.1%	64.9%
model_50m_41B	50m	41B	21.7%	68.3%
model_100m_0.53B	100m	0.53B	8.4%	40.1%
model_100m_2.1B	100m	2.1B	20.9%	60.8%
model_100m_4.3B	100m	4.3B	21.5%	65.1%
model_100m_8.0B	100m	8.0B	23.5%	67.7%
model_200m_0.27B	200m	0.27B	2.5%	18.5%
model_200m_0.53B	200m	0.53B	16.8%	56.0%
model_200m_1.1B	200m	1.1B	17.8%	56.5%
model_200m_2.1B	200m	2.1B	24.6%	65.4%
model_200m_4.0B	200m	4.0B	29.5%	73.3%
model_200m_5.3B	200m	5.3B	30.3%	73.9%
model_200m_11B	200m	11B	33.0%	75.9%
model_200m_21B	200m	21B	35.1%	76.5%
model_200m_40B	200m	40B	35.9%	80.4%
model_200m_53B	200m	53B	36.5%	81.0%
model_680m_0.080B	680m	0.080B	0.8%	7.1%
model_680m_0.32B	680m	0.32B	7.8%	35.5%
model_680m_0.64B	680m	0.64B	18.1%	55.2%
model_680m_1.2B	680m	1.2B	26.6%	66.9%
model_680m_1.6B	680m	1.6B	28.8%	68.0%
model_680m_3.2B	680m	3.2B	32.4%	71.9%
model_680m_6.4B	680m	6.4B	36.7%	76.8%
model_680m_12B	680m	12B	40.3%	78.6%
model_680m_16B	680m	16B	40.3%	79.6%
model_680m_24B	680m	24B	41.5%	78.8%
model_680m_32B	680m	32B	44.2%	80.4%
model_680m_48B	680m	48B	43.4%	81.4%

Citation

If you use the models, please cite it:

@article{pre2post-chess,
  title   = {Understanding Reasoning from Pretraining to Post-Training},
  author  = {Shen, Jingyan and Li, Ang and Rahman, Salman and Sun, Yifan and
             Goldblum, Micah and Telgarsky, Matus and Izmailov, Pavel},
  journal = {arXiv preprint arXiv:2607.16097},
  year    = {2026},
  url     = {https://arxiv.org/pdf/2607.16097}
}