chess-transformer

Four GPT-style transformer checkpoints trained to predict chess moves from UCI-tokenized move sequences. These are the checkpoints behind a controlled study of why chess language models produce illegal moves, and what actually fixes it.

Full writeup, training code, and evaluation scripts: github.com/JOTELLECHEA/chess-transformer

Checkpoints

Names encode corpus size and architecture: 1.2m_L12E384H6 is 1.2M games, 12 layers, 384 embedding dimensions, 6 attention heads.

Checkpoint	Architecture	Params	Final Val Loss	Legal-move rate	Fully-legal games
98k_L6E256H4	6L / 256E / 4H	~5M	2.2593	97.3%	4.4%
1.2m_L6E256H4	6L / 256E / 4H	~5M	2.1035	97.7%	11.4%
98k_L12E384H6	12L / 384E / 6H	~22M	2.7826	97.8%	9.8%
1.2m_L12E384H6	12L / 384E / 6H	~22M	1.7015	99.1%	51.8%

"Fully-legal games" is the share of generated games (n=500) in which every single move was legal. Scaling data alone takes it from 4.4% to 11.4%; scaling capacity alone reaches 9.8%. Doing both reaches 51.8% — roughly 12x baseline, and 4.5-5.3x what either intervention achieves on its own. Capacity and data unlock each other rather than contributing separable gains.

1.2m_L12E384H6 is the flagship. The other three exist for the comparison.

Loading

These are not transformers-compatible. The text-generation tag describes the mechanism (autoregressive next-token prediction), not API compatibility — AutoModel and pipeline() will not work. Loading requires the GPT, GPTConfig, and MoveTokenizer classes from the GitHub repo.

from huggingface_hub import hf_hub_download
from safetensors.torch import load_model

from src.config import GPTConfig
from src.model import GPT
from src.dataset import MoveTokenizer

REPO_ID = "Jotellechea/chess-transformer"
CHECKPOINT = "1.2m_L12E384H6"

weights_path = hf_hub_download(REPO_ID, f"{CHECKPOINT}/model.safetensors")
config_path  = hf_hub_download(REPO_ID, f"{CHECKPOINT}/config.json")
vocab_path   = hf_hub_download(REPO_ID, f"{CHECKPOINT}/vocab.json")

config = GPTConfig.load(config_path)
tokenizer = MoveTokenizer.from_vocab_file(vocab_path)
config.vocab_size = tokenizer.vocab_size

model = GPT(config)
load_model(model, weights_path)
model.eval()

Use load_model() rather than load_file() — the embedding and output-projection weights are tied, and load_model() reconstructs that tie on load.

Tokenization

Word-level, where one token is one complete UCI move (g1f3), not a sequence of characters. The vocabulary is closed-form: 1,968 theoretically possible UCI moves computed from the rules of chess, plus 5 special tokens (<|SOM|> and four result tokens), for 1,973 total. It is identical across all four checkpoints, so token IDs are directly comparable between them.

Unlike a BPE tokenizer, there is no fallback for unencodable input — text that is not a legal UCI move simply cannot be tokenized.

Because UCI encodes only from-square and to-square, the model is never told which piece is moving. It has to infer that from move history alone, which is what makes "does it track board state internally" a real question rather than something handed to it for free.

Training data

Lichess open database, released under CC0. Games were kept if at least one player was GM-titled, across all time controls except correspondence, then converted from SAN to UCI.

• 98k corpus: 2023-01, ~98k games after filtering
• 1.2m corpus: 2023-01 plus 2025-01 through 2026-05, ~1.2M games after filtering

Both architectures were trained for 5 epochs with block_size=192, which covers 99.7% of real games in the corpus.

Interpretability

Linear probes on the flagship checkpoint's internal activations show board state is linearly decodable, peaking at layer 9 of 12 at 84.6% of the gap to perfect (measured against a random-init baseline), then declining through the final two layers. The 6-layer model is still climbing at its last layer, with layer-over-layer improvement collapsed to +0.5 points.

This follows the methodology of Li et al. (OthelloGPT), Nanda et al., and Karvonen.

Limitations

• Legal is not strong. Against Stockfish, the flagship wins ~3.8% of games at skill 0 and effectively never at higher skill levels. High legal-move rates say nothing about playing strength.
• The data comparison is confounded by time period. The 98k corpus is from 2023-01 while the additional 1.2m data spans 2025-2026, so that arm varies population alongside scale. The capacity comparison holds corpus fixed and is unaffected.
• Color is uncontrolled. All evaluation had the model playing White.

License

MIT. Training data is CC0 via the Lichess open database.