felixfabricius/robot-chess-commentator-cnn

Robot chess commentator: square-reading CNN

Classifies what piece - if any - a chess square contains. Used to enable a Reachy Mini robot to commentate chess games.

Results

To commentate chess games, the robot needs to correctly recognise moves. The predictions for individual squares are aggregated into estimates of board positions (or equivalently, moves). Move recognition accuracy is the metric that matters.

	val	test
Move accuracy	94.6%	94.7%
Per-square accuracy	73.8%	74.6%

For comparison, the best vision-language baseline measured on the same test split (Claude Opus 5, per-square log-probabilities, low-effort thinking) reaches 21.3% move accuracy at roughly $0.20 per board. This model runs locally in about a second at no cost.

Usage

The model takes as input a 144 x 144 x 4 image (the fourth channel is a mask) of a chess square. The published dataset ships those input images directly. It has 3 heads and outputs a score for

• Square is empty or non-empty
• Piece is white or black
• 6-dimensional vector for type of piece: king, queen, rook, bishop, knight, pawn These scores can be combined into a 13-dimensional logit vector for each of the possibilities: empty square, or one of the 6 pieces in black or white.

import numpy as np, torch
from datasets import load_dataset
from safetensors.torch import load_file
from modeling import (
    SquareClassifierMultiHead, TARGET_MAP, TOP_LEFT_OHE_MAP, reconstruct_13way_logprobs,
)

model = SquareClassifierMultiHead()
model.load_state_dict(load_file("model_state_dict.safetensors"))
model.eval()

row = load_dataset("felixfabricius/robot-chess-commentator-squares", "squares", split="test")[0]
rgb = np.array(row["image"])                 # (144, 144, 3) uint8
mask = np.array(row["mask"]) // 255          # (144, 144)    uint8, {0, 1}

image = torch.from_numpy(np.dstack([rgb, mask])).permute(2, 0, 1).float()
image[:3] /= 255.0                           # RGB to [0, 1]; the mask channel stays {0, 1}

metadata = torch.zeros(1, 4)
corner = ["a8", "a1", "h1", "h8"][row["top_left_corner"]]
metadata[0, TOP_LEFT_OHE_MAP[corner]] = 1

with torch.no_grad():
    heads = (t.squeeze(0) for t in model(image[None], metadata))
    logprobs = reconstruct_13way_logprobs(*heads, log_prior=model.log_prior)

names = list(TARGET_MAP)
print(names[int(logprobs.argmax())])

Architecture

Class	SquareClassifierMultiHead
Size	1.3 MB fp32 -- 328,853 trainable parameters; 330,088 values in the state dict, the rest BatchNorm buffers and the 13-way log-prior
Input	4x144x144 (RGB + square mask) plus a 4-dim one-hot of which board corner is top-left
Output	three heads -- empty (1 logit), color (2), type (6)

A three-block convolutional trunk with a residual branch (last BatchNorm zero-initialised, so it starts as the identity) whose depthwise dilated convolution widens the receptive field to roughly seven board cells -- enough to see a tall piece leaning in from a neighbouring square.

The three heads are recombined into 13-way log-probabilities by reconstruct_13way_logprobs, under a conditional independence assumption between colour and type given non-empty. Factoring the problem this way lets every piece image teach the colour head, rather than splitting the evidence across twelve piece classes.

Note that empty_head is trained with BCEWithLogitsLoss against is_piece, so sigmoid(logit_empty) is P(piece) despite the name.

Prior correction is on, and this checkpoint expects it

reconstruct_13way_logprobs(..., log_prior=model.log_prior) subtracts the training prior, so scores reflect the evidence for each class rather than how frequent that class happened to be in training -- and empty is 56% of all squares, so the prior is far from flat.

This checkpoint was selected and benchmarked with the correction on. The snippet above steers this by passing log_prior; the repository's BoardEstimator does it with prior_correction=True.

Training data

felixfabricius/robot-chess-commentator-squares contains 23,744 labelled squares from 371 positions.

Limitations

Trained on one board, one piece set and one camera across 50 setups. It might transfer poorly to a different chess set or very different lightning.

License

Apache-2.0. Copyright 2026 Felix Fabricius. modeling.py is generated from the GPL-3.0-or-later repository and published under Apache-2.0; see the header of that file.