domino14/macondo-value-nets

Macondo value nets

These are the value networks used by Macondo, a crossword board game engine (Scrabble and similar games). Given a position after a candidate move, a net predicts how likely the player who made that move is to win. Macondo's FAST_ML_BOT scores the top 50 moves by static equity with the net and plays the one with the highest predicted win chance, without searching ahead.

model	lexicon	architecture	vs HastyBot	head to head
macondo-tf (default)	NWL23 (and other English lexica)	12 layers × 256, 8 heads, 9.6M params	58.42% ± 0.18	beats the 8 × 192 net on the same games 50.64% ± 0.16
macondo-tf-csw	CSW24 (Collins)	12 layers × 256, 8 heads, 9.6M params	58.55% ± 0.18	beats the 8 × 192 net on the same games 50.37% ± 0.16
macondo-tf-selfplay	NWL23	8 layers × 192, 6 heads, 3.7M params	57.07% ± 0.18	beats its 54M-game parent 50.37% ± 0.16

HastyBot is Macondo's static-equity bot: it plays the move with the best score plus leave value. Win rates come from 100,000 game pairs, where both games of a pair use the same tile draws with the seats swapped. Ties count as half, and ± is a 95% interval.

Against a simulation

Macondo's usual strong bots choose moves by Monte Carlo simulation: for each candidate they play out many random continuations (the opponent's reply, their own next move, and so on) and pick the move that wins most often. macondo-tf was matched against such a bot that simulates the same 50 candidates two plies deep, over 5,000 game pairs in NWL23. Both bots used the same quick 2-ply search once the bag was empty.

	win rate	spread
macondo-tf vs 2-ply simulation	49.94% ± 0.78	−0.6 per game

The two are level: one forward pass per candidate plays as well as a 2-ply simulation. For scale, that simulation beats HastyBot 58.62% ± 1.61 (1,250 pairs). Deeper simulations are stronger (5 plies: 62.80% ± 1.58 vs HastyBot, also 1,250 pairs), so the nets are not a replacement for a deep sim. They are a cheap evaluator that matches a shallow one.

Files

The layout is a Triton model repository:

macondo-tf/config.pbtxt
macondo-tf/1/model.onnx
macondo-tf-csw/...
macondo-tf-selfplay/...

Each model.onnx runs as-is with ONNX Runtime. For Triton, build a TensorRT engine on your own machine from the ONNX file. Engines are specific to the GPU and TensorRT version, so none are published.

Using them with Macondo

Macondo downloads the net it needs from this repository the first time a game uses it, into data/strategy/default/models/. Each Macondo version pins one commit of this repository, so it keeps playing with the nets it was tested with.

With no GPU, Macondo runs the nets on the CPU through ONNX Runtime:

MACONDO_TRITON_USE_TRITON=false \
MACONDO_ONNXRUNTIME_LIB=/path/to/libonnxruntime.so \
  ./bin/shell

With a GPU, fetch the nets into Triton's model repository from a Macondo checkout, build the engines and set MACONDO_TRITON_USE_TRITON=true:

pytorch/fetch-models.sh     # the revision this Macondo version pins
pytorch/build-engines.sh

or take this repository's latest with hf download domino14/macondo-value-nets --local-dir data/strategy/default/models --exclude README.md.

Macondo's README has the details.

Input and output

All three nets take the same input: one row per candidate move, describing the position after that move from the mover's point of view.

• board: float32 [batch, 85, 15, 15]. These are 85 planes for the 26-letter English alphabet:
  • 26 planes, one per letter on the board;
  • 1 plane for blank tiles;
  • 26 horizontal and 26 vertical cross-check planes;
  • 4 planes for empty premium squares (2L, 3L, 2W, 3W);
  • 2 history planes: the candidate's squares and the opponent's last move.
• scalars: float32 [batch, 72]:
  • the mover's leave (blank first, divided by 7);
  • the unseen tile counts (as fractions);
  • the opponent's last move;
  • how many power tiles are unseen;
  • vowel/consonant balance;
  • turns since the opponent's last bingo;
  • the candidate's score;
  • tiles in the bag;
  • the mover's spread.

The exact layout and scaling are in Macondo's game/mllayout.go and game/mlhelper.go; the easiest way to build inputs is Macondo's own feature code.

Outputs, one value per row:

output	meaning
value	P(win) − P(loss) for the mover, in [−1, 1] (ties count as neither). Moves are ranked by this
spread	the predicted change in spread to the end of the game, as tanh(points / 130)
opp_bingo_prob	the chance that the opponent bingos on their next turn
opp_score	the opponent's predicted next score, in points

Training

Each net was trained from scratch on HastyBot self-play games, with the true game result as the label:

• Games: 81M for macondo-tf and macondo-tf-csw; 54M for the parent of macondo-tf-selfplay.
• Positions: six passes over the games, each drawing one fresh random turn from every game.
• Loss: win/draw/loss cross-entropy on the main head, plus auxiliary heads (spread, the opponent's bingo chance and next score). Four per-square "spatial" heads are used only during training.
• Optimizer: AdamW at 3e-4, with cosine decay.

macondo-tf-selfplay is the 54M-game net fine-tuned on 3M of its own games, mixed 50/50 with HastyBot games. It loses a little to HastyBot but beats its parent head to head.

The full recipe, and how to train a net for another lexicon or alphabet, is in Macondo's pytorch/TRAINING.md. Every training run and match is logged in pytorch/RUNS.md.

Limitations

• The nets judge positions in tile-placement games on a standard 15 × 15 board with English letter distributions. Other boards, alphabets or variants need a new net.
• They learned only from games between bots (HastyBot and, for selfplay, the net itself). They have been measured against HastyBot, against each other and against a 2-ply simulation (above), but not against strong humans, deeper simulations, or other engines.
• The training data includes no empty-bag positions, so the nets aren't meant for endgames. Endgames call for a search.

License

GPL-3.0, the same as Macondo.