Explore the Reasoning Capability of LLM s in the Chess Testbed (2025)

Abstract: Reasoning is a central capability of human intelligence. In recent years, with the advent of large-scale datasets, pretrained large language models have emerged with new capabilities, including reasoning. However, these models still struggle with long-term, complex reasoning tasks, such as playing chess. Based on the observation that expert chess players employ a dual approach combining long-term strategic play with short-term tactical play along with language explanation, we propose improving the reasoning capability of large language models in chess by integrating annotated strategy and tactic. Specifically, we collect a dataset named MATE, which consists of 1 million chess positions with candidate moves annotated for strategy and tactics. We finetune the LLaMA-3-8B model and compare it against state-of-the-art commercial language models in the task of selecting better chess moves. Our experiments show that our models perform better than GPT, Claude, and Gemini models. We find that language explanations can enhance the reasoning capability of large language models.

## introduction chess reasoning is challenging, requiring analytical calculation and intuitive insights. good chess players employ a dual approach, which includes (i) long-term strategy: it relies on rapid, intuitive thinking based on the pattern recognition of the chess board. (ii) short-term tactic: it involves slow, analytic calculations that typically consider 1-6 moves ahead, depending on the player's skill level. figure 1 shows an example of strategy and tactic. notably, experienced players think out loud: they develop strategic plans in clear language, and they evaluate the afterward position in lucid words after calculating the precise moves of a tactic. drawing inspiration from the thinking approach used by chess experts, we propose a method to enhance large language models' chess-playing capabilities by incorporating both strategy and tactic in language annotation. we collect the mate(move on strategy and tactics dataset), a dataset of around 1 million chess positions, and annotate the candidate moves for each position with long-term strategy and short-term tactic. then, we utilize the mate to finetune open source large language models. finally, we evaluate the performance of our models and compare them against state-of-the-art large language models. our models outperform the best commercial language model by 24.2% when both strategy and tactic are provided. in summary, this work's contributions are threefold: ## related work chess has historically been esteemed as a challenging intellectual pursuit @xcite . with all the rules and the chess board provided, it is a pure reasoning task without any uncertainty or randomness. in 1997, deep blue, created by ibm, defeated the chess world champion-russian player garry kasparov-in a match that astonished the world. modern chess engines such as stockfish, alphazero @xcite , leela chess zero, which integrate search algorithms, deep neural networks, and reinforcement learning, play significantly better than the strongest human players. recent work @xcite ) trains a transformer model on millions of annotated chess games, enabling it to play precise and beautiful chess. though chess is a "solved problem" in the field of artificial intelligence, many researchers used it as a testbed to study the capabilities of language models @xcite @xcite @xcite . large language models have demonstrated remarkable capabilities across a diverse range of tasks @xcite @xcite , and @xcite shows by instruction fine-tuning, language models can learn how to move a pawn or a piece legally. @xcite collects a dataset of chess games and chess-related corpus, then trains language models capable of effectively tracking chess board states. @xcite consider large language models as the action space pruner and the value function approximator, boosting the monte-carlo tree search algorithm for playing chess. unlike other works, our research focuses on whether strategic and tactical explanations can guide language models to find better moves. ## mate we propose the mate(move on strategy and tactic dataset) for exploring the reasoning capability of large language models in chess. in chess, mate is known as checkmate, which occurs when a king is placed in check and has no legal moves to escape. checkmating the opponent wins the game. we collect around 1 million chess positions from the open source chess server -lichess. the data collection guidelines can be found in appendix a.1. the positions are either selected from chess games or chess puzzles. these specific board positions ask players to play moves to achieve a particular goal, such as checkmating or gaining a material advantage. analyzing these positions can be an efficient method to enhance chess skills without committing to full games. we use the forsyth-edwards notation(fen) format to describe the board position. fen is a notation in one line of text with only ascii characters(appendix a.2). for each position, we select multiple reasonable moves and then annotate each move with language explanations of long-term strategy and short-term tactic by expert chess players. we use the universal chess interface(uci) format to denote the move. for a specific move, uci encodes the start and end squares of that pawn or piece. for chess strategy annotation, we categorize the future strategical plan into five kinds: (i) material count, (ii) piece activity, (iii) pawn structure, (iv) space, and (v) king safety. we ask chess experts, including world champion-level players, to formulate the rules to determine the optimal strategy for any position(appendix a.3). for each strategic category, there are approximately 20 distinct linguistic expressions to describe the corresponding plan. for chess tactic annotation, the multitude of categories is overwhelming(appendix a.4): skewer, pin, fork, x-ray, remove the defender, overload, greek gift, windmill, discovered attack, inflection, etc. for simplicity, we list the sequence of moves and provide a factual description of the resulting position. unlike search algorithms that explore long tactical reasoning chains, our approach focuses on short-term calculations, limiting the move sequence length. the move sequences are generated using the open source chess engine stockfish. we evaluate move quality using stockfish, assigning a hidden score to each move. in our dataset, we select two moves for each position whose differences in scores exceed a specified threshold. this significant score gap clearly indicates one move is superior to the other. we create four sub-dataset based on the mate: (i) mate-no-explanation: given chess positions, the candidate moves are provided without strategical nor tactical explanation; a sample is shown in figure 2 . we investigate the difficulty levels of positions for each sub-dataset and find they are at similar levels. most positions in the mate lend themselves to long-term strategic planning. while many positions are generally not very sharp, meaning there are no immediate opportunities to gain an advantage through tactical play, we can still formulate strategic plans for them. consequently, we are unable to identify short-term tactics for these positions. as a result, the mate-strategy subset is significantly larger than both the mate-tactic and mate-strategy&tactic subsets. we show the summary of the mate in figure 3 . ## conclusion we propose a method to enhance llms' chessreasoning capabilities by incorporating strategy and tactic annotations. we craft the mate, train our models and compare them against state-of-the-art commercial language models. our models outperform others in the chess-reasoning task. we find language helps language models' reasoning. we demonstrate combining long-term intuition with short-term analysis can be a promising direction for exploration.