Complete Chess Games Enable LLM Become A Chess Master (2025)

Abstract: Large language models (LLM) have shown remarkable abilities in text generation, question answering, language translation, reasoning and many other tasks. It continues to advance rapidly and is becoming increasingly influential in various fields, from technology and business to education and entertainment. Despite LLM’s success in multiple areas, its ability to play abstract games, such as chess, is underexplored. Chess-playing requires the language models to output legal and reasonable moves from textual inputs. Here, we propose the Large language model ChessLLM to play full chess games. We transform the game into a textual format with the best move represented in the Forsyth-Edwards Notation. We show that by simply supervised fine-tuning, our model has achieved a professional-level Elo rating of 1788 in matches against the standard Elo-rated Stockfish when permitted to sample 10 times. We further show that data quality is important. Long-round data supervision enjoys a 350 Elo rating improvement over short-round data.

## introduction recently, large language models (llms) based on transformer architectures @xcite have demonstrated capabilities well beyond language modeling. a key milestone was the advent of chatgpt @xcite . extensive research has focused on developing efficient llm base models @xcite @xcite @xcite , including supervised models @xcite @xcite and models using reinforcement learning from human feedback (rlhf) @xcite @xcite . recent research @xcite shows that as models scale, their capabilities increase. this raises questions about llms' intelligence and learning structures. chess, an ancient game, has dialogue-like characteristics in its notational structures such as forsyth-edwards notation (fen), standard algebraic notation (san), and universal chess interface (uci). machine learning in chess has evolved to include reinforcement learning and neural networks based on supervised learning from human gameplay. developments include ai-based engines like leela chess zero (lc0) foot_0 and stockfish nnue foot_1 , which refine their algorithms through new learning. deep learning has shown the potential of ai in strategic games. the chessgpt @xcite model demonstrated the ability to choose optimal moves by learning from human language and chess data. however, models like chessgpt cannot generate the best move based on the current game state and complete an entire match. our focus is on match completeness and quality of gameplay. our contributions can be listed as follows: ## a large scale dataset of chess we introduce a large-scale dataset by collecting chess games online and generating the best moves based on stockfish's evaluations. previous research relied on portable game notation (pgn) for strategy learning, interpreting moves as actions in a markov decision process. chessgpt sees additional value in pgn data, such as elo ratings indicating player strength and annotated moves providing computer-generated evaluations. these annotations aid in value function learning, thus chessgpt retains all this information for easier strategy learning. we argue that the core of chess is making the best decision for a given forsyth-edwards notation (fen) position. human players focus on the current position rather than past moves. while chess-gpt uses historical moves, formats like pgn can be inefficient for large language models (llms) due to their expanding token length. the fen format remains constant, making it more suitable for llms. therefore, we constructed our dataset as fen-best move pairs. best move construction our best move dataset was created through a search method using stockfish. it consists of two parts: the short round dataset from chessdb foot_2 and the long round dataset from self-play endgames based on stockfish evaluations. stockfish evaluates positions using heuristic functions and an alpha-beta game tree search. we searched for valid moves from current positions, with search depths of 12-50 for short rounds and 50-200 for long rounds, limiting each search to two seconds. the highest win-rate moves were selected as the best moves. ## model the generative pre-trained transformer (gpt-3) is an autoregressive language model that generates human-like text through deep learning. it trains on casual language modeling, predicting the next word based on previous words. we trained a gpt-like model using open-llama-3b @xcite and the chess resources from section 3. unlike policy behavior data in robotics or gaming, chess state and move data can be expressed textually. this allows chess to be rendered as a text-based game, enabling imitation learning for policy through casual language modeling of the game dataset (figure 2 ). this innovative approach of applying language modeling to chess signifies a novel shift in policy learning, leveraging the game's unique aspects to develop superior gameplay tactics. ## evaluation methods chess requires a dynamic evaluation method beyond a fixed set typical of nlp tasks. we propose supplementing the evaluation set with actual games to better assess the model's capabilities. ## conclusion in this paper, we convert chess to a text game and introduce a large-scale fen-best move pair dataset. with the dataset, we propose the large language model chessllm that can play a complete chess game. considering the limitation of the evaluation set in out-of-distribution data, we propose the need to evaluate model capabilities in actual games. chessllm finally achieves an elo rating of 1788 through the sft method. in subsequent work, we will discuss how to improve chessllm by improving the data quality.