Watching a Language Model Learning Chess (2021)

Abstract: We analyse how a transformer-based language model learns the rules of chess from text data of recorded games. We show how it is possible to investigate how the model capacity and the available number of training data influence the learning success of a language model with the help of chess-specific metrics. With these metrics, we show that more games used for training in the studied range offers significantly better results for the same training time. However, model size does not show such a clear influence. It is also interesting to observe that the usual evaluation metrics for language models, predictive accuracy and perplexity, give no indication of this here. Further examination of trained models reveals how they store information about board state in the activations of neuron groups, and how the overall sequence of previous moves influences the newly-generated moves.

## introduction language models are now used for a variety of applications that are not, or not directly, related to natural language processing tasks, and process data that is not text @xcite ; @xcite ; @xcite . @xcite used a so-called frozen pretrained transformer (fpt) to study finetuning on a variety of sequence classification tasks spanning numerical computation, vision, and protein fold prediction. in this article, we use state-of-the-art methods for language models in an area that at first glance does not seem to be an application area for them, namely the area of computer chess. this is because clear rules determine what happens here, rather than the ambiguities and vagueness that characterise language. brown et al. (2020) demonstrated, among other things, that a language model with increasing model capacity is able to learn the rules of arithmetic to a certain degree by training it with the data of crawled websites. in the process, elementary operations were learned in a certain number space, but not beyond. is this limitation due to the lack of capacity of the model, insufficient training time or training data that did not contain sufficient information? in @xcite , it was demonstrated that regardless of the number of parameters and training examples, transformer @xcite models are unable to learn addition rules that are independent of the length of the numbers seen during training. to test the ability of language models to learn rules, and assess the influence of model size, training time and available training data, we use the commonly-stressed field of "computer chess" as an example. we investigate whether a language model is able to learn the rules of chess only from the records of games played by humans. the test area is well suited for studying the training process of language models, since training data is available in large quantities here thanks to the recording of chess games on the booming internet chess servers. another reason is that the quality of the language model can not only be assessed with the usual evaluation metrics for language models such as perplexity, but also with the help of the chess rules to check whether correct games are generated. chess as an ai testing ground has also been very popular for decades, and it regained strong focus a few years ago, thanks to the work of deepmind foot_0 @xcite ; @xcite together with the game of go. there, only starting from the rules with new techniques of reinforcement learning, a chess engine was created that sur-passed everything that had previously existed in terms of playing strength. in @xcite , as a continuation of the previous work, knowledge of the rules is now not even assumed. when evaluating go, chess and shogi, without knowledge of the rules of the game, the new algorithm muzero achieved the superhuman performance of alphazero foot_1 of the earlier work, which was trained with the rules of the game. we will use a completely different approach as a starting point to create a system that discovers the rules of chess itself. only transcripts of games played will be used to train language models. we will then inspect the models to ascertain how the system has learned the rules of chess. ## formulation of the problem we formulate the problem of learning to play chess within the framework of the usual methodology for language models. a language model is made capable of writing texts by training it with data comprising natural language. it achieves this by completing given passages of text, inserting word by word or part of word by part of word that is likely to be next. in recent years, major progress has been made in this field thanks to the use of neural networks. these very powerful models are not only able to form syntactically correct sentences, but also to keep the context correct across several paragraphs, thus producing texts that are almost indistinguishable from those written by humans. this has become possible because the new model architectures called transformers @xcite ; @xcite are able to capture dependencies in the texts over long distances, and sufficient training material is also available from the www. in order to have a supervised training setting, the systems are either fed with text as input to guess the next word (casual language modelling), or words in a whole sentence are masked, which the model then has to reconstruct (masked language modelling). the best-known representative of the first type is the family of gpt models @xcite ; @xcite and for that of the second type @xcite and its many relatives. we will use the gpt2 model in different model sizes as a basis and train them with chess data. this data is often in the so-called portable game notation (pgn) format foot_2 . these are text files containing some metadata, such as the names of the players, the date, the elo rating foot_3 and more, and the transcription of the actual game in standard algebraic notation foot_4 . this part is a string that can be seen like a sentence of a natural language. the individual moves form the "words" of the sentence. example: " d4 d5 nf3 nf6 e3 bf5 nh4 bg6 nxg6 hxg6 nd2 e6 bd3 bd6 e4 dxe4 nxe4 rxh2 ke2 adding a new word to the san string is equivalent to making a chess move. the context that a language model has in the form of the preceding words for prediction contains all of the information needed to generate the state of the chessboard. accordingly in principle it should be possible to create a model that predicts all legal moves of a position with positive probability and all illegal moves with probability near 0. ## data and pre-processing for training the language model, we need a large amount of game data containing legal moves. we can download these from the internet chess server lichess foot_5 , for example. all games played on the server since 2013 are offered there, grouped by month. a compressed pgn file is available for each month. overall, over 400 gb of compressed data with over 1.7 billion games played. this is a sufficiently large amount of data, even if the amount becomes much smaller after preprocessing (e.g. removing metadata). considering that the original gpt2 language model was trained with 40gb of internet text, we have sufficient room to experiment with different amounts of data during training. the quality of the games played plays a subordinate role in this research, as it is initially only about learning the rules. however, it would be possible to filter games via the metadata of the elo values of the players and examine the influence on the playing strength. however, in order to avoid games that were abandoned early on, sometimes after only one move on the server, we will use a minimum length 1371 for filtering. the pre-processing of the batch of data is undertaken with a command line programme for manipulating the pgn files, which masters almost all of the required steps. pgnextract 7 can perform the required transformations in a reasonable time even with large amounts of data. in these cleansings, all move numbers, results, comments, variations, etc. are removed from the games to obtain only the pure string with the san notation. one line per game is written to a file. all games with fewer than 20 moves are also filtered. ## related work a very similar approach was followed in @xcite . with slightly different pre-processing, a model also based on gpt2 was trained from game data (11,000 games and 2.19 million games). after 30,000 training steps, plausible looking games could be generated, but with about 10 percent illegal moves in the games. it was also clearly observed that fewer errors occurred in the early stages of the game, which is obvious due to the lower context required. in our experiments, we will investigate positions after different number of moves from the starting position. jhamtani et al. ( 2018 ) use text generation not to generate the games themselves, but rather to generate comments in text form. a specially created data set and an lstm @xcite )-based neural network are used for training. in our work, as in @xcite ; henighan et al. (2020) we will also investigate the relationship between the development of test loss in language models as a function of model size, computing capacity and data volume. in their work, power laws were found to be observed very precisely over seven orders of magnitude, showing that the result benefits from a scale up as long as the sizes are increased simultaneously and there is no bottleneck at one. if the model sizes in terms of the number of parameters are increased by eight times, the amount of data has to be increased by eight times. we will investigate the relationships with chess-specific metrics, such as the number of correctly-generated moves. before we start fine-tuning the model with the games data, we would like to test whether the gpt2 models of different sizes trained with english language files are inherently capable of continuing 7 https://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/ chess games. since the training texts comprising web crawls certainly contained chess games in algebraic notation, this could be possible. sequences generated with the gpt2 small, for example, if started with "e4 d5", look like this: "e4 d5 e5 e8 8 f8) e5 e3 f8 e9 f8 f8) 9 0-0-0-0-0-". while this looks a little like a game of chess, but they are hardly correct moves. games generated with the medium gpt2 model look like this, for example: e4 d5 18. f6 nc5 19. nf3 nd4 20. rg1 nd6 this looks more like chess notation, but it still does not contain correct moves. this example also shows a problem that the model has with the notation, given that games are published very differently on the web and therefore also in the gpt2 training data. they are partly with numbering and partly without, which is also reflected in the example. the large model and the xl model do not give better results. we suspect that this is due to insufficient and inconsistent lot data in the training data set. a larger model capacity does not bring any progress here. ## training and evaluation of the models the hardware used for the training was a kubernetes cluster @xcite with nvidia rtx 3090 gpus, each with 24 gb of video ram and 256 gb of main memory. for the implementation, the transformers package @xcite al. (2020) the learning rate search was conducted according to smith (2018). we start fine-tuning the model as the learning rate increases from very low to very high, and stop when the loss starts to truly become out of control. fastai howard and gugger (2020) was used for the training, using the 1-cylce-policy smith and topin (2019). 9 with batch sizes just fitting on the gpu memory, the models were saved after some epochs of training for the evaluations. when language models generate sequences of words, the same sequence will always emerge if the word with the highest probability is always chosen next. furthermore, the models often tend to repeat sequences of words. this also applies to the 1372 generation of chess games here. therefore, random mechanisms such as top-k sampling @xcite and top-p sampling @xcite are used to generate the games. these techniques reduce the tendency of repetition, although it can still occur, as @xcite have investigated. to evaluate the models, games are generated in different ways: the second method presents more of a challenge, increasing as the length of the given number of moves increases. since the game data set used for the test is not included in the training data, as the length of the given moves increases, increasingly more positions will appear that the model has never seen before. therefore, the model has to learn the rules to generate valid moves. the third metric uses starting positions generated by a random sequence of moves. a large proportion of these moves have therefore never appeared in human games, nor in the test data set. furthermore, the move patterns that appear are very different from those in human games, as well as from those in conventional chess programs. it is therefore very difficult for the model to generate regular moves for these sequences. even for humans, handling such random positions is very difficult. @xcite has found in experiments with chess grandmasters and amateurs that while good chess players can easily remember typical positions, they have problems with random positions. for each trained model, the training loss, validation loss, accuracy predicting the moves in the data and perplexity are also calculated. ## results for encoding, we use byte pair encoding, and therefore a typical chess game of 50 moves from both sides requires about 200 tokens for encoding the whole game. however, a game can be much longer. we use a maximum sequence length of 256 and cut of the rest of the moves. we trained different model sizes of gpt2 (small, medium, large) with different numbers of games @xcite to investigate the influence of the two factors on the learning process. to assess the results, the models were each subjected to an evaluation after a few epochs, using the evaluation metrics described in the previous section. appendix a-1 shows how the predictive accuracy of the language model evolves with the number of gpu training hours. the small amount of training data leads to a strong increase in accuracy for all three model sizes after only a few days of training. with more data, no model shows this increase. the different models seem to learn at about the same rate, with the small model being slightly slower. alternatively, if we look at perplexity as an evaluation measure, the same picture emerges. all models with a small training data set lead to a faster drop in perplexity, which indicates a better prediction of the language model. all other combinations of model size and amount of data seem to perform the same. we now want to investigate whether the models with little data are able to learn the chess rules faster and whether it truly makes no difference with the other combinations. with the chess-specific metrics, we can get to the root of this, and look at the performance of the models as a function of training time. from five typical opening positions after one move by both sides, five games were generated with the models with top-p sampling (@xmath0.92), and then it was checked how many moves were correct until the first incorrect move was made. the average of these 25 games was calculated. appendix a-2 shows how this performance for the respective models developed with the training time. top-p sampling is used to check the models' ability to produce not only the most likely move, but also other valid moves. in a chess position, more than one move is usually possible. for each combination of models and data set, the result was plotted over the training time and a logarithmic fit was drawn. no data are available for the gpt2 large model with the larger amount of data because hardly any models could be trained in the available time period due to the long training time per epoch. in contrast to the accuracy, this measure shows that the models with a small amount of data perform significantly worse. the best values are delivered by the medium model with medium data volume. this is in sync with the observation in @xcite that model size and data volume should be increased together for good results. the second metric is a more demanding task, as the games are not generated from a typical starting position after one move, but rather from positions after ten moves, taken at random from games played. these games were not from the training data set. games are generated from 100 random positions using top-p sampling (@xmath1.92), and then tested to see how many moves were correct from this position, whereby the mean value is then calculated. appendix a-3 shows how this performance for the respective models developed with the training time. again, a logarithmic fit was drawn. for this task, all models benefit from more training data, although there are no differences in model size. as a third metric, we have chosen a task that is even more challenging, and it should make it clearer whether the task was solved based on learned rules rather than pure memorisation of variants. again, starting with top-p sampling (@xmath2.92), 100 games were generated from one position after ten random moves, and then checked to see how many moves were correct from this position, whereby the mean value was then calculated. appendix a-4 again shows how this performance for the respective models developed with the training time. it can be seen that this task is much more difficult, as the models now only manage to correctly execute sequences of a few moves on average. however, the same effect can be seen as with the two previous metrics, namely that all models bene-fit from a larger amount of data. in terms of model size, there are few differences. so far, we have explored how language models benefit from more parameters in the model and more training data when learning chess rules. now we want to ascertain whether there are patterns in how the information of the chess rules is stored in the parameters of the models. we will use different visualisations for this purpose. for this purpose, different approaches help to visualise language models. on the one hand, we look at the influence of different inputs on the generated words/moves, as shown in arrieta et al. (2020); @xcite , and on the other hand, we can look at the activation of the different neurons in the models, as shown in @xcite ; @xcite ; @xcite . when properly visualised and studied, neuron activations can reveal the roles played by individual neurons and groups of neurons. we use the ecco library foot_8 alammar (2021) for analysis. in order to combine the groups of neurons involved in the same tasks, factorisation methods for matrices are used, whereby the library used employs nmf for this purpose. let us look at the inner workings of a trained model for moves from some sample positions. the first position is from the opening phase and it is a special situation where only one legal move is possible (fig. 1 ). if we look at the influence of the individual parts of the sequence in fig. 2 , the moves that led to position fig. 1 , on the new move, we see that the last parts have the strongest influence, but otherwise the entire sequence also influences the output. the colour code shows the strength of the influence, and alternatively we can also show the influence of the individual parts as a percentage (fig. 3 ). how certain is the model that the generated move is a correct one? for this purpose, we look at the probabilities that the model assigns to the individual possible tokens at the end of the last layer of the generator part. the move is generated in two parts, and it shows a high probability of the generated move for both parts, but still far from 100 percent for the first part (fig. 4 ). let us now look at the activations of the neurons and try to see how the model stores information about the chessboard from the move sequences. in order to generate valid moves, the language model needs a representation of the chessboard and its pieces in the states of the neurons. if we look at the activation of the neurons by group, we see that one group (red in fig. 5 ) is active on the parts of the text responsible for the row information on the chessboard, and another for the column and piece information (blue in fig. 5 ). the other two are active at the beginning or end of the sequence. let us now look at another position, which comes from the so-called "game of the century" foot_9 . in the complicated position from the middle game in fig. 6 with many possible moves, the model reaches its limits. it recognises that a piece on the d-file was captured last and wants to capture it back with rxd or qxd. however, there is no valid move for this in this position. the representation on the board was not correctly mapped in the neurons here, so no valid moves are generated. the probability for possible next tokens in the last layer of the network is also distributed over many candidates (fig. 7 ). as in position 1, virtually the entire sequence of previous moves has an influence on the new move to be generated, as can be seen in fig. 8 . in this example, again the specialisation of groups of neurons on the row information and the figure and column information can be seen. however, the activations are lower at the beginning of the sequence. ## conclusion and future work in the two example positions, it can be seen that the whole sequence has an influence on the generated move, which is necessary to generate correct moves. by looking at the activation of neurons, we could see that the information about the row, column and type of figure is stored in different groups of neurons. thus, the model seems to organise the storage of the information necessary to represent the state of the board. so the training could benefit from a longer string representation of the games such as the long algebraic notation since the mechanics of moves are more directly accessible. because rows, columns, and pieces are all represented as separate tokens. in order to study the learning process for larger models and larger data sets we would like to use distributed training with falcon (2019) with higher computational capacity, and the optimizations proposed in @xcite al. (2020); rasley et al. (2020); zhang and he (2020) we have only investigated the learning of the rules of the game here. accordingly, investigating the possible playing strength of the language models as a function of the training data quality would be an interesting extension of the investigations. the influence of the selection of the training data according to the playing strength of the players involved and the model size would have to be considered here.