Understanding Game-Playing Agents with Natural Language Annotations (2022)

Abstract: We present a new dataset containing 10K human-annotated games of Go and show how these natural language annotations can be used as a tool for model interpretability. Given a board state and its associated comment, our approach uses linear probing to predict mentions of domain-specific terms (e.g., ko, atari) from the intermediate state representations of game-playing agents like AlphaGo Zero. We find these game concepts are nontrivially encoded in two distinct policy networks, one trained via imitation learning and another trained via reinforcement learning. Furthermore, mentions of domain-specific terms are most easily predicted from the later layers of both models, suggesting that these policy networks encode high-level abstractions similar to those used in the natural language annotations.

## introduction go is fundamentally a game of pattern recognition: from ladders and walls to sente and shape, professional players rely on a rich set of concepts to communicate about structures on the game board. some patterns are relatively simple: walls are lines of adjacent stones, and an atari is a threat to capture stones on the next move; other patterns are less clearly defined: hane refers to any move that "goes around" the opponent's stones, and sente describes a general state of influence or tempo. despite the nebulous definitions of some of these terms, human players use them productively. beginners learn about eyes that determine when groups of stones are alive or dead and are given guidelines for when they should play a cut or extend a ladder; more advanced players learn sequences of joseki and tesuji and are taught to distinguish good shape from bad shape. figures 1-2 depict some example concepts. computers have recently surpassed human performance at go @xcite , but relatively little is known about why these programs perform so well and whether they rely on similar representational units to choose the moves they play. while post-hoc behavioral analyses suggest that alphago and its successor alphago zero @xcite can process complex game situations involving shape, capturing races, sente, tesuji, and even ladders, existing interpretability work has focused on the moves that agents play, rather than the internal computations responsible for those moves. our work instead proposes a structural analysis by correlating the internal representations of gameplaying agents with information from a naturallyoccurring dataset of move-by-move annotations. in this paper, we use linear probing to explore how domain-specific concepts are represented by given a board state and its associated comment, we produce binary feature vectors summarizing which game phenomena (e.g., ko, atari) are mentioned in the comment and use pattern-based feature extractors to determine which phenomena are actually present on the board ( §2.2). we then feed board states into two policy networks with disparate architectures and training methods ( §3.1) to obtain intermediate representations. finally, we use linear probes ( §3.2) to predict the binary feature vectors from our policy networks. generally, we find that patternbased features are encoded in the early layers of policy networks, while natural language features are most easily extracted from the later layers of both models. we release our code and data at https://github.com/andrehe02/go . ## results we find that domain-specific keywords are significantly more predictable than control words, with @xmath0.8 × 10 -5 under the wilcoxon signed-rank test. as shown in figure 3 (left) and consistent with our knowledge that patternbased features can be obtained by applying simple rules to the raw board state, we find that patternbased features are encoded in early layers of both models, as shown in figure 3 (right). meanwhile, keyword-based features are most easily extracted from later layers, suggesting that they correlate with high-level abstractions in the policy network. generally, pattern-based features are much more predictable than keyword-based features, with average roc auc values of (0.96, 0.98) and (0.68, 0.70), respectively. as discussed in section 2.2, this discrepancy can largely be attributed to the noisiness inherent in natural language data. 5 related work @xcite propose a similarly-sized dataset of move-by-move chess commentary. rather than using this commentary for model interpretability, though, @xcite attempt to predict whole comments from raw board states. @xcite use the same dataset to jointly train a policy network and language generation model with a shared neural encoder, but again focus on the pedagogical application of commentary generation rather than interpretation of the policy network. similar work has focused on generating sportscasts in the robocup domain @xcite @xcite . our primary methodology is linear probing @xcite , which has commonly been used to study the intermediate representations of language models like elmo @xcite and bert @xcite . one classic result in this area shows that early layers of contextual language models correlate best with lexical-syntactic information such as part of speech, while later layers correlate with semantic information like proto-roles and coreference @xcite . recent work on control tasks @xcite , minimum description length @xcite , and pareto probing @xcite has focused on improving the methodological rigor of this paradigm. although linear probing is fundamentally a correlational method, other recent work has focused on whether information which is easily extractable from intermediate layers of a deep network is causally used during inference @xcite . most related to our work are contemporary studies by @xcite ; however, to the best of our knowledge, our work is the first of its kind to use features derived from natural language in conjunction with probing techniques for policy interpretability. ## conclusion we presented a new dataset of move-by-move annotations for the game of go and showed how it can be used to interpret game-playing agents via linear probes. we observed large differences in the predictability of pattern-based features, which are extracted from the board state, and keywordbased features, which are extracted from comments. in particular, pattern-based features were easily extracted from lower layers of the policy networks we studied, while keyword-based features were most predictable from later layers. at a high level, this finding reinforces the intuition that written annotations describe high-level, abstract patterns that cannot easily be described by a rule-based approach. accordingly, we argue there is much to learn from this annotation data: future work might attempt to correlate policy network representations with richer representations of language, such as those provided by a large language model. future work might also explore whether similar approaches could be used to improve game-playing agents, either by exposing their weaknesses or providing an auxiliary training signal. we also expect similar approaches may be viable in other reinforcement learning domains with existing natural language data.