Paper: Generative Modeling of Individual Behavior at Scale (ICLR-2024)

Reviewer comment (Experiments):
Even considering limited work in the chess game simulation, the empirical comparison is pretty weak. They could have considered other baselines such as ones with standard DNNs for example from one model.

Author response:
Although the set of comparisons made within our paper are limited, we believe they are sufficient to show the efficacy of our method. The models which we test against represent the state-of-the-art in their respective domains, and also in other domains. For example, we compare against individual model fine-tuning (McIlroy-Young et al., 2022) and to a Transformer-based method (McIlroy-Young et al., 2021); we achieve superior results to both, at much larger scale, while providing *generative* models. In their paper, McIlroy-Young et al. (2021) include comparisons of their Transformer encoder, which is a modified version of the Vision Transformer (ViT) architecture, to other architectures within the stylometry domain. The ViT architecture has been shown to perform at state-of-the-art levels in many other domains. Additionally, Maia is based on the Squeeze-and-Excitation Residual Network, which has been shown to perform well in image classification. Given the strength of these base models in both chess and other domains, and the fact that we compare against state-of-the-art methods that build upon them, we believe that adding weaker models would not yield significant added value to our analysis.  

However, it is possible that we are misunderstanding the comparison the reviewer had in mind; if the reviewer could clarify what they meant by standard DNN architecture, we would be happy to include this comparison in our evaluation.

Reviewer comment (Novelty):
The method looks to some extent incremental to me: some existing techniques such as Lora and Polytropon are combined and employed for a specific task.

Author response:
Thank you for your valuable feedback, and for appreciating the novel aspects of our approach. Although our work applies existing PEFT methods (e.g., Polytropon and MHR) to the domain of chess, our technical contributions go beyond this application, as we explain in our general response above. We address the remaining points you raised below.

Reviewer comment (Theory):
What is the intuition behind to model each player as a task?

Author response:
In multi-task learning, there is a trade-off between sharing information across (similar) tasks and differentiating between (dissimilar) tasks. Given any two tasks, some aspects will be similar and some will be different. We realized that the same is true of players in a game: many players share similar strategies or aspects of their playing style, but each player is still unique. Thus, we saw the potential of modeling shared characteristics across the players (e.g., through skill modules), while allowing them to express their peculiarities (e.g., through style vectors). This paradigm mapped well to recent PEFT approaches used in multi-task learning, which inspired us to make the conceptual connection between gaming and this field.

Reviewer comment (Experiments):
Is there any data imbalance / long tail problem under this setting?

Author response:
Chess player data does have some significant data imbalance, but we do not believe this is a significant issue for our method given our few-shot learning results on <= 100 games. In Table 2, for example, we show that training on completely unseen players using just 100 games is enough to identify them from a set of 10,000 other players. Additionally, Figure 2 shows that our method can model a player with as few as 50 games and achieve nearly 60% move-matching accuracy. Finally, we include a chart linked below showing the cosine similarity of style vectors generated using different sizes of disjoint subsets of games, compared to a style vector trained on 10,000 games. Observe that even with just 10 games, the cosine similarity is 0.422, which is over two standard deviations (sd=0.103) above the average pairwise cosine similarity across the population (0.188). We have added some discussion regarding data imbalance and players with lower game counts to our paper. 

[Link to Figure] https://imgur.com/a/lqt9eiZ

Reviewer comment (Experiments):
Can we have a fig5-like result for interpolation over the style space?

Author response:
This is a great question. Below is a link to the requested chart, where we produce a new player (labeled Interpolated) by averaging the style vectors of two random players from our dataset. While the interpolated player isn’t an exact average, it aligns quite well in most dimensions. We have added this example to our paper. 

[Link to Figure] https://imgur.com/a/NcxRQcI

Reviewer comment (Theory):
Can the method be applied to games with asymmetric information like poker?

Author response:
This is an interesting question to explore. In principle our approach could extend to games with asymmetric information, because aspects of a player’s strategy or playing style can still be learned from their actions. When applying our approach to visual 1-vs-1 games, asymmetry arises due to partial observability of the state space, such as a player only seeing what is in their field of view. In poker, the current state would not include the specific cards held by other players. Nevertheless, such partially observed states can still be encoded and provided as input to the base model, atop which our adapter architecture is applied (see Fig. 1). Also see our general response for further comments on generalizing our work beyond the domain of chess.

Reviewer comment (Theory):
The paper assumes that the players’ styles are stationary and independent of the context or the opponent. However, in practice, players may adapt their styles to the situation or the opponent, which could affect the accuracy and validity of the method.

Author response:
## On implicit stationarity assumptions of our approach:
 
It is true that our approach marginalizes over player opponents/contexts and a player’s evolution over time. This is a standard assumption (also made in previous work) which does not affect the validity of our method, in that it is valid to treat each player holistically and see how the method performs. In fact, we achieve strong results when viewing each player holistically: e.g., over 94% stylometry accuracy over a reference set of ~48K players. However, the reviewer is absolutely right that changing the granularity of our analysis—such as accounting for opponent style, in-game context, or a player’s temporal evolution—are possible with our method and depending on the downstream applications, could be desirable. Here are some relevant observations along these lines: 

- When excluding the opening part of the games (e.g., the first 15 moves) from our analysis, stylometry accuracy reduces, indicating that the opening has an outsized effect on style identification relative to the middlegame and endgame. This phenomenon was also observed by McIlroy-Young et al. (2021).  

- A few players in our dataset open a second account after a certain time period, creating a natural temporal split in their data. When comparing the style vectors across these time periods, we observe that they have very high cosine similarity and achieve almost identical move-matching accuracy on each other’s datasets. This is partly because most players exhibit stationary performance (in terms of Elo/playing strength) over time, and because some elements of style persist across playing strength increases. 

- One of our experiments (Fig. 4) merges the datasets of two random players—in effect forcing a non-stationarity—and compares the resulting style vector to the arithmetic mean of the individual style vectors, showing they have high cosine similarity.  

In general, we can segment our dataset in a variety of ways and readily apply our method to train a separate style vector for each segment. *If the reviewer has a particular segmentation in mind, we would be happy to implement it*. We have also added this discussion to our paper.