Paper: Perceptual Metrics for Video Game Playstyle Similarity and Diversity (ICLR-2024)

Reviewer comment (Experiments):
What other baselines could be used for comparison in these games? The empirical evaluations focus on ablations of the existing metric, but do not compare against any other metrics. This makes it difficult to assess how the current approach fares compared to alternative ways of conceptualizing playstyle measurement.

Author response:
* > What other baselines could be used for comparison in these games?
    * Within the datasets provided by Playstyle Distance, it is challenging to identify more comparable baselines due to the lack of playstyle labels in the training datasets. Additionally, the dataset for TORCS and the Atari games contains only one playstyle, which makes contrastive learning impractical.
    * > The empirical evaluations focus on ablations of the existing metric, but do not compare against any other metrics. This makes it difficult to assess how the current approach fares compared to alternative ways of conceptualizing playstyle measurement.
        * Regarding the empirical evaluations focusing on ablations of the existing metric and not comparing against other metrics, we note that a possible unsupervised clustering method mentioned in Playstyle Distance shows less than 30% accuracy for a 5-class classification (compared to over 80% accuracy for Playstyle Distance). Playstyle Distance itself is claimed as the first metric of its kind for video game playstyles in its abstract, which is why we focused on improving this baseline.

Reviewer comment (Evaluation):
Can multiple seeds or variations of the method be run to quantify uncertainty in the performance?

Author response:
* > Table 1:
    * > Can multiple seeds or variations of the method be run to quantify uncertainty in the performance? The text claims multiscale features ("mix") are superior. Looking at RGSK this result is not clear: the     result for t=1 is quite close to mix for t=1. For TORCS the mix results are more clearly better (modulo the lack of any estimate of uncertainty).
        * Our experiments, including those using HSD, were conducted with three different HSD encoders, as provided in the official release of Playstyle Distance on PapersWithCode. Below is a detailed table with results from these three HSD encoders, including the corresponding standard deviations. If more detailed standard deviation data for each accuracy in the 100-round random subsampling is needed, we can conduct further experiments, though this will require additional time.

| - | $1$ | $2^{20}, t=2$ | $2^{20}, t=1$ | $256^{res}, t=2$ | $256^{res}, t=1$ | $\textbf{mix}, t=2$ | $\textbf{mix}, t=1$ |
| -------- | -------- |  -------- |  -------- |  -------- |  -------- |  -------- |  -------- | 
| TORCS | 35.16 | 66.52/69.40/76.12 (std=4.02) | 56.76/63.32/62.40 (std=2.90) | 6.36/3.72/3.72 (std=1.24) | 49.40/62.84/70.04 (std=8.55) | 73.28/71.40/79.64 (std=3.53) | 72.88/76.68/75.60 (std=1.60) |
| RGSK | 80.14 | 79.92/78.71/76.54 (std=1.40) | 92.67/97.42/90.2 (std=3.00) | 5.58/5.08/6.21 (std=0.46) | 25.50/20.67/33.58 (std=5.33) | 91.75/83.33/90.46 (std=3.70) | 95.54/91.88/95.21 (std=1.65) |

Reviewer comment (Evaluation):
The text claims multiscale features ("mix") are superior. Looking at RGSK this result is not clear: the $2^{20}$ result for t=1 is quite close to mix for t=1. For TORCS the mix results are more clearly better (modulo the lack of any estimate of uncertainty).

Author response:
> From the results in the new table it looks like the 95% confidence intervals overlap for t=1 and mix. So it does seem like the improvements may be quite limited.

The original accuracy in RGSK at $2^{20},t=1$ is already quite high, but it is sensitive to "t" without a hierarchical state. In the mix (hierarchical state) configuration, it is evident that neither $t=2$ nor $t=1$ significantly affects the prediction. This consistency provides clear evidence that our proposed method, which utilizes a hierarchical state, is superior and more general.

Reviewer comment (Experiments):
Better results on 1 of 2 cases evaluated (TORCS) is not very strong evidence for superiority of the method. Perhaps this would be stronger with evidence from Atari as well?

Author response:
* > Better results on 1 of 2 cases evaluated (TORCS) is not very strong evidence for superiority of the method. Perhaps this would be stronger with evidence from Atari as well?        
    * The result shows that the baseline is sensitive to hyperparameters, and the hierarchical state version appears more stable in different scenarios.

Reviewer comment (Evaluation):
The results on probabilistic similarity again show clear results for TORCS, but more ambiguous outcomes for RGSK. Lacking any confidence intervals, the results in RGSK look very close to one another for all methods.

Author response:
* > Figure 3:
    * > The results on probabilistic similarity again show clear results for TORCS, but more ambiguous outcomes for RGSK. Lacking any confidence intervals, the results in RGSK look very close to one another for all methods. Is it possible the dataset is the root cause of these issues? TORCS is from AI agents, which may be easier to classify in style than humans (though it's not obvious a priori why this would be true).
        * Regarding the results on probabilistic similarity, it's true that the improvement on RGSK is minor. In TORCS, each playstyle is closely controlled, with slight variations in driving target speed and noise, necessitating a more precise playstyle metric.

Reviewer comment (Experiments):
Is it possible the dataset is the root cause of these issues? TORCS is from AI agents, which may be easier to classify in style than humans (though it's not obvious a priori why this would be true).

Author response:
* > Figure 3:
    * > The results on probabilistic similarity again show clear results for TORCS, but more ambiguous outcomes for RGSK. Lacking any confidence intervals, the results in RGSK look very close to one another for all methods. Is it possible the dataset is the root cause of these issues? TORCS is from AI agents, which may be easier to classify in style than humans (though it's not obvious a priori why this would be true).
        * Regarding the results on probabilistic similarity, it's true that the improvement on RGSK is minor. In TORCS, each playstyle is closely controlled, with slight variations in driving target speed and noise, necessitating a more precise playstyle metric.

Reviewer comment (Novelty):
The theoretical contribution is marginal, including the connection to video game play. The concept of "Playstyle Distance" itself does not seem to be specific to game play—it is a straightforward distance between policies using encoded state/action pairs [while "straightforward" is a feature (and not a bug) here, it raises the question as to why this paper needs to be framed in this language at all].

Author response:
* Our distance metric still relies on the Wasserstein distance for measuring the effort required to change playstyles. The Bhattacharyya distance, as a variant, aligns with our proposed probability mapping function derived from the Weber-Fechner law (as detailed in Section A.1).

Reviewer comment (Novelty):
As "Playstyle Distance" and related concepts are previous work, the additional theoretical contributions are minimal. Incorporating the Bhattacharyya distance is fine, but the connections made to human psychology are a bit flimsy. Additionally, the "Playstyle Similarity" metric is ad-hoc (which isn't inherently bad); it's also not immediately clear to me (or at least it wasn't argued) if it's a metric, as the product of two metrics isn't a metric in general.

Author response:
* Playstyle Similarity is indeed a metric, incorporating a conditional consideration of intersection over union. In Section 5.3, we explain that without the Jaccard index, the metric focuses only on intersection states, ignoring samples outside this intersection set. With only the Jaccard index, action distribution is not measured. Thus, it's not merely a product of two metrics, but a combination that leverages all observed state-action pairs, including:
        * Playstyle Intersection Similarity: A metric of action distribution similarity in the intersection set.
        * Playstyle Jaccard Index: A metric of all observed states.
        * Playstyle Similarity: A comprehensive metric of all observed state-action pairs.

Reviewer comment (Experiments):
Given the above points, the experimental section is not substantive enough. While the authors compare Playstyle Distance against their various improvements, they use classification accuracy to compare. However, taking a step back—this is now just a supervised learning problem. Why are any of these needed at all? If the actual task is playstyle classification, then algorithms need to be compared against supervised learning approaches.

Author response:
* We use a classification task with a similarity metric to validate these metrics. In these cases, it can be assumed that all supervised learning approaches would exhibit random-level accuracy since there are no playstyle labels in the training dataset. Additionally, contrastive learning is impractical for the TORCS and Atari games datasets as they contain only one playstyle, making it unsuitable for such learning.

Reviewer comment (Reproducibility):
While I appreciate the fact that code was provided, there weren't good instructions in the supplementary material zip file. Clicking the PapersWithCode link in the paper then brought me to a page that clearly has the author names and affiliations listed.

Author response:
* If you encounter any issues with running our code in the Supplementary Material, we are available to assist. The PapersWithCode link in Section 4 directs to the official dataset and codes of "Playstyle Distance," which serves as our baseline. Our implementation is a clone from their Git repository, and we have ensured not to reveal our names and affiliations.

Reviewer comment (Theory):
Is Playstyle Similarity a metric?

Author response:
* Yes, Playstyle Similarity is a metric comprising several aspects:
        * Playstyle Intersection Similarity: Measures action distribution similarity in the intersection set.
        * Playstyle Jaccard Index: Measures similarity across all observed states.
        * Playstyle Similarity: Encompasses all observed state-action pairs for a comprehensive metric.

Reviewer comment (Experiments):
It is worth noting that Lin et al. (2021) were able to distinguish intersection states even with unprocessed screen pixels in Atari games. - Seems to state "In two different datasets, Lin et al. (2021) did find pixel-identical screens"?

Author response:
In Atari games, pixel-identical consecutive screens can exist in two game trajectories due to the simplicity of the game environment and low randomness. In common DRL applications, we often find almost no repeated visited states due to high-dimensional observations.

Reviewer comment (Writing):
In a different scenario, when treating each state as equivalent, we can invariably pinpoint an intersection state. - Relates to identical, not just equivalent?

Author response:
When discretized states are considered, they can be treated as identical. However, when a state has a corresponding action, we do not eliminate the same state-action pair for constructing the sampling action distribution.

Reviewer comment (Novelty):
Unclear what exactly is new in Sec. 3.1 and what was introduced as part of HDS (Lin et al. (2021)).

Author response:
Lin et al. (2021) used a single discrete state mapping and a sample count threshold to discard states with unstable action distributions. They did not use the hierarchical design in HSD for Playstyle Distance as it prefers stable action distributions and not very large discrete state spaces. Our new method in Section 3.1 enables the simultaneous use of different discrete state mappings.

Reviewer comment (Evaluation):
While distance is a common metric for determining similarity, a larger distance value conveys primarily that two entities are different, without giving much insight into the degree of their similarity - Why doesn't a larger distance relate to a smaller degree of similarity?

Author response:
As the distance increases, the number of potential candidates with the same distance also increases significantly. In a 2D space, these candidates form a circle around a reference point; as the distance increases, the perimeter of the circle increases. Therefore, we focus more on the similar parts for a more definitive prediction.

Reviewer comment (Theory):
Drawing from the concept of similarity, we can infer that a smaller distance provides more definitive information about the similarity - Needs to be explained.

Author response:
As the distance increases, the number of potential candidates with the same distance also increases significantly. In a 2D space, these candidates form a circle around a reference point; as the distance increases, the perimeter of the circle increases. Therefore, we focus more on the similar parts for a more definitive prediction.

Reviewer comment (Writing):
General wording: "intersecting states/samples" suggest a partial equivalence of a single state/sample. E.g. "intersecting set of states/samples" could be more appropriate. This is especially relevant, because the HSD model can be used to define an intersection over (single) states (based on the hierarchy), and therefore the concepts are not clearly distinguishable.

Author response:
We can edit this part for a more precise description as long as the page limit is not exceeded in the camera reday version.

Reviewer comment (Writing):
Sec 4.2.: "Space size" seems not to be introduced - probably number of discrete states?

Author response:
The term "Space size" refers to the maximum count of different states in the discrete representation. We will use the term "state space" for more precise descriptions.

Reviewer comment (Experiments):
To evaluate the efficacy of the proposed multiscale state space and to compare it fairly with Playstyle Distance, we primarily focus on the TORCS and RGSK platforms - Needs to be explained.

Author response:
We validate Playstyle Distance with solely multiscale state space on TORCS and RGSK to make sure multiscale state space is effective. For Atari games, single state space version already has a high accuracy with 512 sample size (over 90% accuracy). With the page limit, we move this part with curves to appendix (Figure 11 and 12).

Reviewer comment (Experiments):
In this section, we perform a comprehensive evaluation of various metrics, including leveraging full data with union operations. - Seems to state, that state samples from all games are used as a single set?

Author response:
In Sections 5.1 and 5.2, our focus was on the observed state in the intersecting set of two datasets. In Section 5.3, we expanded this to include all observed states. The introduction of the Jaccard index allows us to define action distribution similarity even outside the intersecting set of samples.

Reviewer comment (Experiments):
This is based on the assumption that variations in game content can be interpreted as different states - This statement was not empirically evaluated?

Author response:
This assumption was validated using the Atari Console, employing a discrete encoder from other games.

Reviewer comment (Evaluation):
Why was Playstyle Similarity not evaluated in Sec. 5.2?

Author response:
In Section 5.2, our focus remained on states within the intersection. The concept of Playstyle Similarity, which includes the Jaccard index, is discussed in Section 5.3, as it encompasses all observed samples.

Reviewer comment (Experiments):
Most importantly, additional comparison baselines from related work should be added. E.g. clustering or supervised methods should be applicable.

Author response:
Common methods like the Fréchet Inception Distance (FID) use latent feature distribution distances, which have been deemed impractical for classification in TORCS (as shown in Playstyle Distance). Supervised or contrastive learning baselines are challenging due to the absence of playstyle labels in training datasets or the presence of only one playstyle in the dataset.

Reviewer comment (Evaluation):
The evaluation is only performed via classification, but a similarity metric should preserve a distance relation beyond "Top-1". Therefore, a ranking/continuous study should be performed. E.g. by creating or ordering existing playstyles along a continuous spectrum (like passive/aggressive driving) and evaluating the correlation.

Author response:
We plan to use the TORCS dataset for an experiment evaluating continuous spectrum playstyles (5 target speeds from 60 to 80, and 5 levels of noise). This experiment will be conducted during the rebuttal period.

Reviewer comment (Presentation):
Sec 5.2 a, b has nearly indistinguishable intervals. A tabular presentation or non-linear graph scaling should be used to improve clarity.

Author response:
We will include tables in the appendix to provide a clearer presentation of the results from Figure 3(a)(b).

Reviewer comment (Experiments):
I find it challenging to grasp the exact task the authors aim to address, even though I recognize it's based on prior work. The paper's objective seems to be playstyle identification based on a few gameplay samples by comparing them with reference datasets, but the experiments are more about player identification. While understanding the need for reference datasets for different playstyles, the creation and existence of these references remain ambiguous. A discussion on this would be beneficial.

Author response:
* The objective of this paper is to define a similarity metric. The experiments for validating this metric involve identification/classification tasks and the application of diversity measurement in DRL agents. The reference datasets used in the experiments serve as examples of targeted playstyles. For instance, to measure an unknown playstyle, we can compare it against known playstyles and identify the most similar one. This scenario is akin to a zero-shot classification task, where no classification task occurs during the training of discrete representations.
* For new games without existing reference sets, it's possible to first collect some trajectories and label them manually (giving meaning to the playstyle) or use a similarity threshold to create reference datasets.

Reviewer comment (Theory):
The assumption is that the more the distributions of actions are different, the further the playstyles are. But is it really what defines playstyle? What about identifying playstyles that are only different in very few states (for instance, two chess players that are using two different openings). Since the definition of playstyle provided here is not grounded on any concrete application, it is difficult to understand the relevance of the work.

Author response:
* Regarding concerns about very few intersection states (as in chess), this highlights why discrete state representation is important for determining comparable cases. If the playstyle (decision-making style) is indeed different, it should show action distribution differences in those intersection states (like opening or initial board). If the playstyle is very similar yet leads to different states, the result is either influenced by environmental randomness or decisions made by other players, which are not under the control of the player we are measuring. Otherwise, the same action in the same state should lead to the same next state, making it unlikely to have very few intersection states.

Reviewer comment (Theory):
The approach hinges on discretizing the state space. While states are assumed to be continuous and actions discrete, it is unclear how this would apply when actions are also continuous, as seen in many games.

Author response:
* Regarding the concern about continuous action spaces, TORCS, as described in Section 4.1, is a game with a continuous action space.

Reviewer comment (Theory):
The method's efficacy seems tied to the state discretization's capability to reflect genuine state distances. This might work for pixel-based games, but it is uncertain how it would apply to more structured observations, like chess.

Author response:
* Regarding concerns about very few intersection states (as in chess), this highlights why discrete state representation is important for determining comparable cases. If the playstyle (decision-making style) is indeed different, it should show action distribution differences in those intersection states (like opening or initial board). If the playstyle is very similar yet leads to different states, the result is either influenced by environmental randomness or decisions made by other players, which are not under the control of the player we are measuring. Otherwise, the same action in the same state should lead to the same next state, making it unlikely to have very few intersection states.

Reviewer comment (Reproducibility):
The multiscale approach's specifics, including the number of mapping functions and their selection, are unclear, yet these details likely influence the outcome significantly.

Author response:
* The multiscale approach is rooted in human cognition and involves several levels of granularity. In Table 1 and Section A.2, we demonstrate that each state mapping function contributes some information to playstyle measurement (accuracy > $\frac{1}{\text{number of playstyles}}$). The combined version further improves accuracy (exceeding the use of just one state mapping). As this research is not focused on training discrete states, we directly utilize the hierarchical states from $\textit{Playstyle Distance}$ and also test with downsampling discretization. Our experiments show that more state mappings lead to more accurate predictions. These mappings can be obtained by training more discrete encoders or leveraging HSD (simultaneous training of several discrete encoders with customized granularity).

Reviewer comment (Experiments):
The approach is an unsupervised method and is evaluated on very few use-cases, making it difficult to understand if the proposed approach is generally effective or if the choices made by the authors have been overfitted to the three single use cases presented.

Author response:
* Regarding the concern about generality, our metric is built on general decision-making and tested on rule-based agents, human players, and DRL agents. Our experiments are performed on video games, and we claim applicability to video games due to the straightforward scenario of demonstrating multiple playstyles. Should open benchmarks for decision-making styles become available, we believe our approach would be suitable for claiming broader generality. These decision-making problems could include robot control, natural language processing, and others, as described in the introduction, where DRL is a solution for general decision-making.

Reviewer comment (Experiments):
The paper lacks a real dataset captured from real video games, with complex states, actions, and many more players than what is in the article.

Author response:
* As for the realism of the games used in our experiments, TORCS and Atari games are common in DRL research. RGSK is a racing game available on the Unity Asset Store (https://assetstore.unity.com/packages/templates/systems/racing-game-starter-kit-22615), designed for human players.

Reviewer comment (Experiments):
What are the real use-cases that the identification of playstyle is targeting? Why is it an unsupervised problem and not a supervised one? How are the playstyle reference datasets built and is this realistic?

Author response:
* > What are the real use-cases that the identification of playstyle is targeting? 
    * Our metrics are designed to concretely define the similarity and diversity of decision-making styles (playstyles). They are useful in validating the diversity of AI policies, assessing whether imitation learning fits a given playing dataset, and for human behavior modeling. For instance, they can detect the similarity of a Go player's playstyle to that of the superhuman AlphaZero AI to prevent cheating. Playstyle metrics are also useful in recommendation systems for identifying potential customers with similar or new playstyles.
* > Why is it an unsupervised problem and not a supervised one ? 
    * The metric and discrete representation are unsupervised, meaning no playstyle labels are used in training. This approach is akin to zero-shot classification, with no learning how to classify during the discrete representation learning phase. States are used for identifying comparable states for action distribution comparison, which implies playstyle information. 
    * According to Appendix D of the HSD paper (Playstyle Distance), it is impractical to perform supervised learning on training the discrete representation or playstyle classification, as either the datasets are sampled from different sources (TORCS and RGSK) or there is only one playstyle (Atari games). 
    * Besides, as described in Section 2.1, playstyle classification can be trained by supervised learning, but it relies on predefined labels and cannot handle playstyles not in the training data (the datasets we used cannot be solved by supervised learning).
* > How the playstyle references dataset are built and is it realistic?
    * The datasets used in playstyle analysis consist of gameplay collected by rule-based agents, human players, and DRL agents. For performing few-sample analysis, we use random subsampling without replacement, as shown in Figure 2(d). 
    * As for the realism of the games used in our experiments, TORCS and Atari games are common in DRL research. RGSK is a racing game available on the Unity Asset Store (https://assetstore.unity.com/packages/templates/systems/racing-game-starter-kit-22615), designed for human players.

Reviewer comment (Experiments):
What is the effect of the state discretization technique that is used? How do you tune the multiscale approach? Since you are only using few datasets, the way you tune it may be overfitting the datasets, isn't it?

Author response:
* > What is the effect of the state discretization technique that is used?
    * The state discretization technique identifies comparable states for comparing action distributions. Without state discretization, it is challenging to measure the action distribution distance directly from playing datasets (observation-action pairs). 
    * In Section A.2, our experiments show that the state discretization technique we used (HSD) performs better than downsampling. The learned discrete representation focuses on features more related to gameplay rather than arbitrary features.
* > How do you tune the multiscale approach ? Since you are only using few datasets, the way you tune it may be overfitting the datasets, isn't it?
    * We do not tune the multiscale approach, as it does not require any hyperparameters for our proposed metric. The elements that can be tuned are the discrete state mappings. However, the scope of this paper is not about training new discrete representations; therefore, we directly use the trained models released by the baseline method (Playstyle Distance).

Reviewer comment (Theory):
How do you deal with continuous actions since your work focuses on discrete action spaces?

Author response:
* > How do you deal with continuous actions since your work focuses on discrete action spaces?
    * Our metrics can handle both discrete and continuous action spaces, with the difference being the adoption of different types of probability distribution distance metrics. Specifically, TORCS has a continuous action space, while RGSK and Atari games have discrete action spaces, as described in Section 4.1.