Paper: Designing Skill-Compatible AI: Methodologies and Frameworks in Chess (ICLR-2024)

Reviewer comment (Presentation):
The results presentation is somewhat murky, particularly in the detailed analysis of mechanisms. In general, the results would benefit from a clear statement of the outcomes of the analyses at the start of the section, followed by the existing detailed description of results, to help readers glean the overall conclusions that may get lost in details.

Author response:
We have modified Section 4.2 to state the main takeaways more explicitly, and abstract the analysis to the appendix. The main takeaways from 4.2.2 and 4.2.3 are that all agents display some tricking effects (the expectors to the greatest extent). Expectors and tree exhibit a measurable helping effect. Finally, tree and attuned agents demonstrate a distributional effect. The analysis performed in 4.2.2 is less comprehensive (it neglects helping), but is significantly simpler than that of 4.2.3. The results of 4.2.2 and 4.2.3 show the same thing, except that 4.2.2 allows us to look at the interplay between the tricking effect and the distributional effect, whereas 4.2.3 gives us a clearer look at everything in isolation.

Reviewer comment (Experiments):
The primary limitation to the impact of the results is the requirement for agents to have models of all relevant agents in the scenario. One experiment slightly relaxes this constraint and the results become substantially weaker. It would be beneficial to add remarks on this point and future directions to mitigate these weaknesses with imperfect partners.

Author response:
While we did not incorporate skill-uncertainty into our specific player compatibility experiments, we address the issue of skill-estimation in the supplement (1100 vs 1900). This does allow us to answer the question of how the results look when the amount of weakness is incorrectly estimated. Indeed, we find that seniors designed to play with stronger juniors fare better against weaker juniors than vice versa, and we suspect that overexploitation is punished whereas under-exploitation is not. This part was formerly in the appendix, and is now in the main body. We agree with you that testing att and tree is valuable on these agents, and we also understand the computational cost of the task. Accordingly, we add a previously omitted experiment where we run the other agents  on these two players. We also implement your suggestion of reducing the 4.2.3 and 4.2.2 in favor of expanding this section notably, and having it play a more important role in the paper, which is our most major implemented revision after reading all reviews.

Reviewer comment (Presentation):
Minor suggestion for Tables 1 & 2: Combine these two tables into one with a row defining the task (STT vs HB).

Author response:
We thank you for the suggestion. We merged Tables 1 and 2 and agree that this has improved readability. We have left the error ranges in the appendix as they tend to clutter the tables and detract from the actual values.

Reviewer comment (Evaluation):
If possible, include the error ranges for Tables 1 & 2 in the main text.

Author response:
We thank you for the suggestion. We merged Tables 1 and 2 and agree that this has improved readability. We have left the error ranges in the appendix as they tend to clutter the tables and detract from the actual values.

Reviewer comment (Evaluation):
Is there any intuition for why TREE does so much better than EXP in HB, given that the improvement is the opposite of what is observed in STT? Is it possible that in HB, EXP has too limited of a search depth to handle the relevant decision horizon?

Author response:
As for our speculation of why tree outperforms exp in hand and brain, we suspect that could potentially be due to the adversarial effect that tree can effect in HB. The exp agents in HB, with their horizon not extending beyond their hand’s moves, are the only agents actually able to reduce the loss of their hands. However, we expect that they will not be able to have an adversarial effect on the opponent hand, which might be quite important. That said, it is difficult to measure HB’s adversarial effect analogously because its interactions with the hands cannot be broken down into corrections and agreements due to it not outputting a full move.

Reviewer comment (Evaluation):
In Section 4.2.2, the final conclusion from the analysis was unclear.

Author response:
To clarify 4.2.2, the experiment demonstrates that exp is not dependent on a distributional effect as far as observed by the fact that its trickiness (which is high) is almost unchanged when measured on Leela’s distribution and its own distribution. For the other agents, we observe that trickiness is modulated by the distribution (the ability of focals to trick is amplified on the distribution of the focals compared to on leela's distribution), as well as a greater inherent tricking effect than leela even when standardized for distribution. The final observation was that if we had att and tree play on leela's distribution, they were less tricky than leela playing on their distributions, suggesting that their distributional effect is, in a sense, responsible for a large part (but not all) of their trickiness.

Reviewer comment (Evaluation):
In Section 4.2.2, what are the implications that the standardized and non-standardized board position distributions differ substantially for TREE and ATT, but not EXP? This seems to suggest tricking is not the main effect (as opposed to some more general distributional difference), but that was not completely clear.

Author response:
To clarify 4.2.2, the experiment demonstrates that exp is not dependent on a distributional effect as far as observed by the fact that its trickiness (which is high) is almost unchanged when measured on Leela’s distribution and its own distribution. For the other agents, we observe that trickiness is modulated by the distribution (the ability of focals to trick is amplified on the distribution of the focals compared to on leela's distribution), as well as a greater inherent tricking effect than leela even when standardized for distribution. The final observation was that if we had att and tree play on leela's distribution, they were less tricky than leela playing on their distributions, suggesting that their distributional effect is, in a sense, responsible for a large part (but not all) of their trickiness.

Reviewer comment (Presentation):
It may be better to remove the analysis in Section 4.2.2 to the appendix and provide more detail in the main text on the normalization applied in 4.2.3.

Author response:
We have modified Section 4.2 to state the main takeaways more explicitly, and abstract the analysis to the appendix. The main takeaways from 4.2.2 and 4.2.3 are that all agents display some tricking effects (the expectors to the greatest extent). Expectors and tree exhibit a measurable helping effect. Finally, tree and attuned agents demonstrate a distributional effect. The analysis performed in 4.2.2 is less comprehensive (it neglects helping), but is significantly simpler than that of 4.2.3. The results of 4.2.2 and 4.2.3 show the same thing, except that 4.2.2 allows us to look at the interplay between the tricking effect and the distributional effect, whereas 4.2.3 gives us a clearer look at everything in isolation.

Reviewer comment (Presentation):
Section 4.2 would benefit from stating the conclusions drawn on tricking, helping, and indirect effects at the start of the section. A similar comment applies to Section 4.3.

Author response:
We have modified Section 4.2 to state the main takeaways more explicitly, and abstract the analysis to the appendix. The main takeaways from 4.2.2 and 4.2.3 are that all agents display some tricking effects (the expectors to the greatest extent). Expectors and tree exhibit a measurable helping effect. Finally, tree and attuned agents demonstrate a distributional effect.

Reviewer comment (Evaluation):
It is unclear from the text which agents demonstrate tricking (and under what analysis), whereas for helping it is more clear based on the more complex analysis.

Author response:
The main takeaways from 4.2.2 and 4.2.3 are that all agents display some tricking effects (the expectors to the greatest extent). Expectors and tree exhibit a measurable helping effect. Finally, tree and attuned agents demonstrate a distributional effect. The analysis performed in 4.2.2 is less comprehensive (it neglects helping), but is significantly simpler than that of 4.2.3.

Reviewer comment (Experiments):
In Section 4.4, are there results when the MAIA agent is selected without a rating prior? This would be interesting to see to understand how well a focal can compensate when the "amount of weakness" of the partner is highly uncertain.

Author response:
While we did not incorporate skill-uncertainty into our specific player compatibility experiments, we address the issue of skill-estimation in the supplement (1100 vs 1900). This does allow us to answer the question of how the results look when the amount of weakness is incorrectly estimated. Indeed, we find that seniors designed to play with stronger juniors fare better against weaker juniors than vice versa, and we suspect that overexploitation is punished whereas under-exploitation is not. This part was formerly in the appendix, and is now in the main body.

Reviewer comment (Experiments):
One approach for Section 4.4 would be randomizing the skill of the partner agents over a reasonable range of ratings.

Author response:
While we did not incorporate skill-uncertainty into our specific player compatibility experiments, we address the issue of skill-estimation in the supplement (1100 vs 1900). This does allow us to answer the question of how the results look when the amount of weakness is incorrectly estimated. Indeed, we find that seniors designed to play with stronger juniors fare better against weaker juniors than vice versa, and we suspect that overexploitation is punished whereas under-exploitation is not.

Reviewer comment (Experiments):
Are there comparable results for using TREE in Section 4.4? While ATT is more computationally demanding to add, those results would also be interesting.

Author response:
We agree with you that testing att and tree is valuable on these agents, and we also understand the computational cost of the task. Accordingly, we add a previously omitted experiment where we run the other agents on these two players.

Reviewer comment (Experiments):
The results in Section 4.4 are considered some of the most important for the work. Less detailed analysis from Sections 4.1, 4.2, and 4.3 could be traded for further elucidation of how well the approach generalizes when given imperfect partner information.

Author response:
We also implement your suggestion of reducing the 4.2.3 and 4.2.2 in favor of expanding this section notably, and having it play a more important role in the paper, which is our most major implemented revision after reading all reviews.

Reviewer comment (Presentation):
In the supplement, the results on using MAIA1900 and stockfish are both valuable information and should be added to the main body, as they provide more context to the discussion about capabilities to partner with other agents.

Author response:
Tying to our expansion of cross-style compatibility, and as mentioned in the main response, we also promote the cross-skill compatibility and Stockfish tests to the main body of the paper. We agree that they represent an important avenue for improvement and further work.

Reviewer comment (Presentation):
Space used in Sections 4.1, 4.2, and 4.3 could be traded for these supplemental results, as the final conclusions on mechanisms remain unclear, while the conclusions about capacity to generalize are more clear.

Author response:
We also implement your suggestion of reducing the 4.2.3 and 4.2.2 in favor of expanding this section notably, and having it play a more important role in the paper, which is our most major implemented revision after reading all reviews.

Reviewer comment (Evaluation):
Why is inter-temporal coordination the right framing for evaluating agent compatibility? It would be useful to sketch specific applications where this is a natural interaction paradigm outside chess (e.g., autonomous vehicles or robotics where humans may take over at arbitrary points from an AI driver).

Author response:
While some prior works mentioned in our paper explore simultaneous settings, there are real life applications where the roles of humans and AI is temporally interleaved. STT can be likened to real life situations, such as autonomous vehicles with humans, where we have 2 agents of different abilities essentially playing the same role, with a lack of determinism concerning who is in the literal driver’s seat. In these situations, the “autopilot” must not put the car in a state where a human driver could not navigate safely, should they choose to disengage the “autopilot”. HB lends itself to asymmetrical roles contributing to a decision, with predictable roles for AI and humans. For example, automated stock screening tools are used to narrow down options prior to selection by humans. Here, we could formulate the human as an agent with a particular probability of selecting a particular stock if it is included in the suggestion. Decision-making in intertemporal symmetric and asymmetric systems is thus a mode of interaction between humans and AI agents worth investigating.

Reviewer comment (Novelty):
How correlated is the notion of "interpretable" here with prior efforts? Given that "interpretable" is a loaded term in the literature, clarification is needed on why it is a good description of the method in this case.

Author response:
We agree that the term "interpretable" is quite loaded in the literature, and indeed, the interpretability research community points to the many different application-specific motivations and formulations of interpretability as a challenge in using the concept. (This point goes back at least to classic surveys like Doshi-Velez and Kim's "Towards a rigorous science of interpretable machine learning" and Lipton's "The mythos of model interpretability: In machine learning, the concept of interpretability is both important and slippery".) As such, there have been many calls to broaden the scope of what we might view as "interpretable" behavior, so as to expand the range of applications where it might be applicable. This has led, over time, to expansions of the scope in which interpretability is used -- for example, for purposes as diverse as searching for pathways to improve a model, finding flawed assumptions in a model, or providing people being evaluated by a model a mechanism for recourse in the face of the model's decision.

Reviewer comment (Evaluation):
Prior efforts on interpretability normally aim to enable inspection of the behavior of a model, which is not obviously demonstrated here. Weak agents performing better in scenarios prepared by strong agents may not necessarily imply interpretability, but rather that they are given better initial states to use.

Author response:
So if we want to extend notions of interpretability to the current setting, it cannot be through succinct explanations. This is why we propose that a natural extension of interpretable actions to a case such as ours is to ask whether a weaker agent is able to effectively follow up on the action of a stronger agent, to it, thus motivating our formulation of interpretability in a functional, as opposed to a semantic, notion. This is more acute in the HB framework, where the brains’ actions are to simplify the action-space (as opposed to taking the game to a more favorable state as could be said about STT), and we see it reasonable to use the word interpretability if the reduced action space leads to maia producing a better move. On an auxiliary note, we see that HB focals tend to agree with maia more than leela does, and that matches the notion of interpretability in that not only is maia likely to select better moves with the new prior enforced by our focals, but also moves that "agree" more with the focals in a sense.

Reviewer comment (Presentation):
Tables 1-3 could be explained more clearly. Please consider bolding best results.

Author response:
Tables 1-3 could be explained more clearly. Please consider bolding best results.

Thank you for the suggestion. We have bolded the best results in Tables 1 and 2 (now merged for clarity) and rewrote the explanation for Table 3 for clarity.

Reviewer comment (Experiments):
Why is the EXP method N/A in table two?

Author response:
Why is the EXP method N/A in table two?

The exp agent designed for HB only selects a piece based on its anticipation of maia’s actual move, and therefore is unable to actually play chess. The other agents are adapted to HB by simply masking the second part of the move, meaning Nf6 simply gets output as N in HB, with Nf6 still technically being there. exp simply outputs N, we don’t have anyway to get Nf6.

Reviewer comment (Evaluation):
Do you have any intuition as to why EXP performs best overall? Why does the paper not claim this as the best method if it has the highest winrate?

Author response:
Do you have any intuition as to why EXP performs best overall? Why does the paper not claim this as the best method if it has the highest winrate?

Our intuitive guess as to why EXP performs well, especially in STT, is that accurately modeling the game for a few turns is more valuable than a long-term approximation in a highly noisy environment where the junior’s blunders are likely to throw the game off into different paths. The MCTS employed by other seniors refines the search assuming the opponent and the self are using a strong policy, but the presence of the weaker juniors means this behavior may not be not entirely accurate.

Reviewer comment (Presentation):
The results are explained well after reading through the experiments section many times. However, I believe that this could be clearer. Please consider using a chart that summarizes the best combinations (no score, just to guide the intuition).

Author response:
The results are explained well after reading through the experiments section many times.

In accordance with your suggestion, we have modified our results section to more clearly state our results immediately after the analysis.

Reviewer comment (Experiments):
Since you use the weaker chess engine as a subroutine in search, it isn’t clear how you could make a version with humans since they can’t communicate at test time.

Author response:
We apologize for the insufficient clarity on this point, which we have addressed in our updated manuscript. The attuned agents do not use the juniors as subroutines; they only train with them prior to testing. Thus one contribution of ours toward broader skill- and human-compatible collaboration is developing an agent that doesn’t need to communicate at test time. In response to your feedback and to that of the other reviewers, we have emphasized Section 4.4 and the points made within it to clarify the fact that our models do not need exact models of their opponents at either training or testing times. The updated section explicitly addresses your point concerning the ability of our agents to work without having an exact model of their juniors. Our results in this section show that even in an environment where we cannot communicate with the juniors and don’t have a correct model of them, generic approximations still perform well (e.g. one experiment takes expector seniors that use generic maia models as subroutines,  and they are able to beat leela when they matched with specific player models that they do not have access to).

Reviewer comment (Evaluation):
How can the methodologies be adapted for more dynamic, less rule-bound environments outside of chess?

Author response:
This is a key question for the line of work that we hope this paper helps kickstart. Similar to how AlphaZero was a separate follow-up to the already-impressive AlphaGo work, adapting our methodology for chess to more dynamic, less rule-bound environments is its own worthy goal.

Reviewer comment (Experiments):
Would the proposed techniques be effective in real-time scenarios where the lower-skilled entity is a human?

Author response:
We believe the answer is yes. Our closest approximation of this is playing versus specific human models partnered with generic seniors. In this setup, our agents do not have access to exact models of their junior partners (only to generic approximations). Although these player-specific models are not identical to individual human players, they are high-fidelity models of them, and they perform well. Real-time constraints are not an issue here, as moves are generated in the order of seconds by our agents.

Reviewer comment (Theory):
How does the proposed skill-compatible AI adapt to the continuous learning and evolving capabilities of lower-skilled agents?

Author response:
In this work we considered static agents, but as shown in section 4.4 the resulting models generalize to different agents (models of individual human chess players). As such, we expect that our methods could be adapted to work with continuously changing agents via periodic updates to the model.

Reviewer comment (Evaluation):
In real-world applications, how can the balance between raw AI performance and skill-compatibility be optimized without compromising on crucial tasks?

Author response:
This is a question that depends significantly on the domain. In chess AI performance is significantly greater than human so we are happy to trade a large amount of performance for skill-compatibility. If these methods were deployed in a real-world situation both the senior and junior agents would be required to act, e.g., the senior has an unpredictable delay between uses (an overtaxed server). In a situation like this obtaining compatibility would be of utmost importance as the junior agent might at any time have to complete the task with no support.