Paper: Opponent Modeling based on Sub-Goal Inference (ICLR-2024)

Reviewer comment (Theory):
In Section 3.2, this paper adopts a state-extended MDP. Therefore, for the next state, the Q-value equation should be \( Q(s_{t+1},g_{t+1}, a) \). However, the Eq. (4) writes it as \( Q(s_{t+1},g_{t}, a) \). It should be explained why \( g_t \) is the same as \( g_{t+1} \).

Author response:
We extend the state space of the MDP with state-subgoal pairs to form a state-augmented MDP $\mathcal{S}$. We assume that the next state of $s_t, g_t$ follows the same goal, so it is $s_{t+1}, g_t$. The $g_t$ is selected from $\mathcal{N}_t$. When the trajectory terminates, $g_t$ and $g_{t+1}$ reach the final state.

Reviewer comment (Theory):
In Algorithm 1, the action is sampled based on \( Q(s,\hat{g},a) \) given \( \hat{g} \) (line 9). However, when updating the Q-network, the subgoal may be \( \bar{g} \) instead of \( \hat{g} \) based on Eq. (8). That means given the same experience tuple \( (s, a, a^o, r) \), the Q-network input may be different for the sampling and learning phase, which does not conform with the canonical RL theory. What would the theoretical impacts be?

Author response:
The hyperparameter $\epsilon$ is decreased during training. The probability of $g$ taking $\hat{g}$​ increases linearly up to 50,000 episodes. It does produce better results than just using $\hat{g}$​, as shown in Section 4.5.

We do not focus on the theoretical analysis of this problem, because the non-stationary problem is complex and the inference error of opponent modeling is intractable and unavoidable.

Reviewer comment (Writing):
In Eq. (5), \( \bar{g} \) is output by a prior model \( p_{\psi} \) while in Eq. (6) and Eq. (7), it is from \( f_{\psi} \). What is the relationship between \( p_{\psi} \) and \( f_{\psi} \)? Moreover, when \( f_{\psi} \) first appears in Page 5, it outputs \( \bar{g}_t \) given \( s_t^g \). However, in Eq. (6) and Eq. (7), its input format becomes \( \mathcal{N}_t^H \). Why?

Author response:
In essence, $f_\psi$ and $p_\psi$ are the same functions. The subgoal probability uses a normal distribution, with $p_{\psi}(\bar{g}|s^g)$ describes the probability distribution of subgoals. The mapping from state $s^g$ to subgoal $\bar{g}$ is expressed as $\bar{g} = f_\psi(s^g)$. This process involves encoding the state $s^g$ into the distribution and sampling from the distribution to derive $\bar{g}$. Apologies for the confusion, we makes it clearer in the revision.

Reviewer comment (Theory):
How to compute \( card(\mathcal{S}_g) \) in Appendix A.1? Based on my understanding, if we assume every state can be transit to a goal state after a certain action sequence, \( card(\mathcal{S}_g) \) should be \( \frac{n_A^k - 1}{n_A - 1} |G| \). In addition, the sentence "Moreover, a lower \( k \) value, signifying predicted subgoals, facilitates generalization. So, it is loosely bound of \( |G| \) for OMG. Due to our method favoring the adoption of extreme values as goal states, a limited quantity of such states exist" is confusing.

Author response:
It's important to note that not every state can be transit to **every** goal, so $\frac{n_A^k-1}{n_A-1}|G|$ is not correct. Intuitive explanation for $card(\mathcal{S}_g) \leq k|G|$ is that each goal as a leaf node in a tree of depth k, have at most k ancestor state. Only each ancestor state can transition to the corresponding goal state after a certain sequence of actions. Therefore, there are at most $k|G|$ of these ancestor states.

We modified the confusing statement in the revision, as follows: "When $|G|$ is below $n_A/k$ times the number of observed states, the goal-based opponent modeling method proves more advantageous compared to the methods based on action modeling. Consequently, this criterion can be met by maintaining a relatively modest value for $k$. Due to our method favoring the adoption of extreme values as goal states, a limited quantity of such states exist. So, it is loosely bound of $|G|$ for OMG."

Reviewer comment (Reproducibility):
In Appendix A.2, this paper uses a set of identical states to acquire the action vectors of the policy in the test set. What is the detailed process of obtaining the set of identical states? Are these states sampled by a certain policy?

Author response:
These states are stored when the policies in the test set are used, just to verify that the policies in the test set are the same. The states can be collected with any policy, and this does not affect the results.

Reviewer comment (Experiments):
The reason for baseline method choice is unclear. Why not compare with some newer opponent modeling methods, e.g., [1] and [2]?  

[1] Greedy when sure and conservative when uncertain about the opponents. ICML 2022.  
[2] Model-based opponent modeling. NeurIPS 2022.

Author response:
OMG is a new exploration in the opponent modeling method, which is different from the previous opponent modeling ideas, so we choose the classic opponent modeling methods as the baselines.

[1] was not suitable for our testing environment for two reasons. Firstly, it is designed to work in a competitive environment. Secondly, it is essentially a fusion of LIAM-VAE and EXP3, where EXP3 is a bandit algorithm.

[2] is a method to deal with a constantly learning opponent. When the opponent’s policy remains unchanged, the method that degenerates to naive action of opponent modeling yields similar results to Naive OM in baselines.

Reviewer comment (Novelty):
I don't think it is a correct statement to say that most previous work on opponent modeling predicting the opponent's actions. For instance, [2] predicts a policy embedding. [3] is a meta-learning method for opponent modelling. [4] predicts which type of policy.

Author response:
We revise this statement to be more rigorous and add those related work in the revision.

Reviewer comment (Novelty):
The general architecture of OMG that uses VAE to predict something is quite similar to many previous opponent modelling methods that use VAE as well: for instance [5] and [6].

Author response:
From the level of method and algorithm idea, our method OMG has no similarity with [5] [6]. It is just that the same VAE network is used, but for the different purpose.

Firstly, [6] is the paradigm of CTDE, which is different from our setting of Autonomous agent. Although VAE is used in [6], it is still an action modeling method in essence. [5] is LIAM's previous work, which used supervised learning methods to construct mapping $<o_t, a_{t-1}, r_{t-1}, d_{t-1}>$ to $<a_t^{-1}>$ via VAE. OMG use VAE to encode future state as subgoal prior. It is worth noting that the future state is associated with the Q-value, which is novelty.

Reviewer comment (Novelty):
Very related work on opponent modeling are missing in the literature review. For instance, [2] [3] [4]. A full literature review on opponent modeling is suggested.

Author response:
We revise this statement to be more rigorous and add those related work in the revision.

Reviewer comment (Theory):
The last sentence of section 3.2: "In short, the number of (s,g) is significantly smaller than that of (s,a)". Please provide strong evidence for this, as I do think it is domain-dependent whether the number of (s,g) is larger or smaller than (s,a).

Author response:
"In short, the number of (s,g) is significantly smaller than that of (s,a)" does not always hold, depending on the method of GOAL selection. In the framework of our method, this is a reasonable conclusion. We prove our point in Appendix A.1. It is a well-known fact that no one method is suitable for all domains. Although OMG is not a panacea, its conditions are not strict and verified by experiments.

Reviewer comment (Theory):
Except for the experimental results, I am not convinced why predicting subgoals (i.e., future states) as OMG does are better than predicting something else about the opponents (e.g., actions, policy embeddings, policy categories, etc.). Could you please provide more justifications on this?

Author response:
Your examples of opponent modeling methods [1]-[6] have all been verified by experimental results to demonstrate the effectiveness of the algorithm, which is the standard paradigm in the field of opponent modeling. We conducted comprehensive experimental verification by comparing with modeling action of opponent methods (Sec 3.2), performance of training (Sec 4.3), generalization to unknown opponents (Sec 4.4), and inferred subgoal analysis(Sec 4.6). The conclusions also show that our method has certain advantages than modeling action of opponent methods. In addition, our method is intuitive, as agreed by other reviewer.

Reviewer comment (Writing):
There is confusion regarding what \( f_\psi \) and \( p_\psi \) are supposed to be. They share the same parameters, \( \psi \), which raises the question of whether they are the same function, related functions, or if the notation is accidentally overloaded.

Author response:
In essence, $f_\psi$ and $p_\psi$ are the same functions. The subgoal probability uses a normal distribution, with $p_{\psi}(\bar{g}|s^g)$ describes the probability distribution of subgoals. The mapping from state $s^g$ to subgoal $\bar{g}$ is expressed as $\bar{g} = f_\psi(s^g)$. This process involves encoding the state $s^g$ into the distribution and sampling from the distribution to derive $\bar{g}$. Apologies for the confusion, we makes it clearer in the revision.

Reviewer comment (Theory):
It is unclear how \( p_\psi(\bar{g}|s^g) \) can be pre-trained "with the prior subgoal state \( s_g \) being derived from the subgoal selector," since the subgoal selector reportedly leverages the value function. This raises the question of how the value function is already known during the pre-training phase.

Author response:
$p_\psi(\bar{g}|s^g)$ represents a pre-trained variational autoencoder.

In the pre-training phase, the value function is unnecessary. We utilize data, namely states collected from any policy, to train the encoder capability of $p_\psi$. This phase does not involve a subgoal selector, rendering the need for a value function unnecessary.

In the training phase, the parameters of $p_\psi$ is fixed, and the prior subgoal state $s^g$ derived from the subgoal selector is encoded as the subgoal $\bar{g}$.

Reviewer comment (Reproducibility):
There is uncertainty about how goals are represented to the Q-function, and whether they are encoded via \( f_\psi \).

Author response:
Yes, the subgoals encoded via $f_\psi$, and concatenate with the state as the input of Q-function.

Reviewer comment (Writing):
There is an assumption that the training update at line 15 in Algorithm 1 can only be performed using the experience at time \( t \) after \( \bar{g} \) has later been inferred at time \( t+H \). If this is correct, it should be explicitly stated, as there was initial confusion about how \( \bar{g} \) could be known when performing the update.

Author response:
There are some misunderstandings here. OMG is an episodic off-policy method and the training update phase is performed on the replay buffer $\mathcal{D}$. That is to say, lines 6-10 in Algorithm 1 is executed repeatedly to collect the trajectories. Then, lines 11-15 in Algorithm 1 represent that OMG is trained on the replay buffer $\mathcal{D}$, where the state of the trajectory at timestep $t+H$ can be get to figure out the $\bar{g}$ at timestep $t$.

Reviewer comment (Reproducibility):
The main weakness of the work is the presentation, particularly the description of the OMG framework itself. While it is implied that the parameters of the subgoal feature mapping (denoted as $\psi$) are trained using a VAE reconstruction loss on an initial batch of opponent data, the details of this process do not seem to have been provided. As the subgoal representation encoded by $\psi$ would seem to play a large role in success or failure of the algorithm, the loss which $\psi$ minimizes should be provided.

Author response:
The subgoal's prior model $p_{\psi}(\bar{g}|s^g)$ is a VAE that learns from a set of states that are collected while training opponents. The optimization objective of VAE is :

$<\hat{\omega}, \hat{\psi}> = \mathop{argmax}\limits_{\omega, \psi} \mathop{\mathbb{E}}\limits_{g \sim q_{\psi}(g|s)}\Big[\log p_{\omega}(s|g)\Big] - \mathrm{KL}\Big( q_{\psi}(g|s)||\mathcal{N}(0,1)\Big).$

where $\psi$ and $\omega$ are parameters of the encoder and the decoder, respectively. The decoder $p_{\omega}(s|g)$, also denoted by $f^{-1}_{\psi}$, is also used to reconstruct the subgoal state, as shown in Section 4.5.
We have added details about $\psi$ in the appendix of revision.

Reviewer comment (Reproducibility):
What form does the posterior model $q_{\phi}$ take.

Author response:
The distribution of subgoal prior $p_{\psi}$ and subgoal posterior probability $q_{\phi}$ are used normal distribution. The prior subgoal $\bar{g}$ and posterior subgoal prediction $\hat{g}$ are sampled from $p_{\psi}$ and $q_{\phi}$ using reparameterization trick respectively, which is used to feed in policy net. The KL term in Eq(5) calculated using $\mu$ and $\sigma$ of normal distribution $p_{\psi}$ and $q_{\phi}$.

Reviewer comment (Reproducibility):
How is the posterior goal prediction $\hat{g}$ derived from $q_{\phi}$? Is it sampled from $q_{\phi}$, or do you choose $\hat{g}$ maximizing the likelihood under $q_{\phi}$?

Author response:
The distribution of subgoal prior $p_{\psi}$ and subgoal posterior probability $q_{\phi}$ are used normal distribution. The prior subgoal $\bar{g}$ and posterior subgoal prediction $\hat{g}$ are sampled from $p_{\psi}$ and $q_{\phi}$ using reparameterization trick respectively, which is used to feed in policy net. The KL term in Eq(5) calculated using $\mu$ and $\sigma$ of normal distribution $p_{\psi}$ and $q_{\phi}$.

Reviewer comment (Reproducibility):
It isn't completely clear whether the subgoal selection process used during the training of $\phi$ and $\theta$ is the same as that used during policy execution (either equation 6 or 7).

Author response:
There is no subgoal selection process during execution phase because the state at timestep $t+k$ cannot be obtained at timestep $t$. The subgoal selection process exists only during training phase and uses the future state on the trajectories as prior about the subgoal.

Reviewer comment (Experiments):
A less significant weakness is that the experimental results could be more comprehensive. One concern is that OMG is only compared against IQL and other opponent modelling methods. It would be helpful to see how well these perform when compared against centralized multi-agent RL methods such as QMIX.

Author response:
We have added the learning curves during SMAC to the appendix of the revision. We focus on autonomous agent that use opponent modeling to assist decision making, rather than depending on a centralized critic. Therefore, we do not compare our method with centralized multi-agent RL such as QMIX, as this would be an unfair comparison. In addition, QMIX has shown good performance on SMAC.