Paper: An Information Theoretic Approach to Interaction Grounded Learning (ICLR-2024)

Reviewer comment (Experiments):
I did not find the argument for the necessity to include the regularization term particularly convincing. I would have liked to have seen stronger evidence (particularly before the regulization term was introduced) to suggest that the mode of overfitting discussed in the paper actually occurs. Notably, without the regularization term, the algorithm/optimization problem is considerably simpler.

Author response:
The reviewer asks for stronger evidence for including the regularizer $I(X,A;R_\psi)$ in Objective (4) and explanations for the "over-fitting" problem. As this has been empirically shown in our numerical results in Table 4, we would like to discuss it in greater details. Indeed, we can show that when minimizing only $I(Y;X,A|R_\psi)$: **(A1)** *any* feedback-dependent reward decoder $\psi:\mathcal{Y}\to[0,1]$ attains a small value, and **(A2)** the minimum is attained by a subset of feedback-dependent reward decoders $\psi:\mathcal{Y}\to\{0,1\}$ that assigns value 1 to "an arbitrary half of" the feedback in the data distribution, if the feedback variable $Y$ is (nearly) deterministic to the context-action $(X,A)$. Both cases would lead to the "over-fitting" problem. We totally agree with the reviewer that presenting these results before defining the regularized objective helps motivate our approach. We have included this discussion in the revised Section 4.1.  

**Detailed analysis.** To showcase these arguments, first note that minimizing $I(Y;X,A|R_\psi)$ is equivalent to
$$\min_\psi\mathcal{L}'(\psi):=(H(R_\psi|X,A)-H(R_\psi|X,A,Y))-(H(R_\psi)-H(R_\psi|Y))$$
where $H$ is the Shannon entropy. This holds by the facts that $I(Y;X,A|R_\psi)=I(Y;R_\psi|X,A)-I(Y;R_\psi)+I(Y;X,A)$ and the last term $I(Y;X,A)$ is independent of the reward decoder. Hence, the value is non-positive for any reward decoder $\psi:\mathcal{Y}\to[0,1]$ by noting that $H(R_\psi|X,A,Y)=H(R_\psi|Y)$ and $H(R_\psi|X,A)\le H(R_\psi)$. In contrast, $\mathcal{L}'$ can be positive for reward decoders that also depend on the context(-action), which justifies **(A1)**. Further, if the feedback $Y$ is (nearly) deterministic to the context-action $(X,A)$, we have that $H(R_\psi|X,A)\approx H(R_\psi|Y(X,A))\approx0$, and hence the minimum $-\log2$ is attained by any reward decoder $\psi:\mathcal{Y}\to\{0,1\}$ that assigns value 1 to "an arbitrary half of" the feedback in the data distribution (i.e., $\mathbb{E}\_{x\sim d_0,a\sim\pi_\textup{b}(\cdot|x)}[\psi(Y(x,a))]=0.5$), which justifies **(A2)**.

Reviewer comment (Theory):
Does the context distribution have to be fixed throughout training (or can it, for example, be adversarial)?

Author response:
Yes, while the results are reported for fixed context distribution throughout training, the algorithm can be also applied to non-stationary context distribution.

Reviewer comment (Theory):
I believe there is a mistake in the notation of $V(\pi)$, particularly with $(x, a)\sim d_0\otimes\pi$. This looks like $x, a$ are sampled independently, but really $a$ sampled conditionally on $x$. I believe it should be more like $x\sim d_0,a\sim\pi(\cdot\mid x)$.

Author response:
We thank the reviewer for pointing out the possible ambiguity in the notation. We have revised it to $V(\pi):=\mathbb{E}\_{x\sim d_0}\mathbb{E}_{a\sim\pi(\cdot|x)}[\mu(x,a)]$ in the paper.

Reviewer comment (Theory):
Admittedly I am not intimately familiar with MI optimization, but it is not obvious to me why we should expect minimization of the standard MI objective to "overfit" to maximize $I(Y; R_\psi)$ as you claim. I see that the objective can be decreased by increasing this term, but I see no reason why this wouldn't be balanced by a decrease of the $I(Y; X, A, R_\psi)$ term. Has this been demonstrated experimentally? If so, can you provide citations? If this paper is the first demonstration of this overfitting phenomenon, it might be nice to show those results before defining the regularized objective, it would help motivate your approach.

Author response:
The reviewer asks for stronger evidence for including the regularizer $I(X,A;R_\psi)$ in Objective (4) and explanations for the "over-fitting" problem. As this has been empirically shown in our numerical results in Table 4, we would like to discuss it in greater details. Indeed, we can show that when minimizing only $I(Y;X,A|R_\psi)$: **(A1)** *any* feedback-dependent reward decoder $\psi:\mathcal{Y}\to[0,1]$ attains a small value, and **(A2)** the minimum is attained by a subset of feedback-dependent reward decoders $\psi:\mathcal{Y}\to\{0,1\}$ that assigns value 1 to "an arbitrary half of" the feedback in the data distribution, if the feedback variable $Y$ is (nearly) deterministic to the context-action $(X,A)$. Both cases would lead to the "over-fitting" problem. We totally agree with the reviewer that presenting these results before defining the regularized objective helps motivate our approach. We have included this discussion in the revised Section 4.1.  

**Detailed analysis.** To showcase these arguments, first note that minimizing $I(Y;X,A|R_\psi)$ is equivalent to
$$\min_\psi\mathcal{L}'(\psi):=(H(R_\psi|X,A)-H(R_\psi|X,A,Y))-(H(R_\psi)-H(R_\psi|Y))$$
where $H$ is the Shannon entropy. This holds by the facts that $I(Y;X,A|R_\psi)=I(Y;R_\psi|X,A)-I(Y;R_\psi)+I(Y;X,A)$ and the last term $I(Y;X,A)$ is independent of the reward decoder. Hence, the value is non-positive for any reward decoder $\psi:\mathcal{Y}\to[0,1]$ by noting that $H(R_\psi|X,A,Y)=H(R_\psi|Y)$ and $H(R_\psi|X,A)\le H(R_\psi)$. In contrast, $\mathcal{L}'$ can be positive for reward decoders that also depend on the context(-action), which justifies **(A1)**. Further, if the feedback $Y$ is (nearly) deterministic to the context-action $(X,A)$, we have that $H(R_\psi|X,A)\approx H(R_\psi|Y(X,A))\approx0$, and hence the minimum $-\log2$ is attained by any reward decoder $\psi:\mathcal{Y}\to\{0,1\}$ that assigns value 1 to "an arbitrary half of" the feedback in the data distribution (i.e., $\mathbb{E}\_{x\sim d_0,a\sim\pi_\textup{b}(\cdot|x)}[\psi(Y(x,a))]=0.5$), which justifies **(A2)**.

Reviewer comment (Evaluation):
Why is it that the experiments with $\beta=0$ exhibited the most variance? In this case, I would expect variance to be lower, since you're optimizing over fewer neural network parameters / the optimization has one less level of nesting.

Author response:
We believe that the observation that the experiments with $\beta=0$ displayed the highest variance is closely related to the minimization of $I(Y;X,A|R_\psi)$, which we have discussed in response to the reviewer's Comment 1 above. Since any feedback-dependent reward decoder can attain a small value of $I(Y;X,A|R_\psi)$, the training process will be highly unstable and the optimized reward decoder will be constantly changing at every iteration that could lead to high variance.

Reviewer comment (Writing):
In Algorithm 1, what does "Ensure: Policy $\pi$" mean?

Author response:
We thank the reviewer for pointing out the typo in Algorithm 1. We have corrected this typo in the revision.

Reviewer comment (Novelty):
I believe the authors ignore large amounts of work on (conditional) mutual information estimation, that covers most of the theoretical derivation in the paper, e.g. [1][2].

Author response:
We note that the papers mentioned by the reviewer focus on the variational estimation of mutual information. On the other hand, our work focuses on the specific RL problem in the IGL setting. Please note that our main contribution is to propose an information theoretic approach to this RL problem, while the reviewer seems to evaluate the novelty of our estimation method for conditional mutual information. We sincerely hope that the reviewer would also consider our contributions on addressing the IGL-based RL setting in his/her evaluation of our work.

Reviewer comment (Reproducibility):
The authors do not provide the appendix, which should be a crucial issue that can lead to the rejection.

Author response:
We wish the reviewer could be more specific on what is missing in the main text. Since all our theoretical derivations and numerical results have been already included in the main text, we did not include an Appendix in the submission. According to the conference rules, submitting an Appendix is not mandatory for the submissions. We will be happy to answer any follow-up questions the reviewer might have on what he/she thinks is missing in the paper. Also, we have provided our code in the revision as a supplementary file, which can be used to verify the numerical results if necessary.

Reviewer comment (Novelty):
Can the authors discuss the relationship between the proposed estimation method and the existing work, not limited to the references I provide?

Author response:
We note that the papers mentioned by the reviewer focus on the variational estimation of mutual information. On the other hand, our work focuses on the specific RL problem in the IGL setting. Please note that our main contribution is to propose an information theoretic approach to this RL problem, while the reviewer seems to evaluate the novelty of our estimation method for conditional mutual information. We sincerely hope that the reviewer would also consider our contributions on addressing the IGL-based RL setting in his/her evaluation of our work.

Reviewer comment (Theory):
Many steps in the derivation/explanation are not explained. For example, in my opinion the quantity (without an equation number) after Eq (4) does not follow from Eq 4.

Author response:
We thank the reviewer for pointing out the typo in the quantity after Equation (4). In the revision, we corrected the typo and replace $I(X,A|R_\psi)$ with $I(X,A;R_\psi)$.

Reviewer comment (Theory):
Is Theorem 3 trivial that it does not require proof?

Author response:
We note that the derivation of Theorem 3 has been already explained in Section 4.2. Based on the reviewer's question, we have further discussed the derivation of Theorem 3 in Section 4.2 of the revised text.

Reviewer comment (Evaluation):
Why is it important to show results for a few different f-divergences?

Author response:
Different $f$-divergences have been shown to lead to different convergence and generalization behaviors [1, 2]. As a result, a proper choice of $f$-divergence could heavily depend on the target RL setting. This point is confirmed by the results in Table 2 showing that the optimal performance in different RL problems can ne attained by the algorithms based on different $f$-divergences.  

[1] Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems 29 (2016).

[2] Rubenstein, Paul, et al. "Practical and consistent estimation of f-divergences." Advances in Neural Information Processing Systems 32 (2019).

Reviewer comment (Theory):
Why were these f-divergences chosen?

Author response:
Different $f$-divergences have been shown to lead to different convergence and generalization behaviors [1, 2]. As a result, a proper choice of $f$-divergence could heavily depend on the target RL setting. This point is confirmed by the results in Table 2 showing that the optimal performance in different RL problems can ne attained by the algorithms based on different $f$-divergences.  

[1] Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems 29 (2016).

[2] Rubenstein, Paul, et al. "Practical and consistent estimation of f-divergences." Advances in Neural Information Processing Systems 32 (2019).

Reviewer comment (Evaluation):
How to choose "f-divergences"? Table 2 shows that different scenarios require different "f-divergences".

Author response:
Different $f$-divergences have been shown to lead to different convergence and generalization behaviors [1, 2]. As a result, a proper choice of $f$-divergence could heavily depend on the target RL setting. This point is confirmed by the results in Table 2 showing that the optimal performance in different RL problems can ne attained by the algorithms based on different $f$-divergences.  

[1] Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems 29 (2016).

[2] Rubenstein, Paul, et al. "Practical and consistent estimation of f-divergences." Advances in Neural Information Processing Systems 32 (2019).

Reviewer comment (Theory):
The paper does not theoretically discuss how the changes in the (4) objective can lead to worse performances. How much the KL approximation optimum can be far from the original optimal solution?

Author response:
**1. "How much the KL approximation optimum can be far from the original optimal solution?"** 

We follow a standard approach to apply variational analysis to estimating the  mutual information. In Proposition 2, if the function class $\mathcal{T}$ includes all functions, then the KL approximation is tight. As our proposed algorithm leverages neural networks and gradient-based method, we adopt the  parameterized function class. We note that lower bounding the mutual information by the variational optimization is standard [1, 2]. For a comprehensive review on this topic, please refer to the work of [3].

[1] Nowozin, Sebastian, Botond Cseke, and Ryota Tomioka. "f-gan: Training generative neural samplers using variational divergence minimization." Advances in neural information processing systems 29 (2016).

[2] Belghazi, Mohamed Ishmael, et al. "Mutual information neural estimation." International conference on machine learning. PMLR, 2018.

[3] Nguyen, XuanLong, Martin J. Wainwright, and Michael I. Jordan. "Estimating divergence functionals and the likelihood ratio by convex risk minimization." IEEE Transactions on Information Theory 56.11 (2010): 5847-5861.

Reviewer comment (Evaluation):
Why does the proposed method achieve worse results in the No noises setting compared to Xie et Al.?

Author response:
**3. Numerical results in the noiseless setting.** 

We note that the algorithm proposed by Xie et al. (2021) is designed for the noiseless IGL setting, while our proposed algorithm $f$-VI-IGL is designed to address the RL task under a noisy feedback, which is often the case in real-world scenarios. As we observed in our numerical results, while Xie et al.'s algorithm attains better numerical results in the noiseless setting, its performance can degrade significantly under a noisy observation of the feedback variable. In contrast, our proposed method managed to achieve better results in the noisy case.

Reviewer comment (Theory):
In the conclusion the authors mentioned that the algorithm is computationally expensive. Could you compare the computational complexity of the proposed method with the one of Xie et Al.?

Author response:
**4. Computational costs of the proposed RL method.**

While our proposed $f$-VI-IGL algorithm requires training of deep neural nets to estimate the mutual information terms in the objective, we observed that the training often enjoys a fast convergence. In our experiments, the training times of $f$-VI-IGL were in worst case $2\times$ more than the training time of Xie et al.'s algorithm, while achieving satisfactory numerical results in the noisy IGL environment.