Paper: Generalized Temporal Difference Learning Models for Supervised Learning (ICLR-2024)

Reviewer comment (Theory):
Assumption 2 seems to be a little strong to require \( f \) is invertible. In addition, how will this assumption be when the logit \( z \) is a vector? For example, when \( f \) is a softmax function?

Author response:
Please note that the Assumption 2 about invertible $f$ is not strong, as it is a common property of the inverse link functions used by generalized linear models, which can be found in Table 2.1 of McCullagh \& Nelder (1989) and the [wiki page of GLMs](https://en.wikipedia.org/wiki/Generalized_linear_model#Link_function). Note that almost all distributions corresponding to popular loss functions satisfy this assumption. 
Additionally, when the logits form a vector $\mathbf{z}$, Assumption 1 is translated into the requirement that the antiderivative $F$ (i.e., $\nabla F=f$) is a continuously-differentiable, strictly convex function. As a concrete example, when $f$ is the softmax transformation, $F$ is the log-sum-exp function. 
One can refer to Banerjee et al. (2005) for more details for the multivariate cases.

Reviewer comment (Theory):
The algorithm 1 requires the knowledgement of transition \( P \), which seems not realistic.

Author response:
Regarding the transition probability, it is designed by us, and therefore, we have complete knowledge of it. In theory, the convergence result only necessitates $P$ to be ergodic, which is a weak assumption commonly used by many previous works in TD-like algorithms' convergence. In practice, Section 4.2 and 5 demonstrate the ease of designing such a matrix, and a simple choice could be a uniformly constant matrix.

Reviewer comment (Theory):
I wonder how is lemma 1 derived? I think lemma 1 requires that the function \( f \) is \( L \)-Lipschitz continuous. It seems that this can not be implied by assumption 1 and 2.

Author response:
For Lemma 1, one can see that when the features $\phi$ and parameters $w$ are bounded, effectively the domain and image of $f$ will also be bounded. Thus there exists a constant $L=\max(L_f, L_{f^{-1}})$ that satisfies the lemma where $L_f, L_{f^{-1}}$ are the Lipschitz constants of $f$ and $f^{-1}$ for the domain/image respectively. While we didn't provide a formal proof here due to its simplicity, we will make sure to clarify this point in revision.

Reviewer comment (Theory):
I want to understand whether the transition kernel \( P \) will influence the theoretical results. I do not find any term related to \( P \) in theorems.

Author response:
Regarding the transition probability, it is designed by us, and therefore, we have complete knowledge of it. In theory, the convergence result only necessitates $P$ to be ergodic, which is a weak assumption commonly used by many previous works in TD-like algorithms' convergence. In practice, Section 4.2 and 5 demonstrate the ease of designing such a matrix, and a simple choice could be a uniformly constant matrix.

Reviewer comment (Experiments):
I believe the authors could have provided a few more datasets for the regression problem and the binary classification class imbalance problem.

Author response:
Thank you for reading our paper and the positive comments. 

Please note that we do have additional results on regression (Table 6) in the Appendix B.2. As requested, we added another classification experiment (cancer dataset in Table 4), but they do not alter any conclusion. Now we have 4 regression datasets, 3 binary classifications, 4 popular image classification tasks in total. For regression and binary classification, we studied a variety of settings including different noise settings, and transition probability matrix. 

We hope you find our experiments are well-designed, including careful study of each design choice of our algorithm, and the datasets are representative and widely recognized as popular benchmarks.