Paper: RegQ: Convergent Q-Learning with Linear Function Approximation using Regularization (ICLR-2024)

Reviewer comment (Evaluation):
Adding regularization term to linear system to obtain/enhance solution is a common idea. I believe the real question is the performance of the resulting solution in terms of cumulative rewards or policy performance, which is ultimately what we care about. This paper only shows convergence of the algorithm to some solution, but the quality of that target solution resulting from regularized projected Bellman equation is not analyzed.

Author response:
We would like to note that an error bound from the original solution is provided in Lemma 3.2. Moreover, performance was evaluated in three different domains in Section 6, and RegQ shows better or comparable results compared to existing algorithms.

Moreover, although adding a regularization term to linear system to enhance solution is a common idea, this idea was not fully studied for switching systems to the authors' knowledge. The main technical breakthrough in this paper is that a common quadratic Lyapunov function is found for the underlying switching system model, and it is used for the convergence analysis.

Reviewer comment (Theory):
Although it's good to have a new algorithm in linear approximation Q-learning to converge to a unique solution asymptotically, the added regularization term increases the error bound between the estimated action value function and the optimal one. Based on the result in Lemma 3.2, it's not clear how the \eta parameter should be chosen. A relatively small \eta could potentially have very small denominator in the RHS, causing big error bound. Instead, if the \eta is chosen super big, the error tends to remain the same. The experiment in the Appendix also shows the larger \eta helps with the convergence rate. Does that mean we should always use a super big \eta?

Author response:
Too large $\eta$ will cause high variance in the update because the algorithm is implemented in a stochastic sense. In general, the choice of $\eta$ leads to a trade-off between the error and convergence property, and one needs to find a compromise between the two extremes, which may need further studies. In practice, we may choose decreasing $\eta$, i.e., $\eta_k = \frac{\eta_0}{k+1}$ with some positive constant $\eta_0$ to mitigate the bias coming from the regularization coefficient.

Reviewer comment (Novelty):
The added regularization + asymptotic analysis seems not requiring that much effort given the previous works on Q-learning about O.D.E. analysis, switching system, and off-policy TD-learning.

Author response:
Compared to the existing off-policy TD-learning analysis, the greedy policy of Q-learning, which is time-varying, makes the analysis much more intricate. 

Compared to the recently developed switching system-based analysis of Q-learning, adding regularization term to the system in our work requires more careful analysis. In particular, the main breakthrough in the current work is that we found a common quadratic Lyapunov function and used it for convergence analysis using a special structure of the switching system. This new finding is significant because it is in general hard to find a common quadratic Lyapunov function for switching systems. We expect that this new insight can be also used for Q-learning to accelerate its convergence speed, which can be potential future topics.

Reviewer comment (Experiments):
Based on the experiment in the Appendix, larger \eta seems to be helping increase convergence rate and reduce the estimation error. Does that mean we should always go with larger \eta value?

Author response:
As can be seen from the experiment in the Mountain car, large $\eta$ does not necessarily imply better performance. 
In general, the regularization coefficient $\eta$ provides a trade-off between the convergence property and the bias error. 
In particular, larger $\eta$ leads to (faster) convergence while increased error. On the other hand, with small $\eta$, the proposed regQ becomes more similar to the vanilla Q-learning, and hence, it can fail to converge, while the error may be reduced.  

Compared to the Mountain car, the solution of $\theta\to 2\theta$ and Baird environment are the origin. Hence, adding regularization term does not incur any bias and improves the convergence rate.

Reviewer comment (Novelty):
Seemingly, the novelty is an additional regularization term in the TD error to ensure the existence and uniqueness of the solution. It is unclear whether this new algorithm is better than vanilla Q-learning.

Author response:
We highlight several points that show that the proposed RegQ is better then vanilla Q-learning : 

 First, as for vanilla Q-learning, when linear function approximation is used together, one cannot guarantee its convergence, whereas convergence of our algorithm can be guaranteed theoretically, which is given in Theorem 5.2.

 Secondly, as for vanilla Q-learning, there are cases where fixed points of the projected Bellman equation does not exist, and the corresponding example is given in Example A.9 in the Appendix. However, as for our modified projected Bellman equation, solutions always exist, and the corresponding example is given in Lemma 3.1

 Lastly, our experiments included in the paper demonstrate that the proposed algorithm shows better or comparable results than vanilla Q-learning.

Reviewer comment (Theory):
Although convergence of the regQ is provided, the paper does not show whether one can learn the optimal state-action value function $Q^*$ well. Indeed, we do not know whether the $Q$-value associated with $\eta > 0$ converges to the optimal $Q^*$ of the MDP problem when $\eta$ goes to zero.

Author response:
The iterates of the proposed algorithm is proved to converge to the optimal Q-function with some biases which can be controlled by the regularization coefficient. Moreover, an error bound with the optimal solution is given in Lemma~3.2. We note that to remedy the bias, we can use a diminishing regularization coefficient, i.e., $\eta=\frac{\eta_0}{1+k}$ at $k$-th iteration.

Reviewer comment (Experiments):
The experiment setting cannot reflect the practical need. More comprehensive comparison with benchmarks is necessary.

Author response:
We will consider adding more comprehensive comparisons with benchmarks. However, we want to emphasize that validating on simple environments is also important as sanity check and for providing further insights.

Reviewer comment (Theory):
It is not clear whether linear function approximation makes analysis much harder. What are the technical difficulties in the convergence analysis?

Author response:
Unlike the tabular setting, the linear function approximation case does not guarantee convergence for Q-learning, which is the main difficulty. This challenge can be overcome through the switched system analysis and introduction of a regularization coefficient.

Reviewer comment (Theory):
Do you have a sample complexity analysis of regQ? For Q-learning algorithms, both asymptotic and non-asymptotic analyses are well-understood.

Author response:
In this work, we did not provide sample complexity result for regQ. However, we believe the sample complexity result can be also derived for regQ following the recent technique in the literature of non-asymptotic analysis.

Reviewer comment (Experiments):
Do the authors believe the method will be useful in practice, or that the regularization introduced will harm the quality too much? Have the authors performed more experiments to understand this?

Author response:
We believe that the method will be useful in practice because many RL algorithms are known to be unstable. Moreover, in mountain-car environment, which is still a toy example, but close to practice, the suggested algorithm shows better or comparable results to the original Q-learning. Furthermore, we can also use a diminishing regularization coefficient, i.e., $\eta=\frac{\eta_0}{1+k}$ at $k$-th iteration to mitigate the effect of the regularization term for some positive constant $\eta_0$.