Paper: Generalized Policy Iteration using Tensor Approximation for Hybrid Control (ICLR-2024)

Reviewer comment (Experiments):
How well will the algorithm perform on a somewhat higher dimensional problem, for example a 6 or 12 dimensions?

Author response:
> *Although the example on the real robot is very impressive, the examples in simulation are less so. Four-dimensional state space is not that high, barely beyond what can be represented with a look-up table on modern computers. (10^8 cells will take around 400MB of FP numbers.) The authors clearly state that their algorithm is not meant to approximate value functions on very high-dimensional state spaces, such as images, but most robotics applications on 6 degree of freedom robots have 12-dimensional state space, so this is perhaps the dimensionality of highest interest.*
> 

> *How well will the algorithm perform on a somewhat higher dimensional problem, for example, a 6 or 12 dimensions?*
> 

Indeed the dimension of the state-space in the benchmark simulation examples for hybrid control is rather low (4D). However, the benchmark simulation problems are representative of high-dimensional action space problems and it is quite large (up to 32D).  In general, in our approach what matters the most is the dimension of the state-action space. We believe the crucial aspect is that TTPI can handle high-dimensional action space which is difficult for existing ADP algorithms.

 In the software library provided, we have included the regulation problem of the point-mass system with obstacles. We have tested it for various dimensionality as it easily forms a testbed for high-dimensional state space. Here the state space is $2m$  where $m$ is the dimension of the action space (acceleration of the point-mass which is the same as the dimension of the environment it lives in). We have tested it for $m = 2, 3, 4, 5$  (so state space of size $4D$  to $10D$) and observed its performance matches the classical algorithms. So, we believe TTPI should be able to handle a relatively larger state space dimension of interest in robotics problems. We will include more examples of robotics problems in the future release of the software.

Reviewer comment (Novelty):
Did you use ideas from _opitma_tt_ (paper Chertkov et al. (2022)) which utilize _top-k_ scheme and thus more accurate?

Author response:
As mentioned in the paper, we have included the ideas from optima_tt (which is the deterministic version of TTGO). The main advantage is that the policy iteration is stable due to the deterministic nature of the policy.

Reviewer comment (Reproducibility):
What are the typical TT-ranks for double pendulum swing-up in the case of small $\Delta t$ when the model does work?

Author response:
For the cartpole swingup and double pendulum experiments, we had to use a time step lower than 1 ms. This resulted in a TT rank of the advantage function between 50 to 100 over the policy iteration stages while the value function had a rank was about 50.

Reviewer comment (Experiments):
As the authors themselves mentioned, hand-coding the dynamics in the experiments is very hard to do for complex environments.

Author response:
Regarding hand-coding the dynamics, TTPI can work with a simulator as it only requires samples of the tuple (state, action, next-state, reward) queried by TT-cross for modeling the state-value and advantage function. The main requirement from the simulator is that we can query this tuple from an arbitrary state-action pair specified by TT-cross in batch form.  However, as modern simulators such as  NVIDIA Isaac Gym are primarily geared for RL, it still requires some effort in such customization. We are currently investigating this.

Reviewer comment (Theory):
Tensor operations, especially in high-dimensional spaces, can sometimes introduce numerical instability. How does the TTPI algorithm ensure numerical stability, especially in long-horizon problems?

Author response:
In our algorithm, we specify the accuracy of approximation of the value function and advantage function in TT-cross. In our experience, an accuracy of 0.001 is usually enough to avoid any instability. The numerical inaccuracy in the forward simulation is another challenge. In general, TTPI can be robust to this as it is an ADP algorithm (so it approximates the value functions almost everywhere in the state space) and our algorithm does not require long rollouts in the Bellman update (we used rollout of only one step (about 1ms) in the algorithm).

Reviewer comment (Experiments):
My main concern of the paper is the experiment, which is quite limited and there are many remaining questions. Particularly, there are many hyperparameters, it is expected to have ablation study of showing the performance of the method versus the hyperparameters, such as accuracy $\epsilon $ of TT representation, the max TT rank $r_{max}$, discretization resolutions of the state-action spaces....

Author response:
The hyperparameters in TTPI are intuitive and we have observed them to be robust to the changes in the hyperparameters. The four main hyper-parameters are (1) the upper bound on the TT rank $r_{max}$ of the state-value function and the advantage function (2) the accuracy of tt-approximation $\epsilon$ (3) the number of samples used in TTGO in computing the policy (4) the number discretization of the state and action space. We have maintained the same choice of hyperparameters in all the experiments across different robotic tasks, so believe it is quite robust.

In general, due to the properties of TT modeling, the computation involved and memory required grows linearly with respect to the number discretization and the dimensionality of the state-action space, and quadratically w.r.t the rank. 

In our experiments, we kept the maximum rank to a large enough value (100 but almost all the experiments never reached the maximum rank in TT-cross modeling over the policy iteration procedure). The accuracy of approximation $\epsilon=0.001$ was sufficient and decreasing epsilon will improve the performance slightly but at a higher computational cost. The number of discretizations of each variable (state or action) is something that depends on the problem at hand and it is fairly straightforward in robotics problems. For example, if it is a joint angle ranging from $(-\pi, \pi)$, a discretization of about 50 to 100 suffices as we employ interpolation for continuous variables. For (3), the higher the number of samples used, the more optimal the policy computation at the cost of linear growth in computational cost and the memory requirement. We set it in the range of 10 to 100.

Reviewer comment (Reproducibility):
As a reader to implement the algorithm, I wanted to see a straightforward illustration between the running time of the algorithm and dimensions of the state/action spaces of the system, or directly give an overall complexity of the algorithm for one policy iteration.

Author response:
In general, due to the properties of TT modeling, the computation involved and memory required grows linearly with respect to the number discretization and the dimensionality of the state-action space, and quadratically w.r.t the rank.

Reviewer comment (Reproducibility):
Since the majority of the paper is about the background of TT (previous work), I think the main contribution, which is TTPI algorithm, needs more explanation, including its parallel implementation, which seems a key for applicability of the algorithm. For example, why we need a TT-round operation for the value function in line 8?

Author response:
The reason for TT-round operation is that the TT representation obtained by TT-cross need not be optimal in terms of the number of parameters used and the same tensor in TT format can be represented more compactly with the TT-round operation.

We will add more information about the implementation details in the paper.