Paper: Achieving Sample and Computational Efficient Reinforcement Learning by Action Space Reduction via Grouping (ICLR-2024)

Reviewer comment (Novelty):
Since this work focuses on MDPs with large action spaces that exhibit group-wise similarity, so it's application could be narrow.

Author response:
We respectfully disagree with the statement that the MDPs that exhibit group-wise similarity are narrow.  Our action group method is relevant in numerous practical settings. For instance, actions that correspond to proximate directions can be aggregated into a group in autonomous driving. Another example is congestion control in communication networks, where the actions that send the same (or similar) number of packets are placed in the same group. Further, note that for applications in which grouping functions are not explicitly given by domain knowledge, we can also employ supervised learning to learn the action groups.  For example, we can use more easily accessible offline datasets to classify actions into multiple groups based on the similarity of transition kernels and rewards.

Reviewer comment (Experiments):
How would the proposed perform when compared with other kinds of action abstraction methods?

Author response:
We compare our methods with other action abstractions in the following. One well-researched action abstraction type is temporally related actions from an initial state and terminal state [1].  However, these are specifically targeting the temporally extended action trajectories instead of actions, therefore they are significantly different from our works.

Another line of action abstractions is about factored action space. [2] represents the action space as the binary sequence based on prior knowledge, and learns the value function related to each bit. [3] further uses deep RL to learn the compositional structure of discrete action space to learn better policies and value functions. There are also works that learn the action representations based on prior knowledge that is extracted from expert demonstrations [4]. Without using prior knowledge, [5] learns the underlying structure of actions. However, our work diverges from these methods on applications where domain knowledge of the RL task is available but not yet fully leveraged to effectively balance the complexity and performance loss. Our research targets these applications, demonstrating that an appropriately designed grouping function can effectively reduce complexity with minimal impact on performance. 

[1] Riemer, M., Liu, M. and Tesauro, G., 2018. Learning abstract options. Advances in neural information processing systems, 31.

[2] Pazis, J. and Parr, R. Generalized value functions for large action sets. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1185– 1192, 2011.

[3] Sharma, S., Suresh, A., Ramesh, R. and Ravindran, B., 2017. Learning to factor policies and action-value functions: Factored action space representations for deep reinforcement learning. arXiv preprint arXiv:1705.07269.

[4] Tennenholtz, G. and Mannor, S., 2019, May. The natural language of actions. In International Conference on Machine Learning (pp. 6196-6205). PMLR.

[5] Chandak, Y., Theocharous, G., Kostas, J., Jordan, S. and Thomas, P., 2019, May. Learning action representations for reinforcement learning. In International conference on machine learning (pp. 941-950). PMLR.

Reviewer comment (Novelty):
I am not sure when do we need the proposed method. In which RL tasks do we specific need the proposed method to perform well?

Author response:
We respectfully disagree with the statement that the MDPs that exhibit group-wise similarity are narrow.  Our action group method is relevant in numerous practical settings. For instance, actions that correspond to proximate directions can be aggregated into a group in autonomous driving. Another example is congestion control in communication networks, where the actions that send the same (or similar) number of packets are placed in the same group. Further, note that for applications in which grouping functions are not explicitly given by domain knowledge, we can also employ supervised learning to learn the action groups.  For example, we can use more easily accessible offline datasets to classify actions into multiple groups based on the similarity of transition kernels and rewards.

Reviewer comment (Experiments):
There is no empirical demonstration of the practical viability of the practical method to estimate the performance-complexity trade-off. Since this is proposed as a "practical" approximation to the original joint optimization objective, it should be shown how this enables one to select a better action grouping strategy in an RL environment.

Author response:
Thanks for your suggestion on the empirical demonstration of the proposed practical method. To address this, we applied the grouping selection method in the downlink transmission setting (See figure in Appendix F. 2). In this particular setting, solving the approximated optimization problem (P2) allows us to identify the most effective grouping function to balance the tradeoff between the sample complexity and performance loss. For instance, with a large sample size, the optimization process tends to select a more refined grouping function ($G=20$). On the other hand, in scenarios where the sample size is constrained, the method gives a coaster grouping function.

Reviewer comment (Theory):
Extension to practical domains: How do the insights and theorems built on tabular value and policy iteration apply to deep RL? This is an important concern because, at the scale of large action spaces, such as those involved in recommender systems, deep RL is the only viable solution.

Author response:
There are scenarios where deep RL is not an appropriate tool for solving problems. Deep RL uses samples to simultaneously learn the relationships between state-action pairs and a good policy. However, when the domain knowledge of the RL task is given, how to use domain knowledge to reduce the sample complexity is not known in deep RL. We target these applications and show that the grouping function can effectively reduce the complexity while inducing little performance loss.   

Our methodology is quite general. So, we could use domain knowledge and apply action grouping as the preprocessing step to reduce the dimensionality of the action space, thus making deep neural network training more feasible and efficient. Thank you for the suggestion, this is a good problem to investigate further as a future direction.

Reviewer comment (Theory):
If the groupings are learned, how can the lower-level policy be feasibly obtained using domain knowledge? With the group compositions potentially changing non-stationarily, the lower level policy might itself be very challenging to learn or provide with domain knowledge.

Author response:
Note that from Theorem 1, we show that when actions in the same group are similar, the approximation error is small. Hence, when the transition kernel and rewards within a group are closely aligned, variations in lower-level policies do not significantly impact overall performance.

Our current focus on utilizing a domain knowledge-based grouping function is an initial step in our research. The exploration of learning the lower-level policy, particularly in dynamic MDP environments, can be an exciting direction for future work.

Reviewer comment (Theory):
In section 3.1 it is not explained if the state space and action are discrete, finite or bounded. Because of that it is also unclear if |g| defined in section 3.2 can be infinite. In the same section for the definition of $\pi^i$ should it be $\{h\}$ instead of $h$?

Author response:
Thanks for pointing out the need for clarity in Section 3.1. We confirm that in our current framework, both the state and action spaces are assumed to be discrete and finite. This assumption aligns with our grouped model, where $|g|$, the size of each action group, is also considered as a discrete and finite function. We have revised the paper accordingly.

We also acknowledge the potential of extending our model to continuous action spaces. In such scenarios, we propose to utilize domain knowledge to segment the continuous action space into multiple discrete and finite groups. This adaptation would allow us to apply our current algorithms to the lifted MDP, effectively reducing the sample complexity inherent in continuous action space.

Reviewer comment (Writing):
How is K defined in section 3.3? The notation used there is not understood.

Author response:
$K$ is the number of samples to evaluate the transition kernel of all state-group pairs. Essentially, $K$ can be regarded as the total required samples for estimating the lifted MDP.

Reviewer comment (Theory):
There is a major concern regarding how Algorithm 1 can lead to an almost optimal policy. It appears that an exploration term is missing to ensure good coverage of the action/state space. Without exploration, the algorithm could be stuck in a local minima while still accurately modeling the lifted MDP.

Author response:
Note that once we have modeled the lifted MDP accurately, we do not need to use exploration. We can simply use value iteration to converge to the nearly optimal solution (which is exactly what Algorithm 1 does). Our approach of grouping provides a tradeoff between sub-optimality and complexity. The sub-optimality occurs because of (i) grouping, (ii) the estimation error of the lifted MDP, and (iii) the algorithmic error. For the approximation error caused by grouping,  we show that when actions in the same group have similar transition probability and rewards (i.e., $\beta_P$ and $\beta_R$ are small), the performance loss due to different lower-level policies within groups is small. The estimation error of the lifted MDP is only proportional to the group size and inversely proportional to the number of samples. The algorithmic error decreases exponentially w.r.t. the iteration number.

Reviewer comment (Novelty):
The reviewer finds the stated observation about the relationship between performance loss and the grouping function—that a grouping function with a larger number of groups can sometimes achieve better performance—unsurprising. This is because abstractions are precisely used for such effects. Similar results exist in state abstraction literature, where finite time guarantees show that despite introducing approximation error, abstraction can lead to better performance.

Author response:
Thanks for your comments and for providing these references. The “surprising” part that we were referring to was not that fewer groups would lead to lower complexity at the cost of performance. The surprising part was that in scenarios with a limited number of samples, a smaller number of groups may in fact prove to be more effective than using a larger number of groups. This is because even though the approximation error is smaller in the case with a large number of groups, the estimation error caused by the limited samples for lifted MDP estimation with a large number of groups dominates the approximation error. Therefore, the
overall performance loss of the smaller number of groups case is still significantly larger than the setting with a large number of groups. To the best of our understanding this has not been shown in [1] and [2]. Moreover, we want to emphasize that since we quantitatively analyzed the sampling/computational complexity to learn a good policy with action grouping, we can additionally explore a computationally efficient approach to find the nearly optimal grouping function that carefully balances the complexity and performance loss.

Reviewer comment (Theory):
In section 5.2, given that $\mathbb{P}$ and R are bounded, isn't assumption 4 always verified?

Author response:
Yes, we agree that Assumption 4 always holds. We will appropriately rewrite $\eta_P=\max_{s,h,g(a_1)=g(a_2)}\|\mathbb{P}(\cdot|s,a_1)-\mathbb{P}(\cdot|s,a_2)\|$ and $\eta_R=\max_{s,h,g(a_1)=g(a_2)}R(s,a_1)-R(s,a_2)$.