Paper: Action Shapley: A training data selection metric for high performance and cost efficient reinforcement learning (ICLR-2024)

Reviewer comment (Novelty):
The motivation for using the Shapley value for selecting training actions is not sufficiently detailed in the introduction section. To a large extent, it remains unknown why it is necessary or important to use Shapley value to guide the selection of training actions, especially when existing research works have already studied various ways of selecting training actions for model-based and model-free reinforcement learning.

Author response:
* The use of the Shapley value to guide the selection of training data for the environment model within a RL system is important for the following reasons: (1) Fair Attribution of Contribution: The Shapley value ensures a fair and equitable distribution of credit among the different subsets of training data. By considering all possible permutations and combinations of data points, it quantifies the marginal contribution of each data point to the model's performance. (2) Avoidance of Bias: Without a systematic method like the Shapley value, there is a risk of bias in evaluating the importance of individual data points. Some data points might be overlooked or undervalued, leading to an inaccurate representation of their impact on the model's learning. (3) Robustness and Stability: The Shapley value provides a stable and consistent measure of the contribution of each data point across different scenarios. This is crucial for maintaining robustness in model training, especially when faced with varying datasets or changes in the distribution of data. (4) Interpretability: Shapley values offer interpretability by quantifying the influence of each data point on the model's predictions. This can aid in understanding the significance of specific instances in the dataset and inform data-driven decision-making. (5) Optimal Resource Allocation: By identifying the most influential data points, the Shapley value helps in allocating computational resources more efficiently. Focusing on the most informative data can enhance the learning process and reduce unnecessary computational overhead. (6) Ethical Considerations:  In certain applications, such as sensitive domains or when dealing with user data, fairness and ethical considerations become paramount. The Shapley value provides a principled approach to treating each data point fairly, promoting ethical practices in model development.
* In summary, employing the Shapley value for selecting training data ensures a systematic and unbiased approach to attributing importance to individual data points, contributing to a more robust, fair, and interpretable machine learning model.

Reviewer comment (Theory):
The mathematical definition of the action Shapley value in eq. (1) is not sufficiently clear. In fact $\phi$ is originally declared as a function of D and A, where A represents the learning algorithm. However, eq. (1) is clearly irrelevant to any learning algorithm.

Author response:
* Action Shapley exhibits a level of abstraction concerning the specific implementation details of reinforcement learning methodologies. In the context of Action Shapley, the requisite inputs consist of a set of data points and an evaluation function. This evaluation function is responsible for calculating the cumulative reward attained by the agent. Importantly, there exists no inherent impediment to the extension of Action Shapley different reinforcement learning frameworks.

Reviewer comment (Reproducibility):
$\mathcal{U}$ in eq. (1) is introduced as a valuation function. However, this paper did not give a clear idea regarding how $\mathcal{U}$ is defined or learned for general reinforcement learning problems.

Author response:
* The evaluation function, denoted as U, is delineated as the cumulative reward score corresponding to the output of the reinforcement learning (RL) agent.

Reviewer comment (Writing):
Eq. (3) is quite confusing. Hence, it is hard to judge the practical value of using eq. (1) for arbitrary real-world reinforcement learning problems.

Author response:
We value your feedback. Regrettably, due to time constraints, further benchmarking on additional datasets is not feasible at the moment. Nevertheless, we have incorporated PPO-PID for the purpose of comparing Action Shapley analytics with SAC-PID, the algorithm currently in use. Both PPO and SAC represent state-of-the-art in the field of reinforcement learning. Additionally, clarifications for Equation (1) and Equation (3) have been provided, and the notation section has been revised accordingly.

Reviewer comment (Theory):
The authors stated that the best possible training action set includes as many training actions as the global cut-off cardinality with the highest Action Shapley values. However, this important algorithm design decision is only explained intuitively without clear theoretical justifications. It remains questionable whether this is the best way to select the set of training actions and what can be guaranteed by using such a set of selected actions.

Author response:
* A comprehensive exploration of the combination tree necessitates $\mathcal{O}(2^n)$ time. Consequently, there is a preference for working with a minimal set of training data points. However, it is imperative to strike a balance, as a threshold number of training data points is essential for the development of a proficient Reinforcement Learning (RL) agent. Insufficient training data points compromise the capacity to construct an effective environment model, hindering the agent's ability to fulfill its designated goal. The determination of an optimal trade-off lies in identifying the global cut-off cardinality, which represents the point of equilibrium between computational efficiency and the requisite data for robust RL agent training.

Reviewer comment (Experiments):
For the experiments, no comparison with existing baselines was reported, making it hard to understand whether the new algorithm can achieve state-of-the-art performance on any benchmark reinforcement learning problems.

Author response:
We value your feedback. Regrettably, due to time constraints, further benchmarking on additional datasets is not feasible at the moment. Nevertheless, we have incorporated PPO-PID for the purpose of comparing Action Shapley analytics with SAC-PID, the algorithm currently in use. Both PPO and SAC represent state-of-the-art in the field of reinforcement learning. Additionally, clarifications for Equation (1) and Equation (3) have been provided, and the notation section has been revised accordingly.

Reviewer comment (Theory):
The authors stated on page 5 that they assume the next environment state depends purely on the agent action. This assumption is often wrong for reinforcement learning. Therefore, the validity of their algorithm is also questionable.

Author response:
* The stated proposition (the next environment state depends purely on the agent action) lacks general validity, and an error in the initial draft has been identified. Our empirical findings indicate that, in specific case studies such as VM right-sizing and load balancing, the subsequent state of the environment is contingent solely upon the agent's action. However, in the context of database tuning and Kubernetes management, the determination of the subsequent environment state is contingent upon both the agent's action and the prevailing state of the environment.

Reviewer comment (Evaluation):
Computationally expensive.

Author response:
We have incorporated a discussion on computational complexity on Page 4, with specific reference to the discourse surrounding Equation 2. The computational complexity of Algorithm 1 is situated between $\mathcal{O}(2^n)$, denoting the worst-case scenario, and $\mathcal{O}(\epsilon)$, reflecting the best-case scenario. Combining these two extremes of $\mathcal{O}(2^n)$ and $\mathcal{O}(\epsilon)$, the performance of Algorithm 1  can be represented as a ratio of the exponential of global cut-off cardinality and the exponential of the total number of training data points subtracted from $1$, as shown in Equation 2.

Reviewer comment (Experiments):
Applicable to very simple domains (discrete and deterministic with few actions).

Author response:
The domain under consideration exhibits a continuous-state space paired with a discrete-action space. The diversity of potential action values is evident in the trajectories depicted in Figures 6 through 9 in the Appendix. The training set utilized for the environment model comprises 4, 5, 6, and 15 training actions, each accompanied by a corresponding state time series, encompassing 1440 data points. This configuration closely mirrors the intricacies of real-world systems.

Reviewer comment (Theory):
It is not clear why the authors mention that trial-and-error for RL in domains like Go or StarCraft are relatively inexpensive.

Author response:
Trial-and-error in reinforcement learning (RL) for domains like Go or StarCraft is relatively inexpensive for several reasons: (1) Simulated Environments: In games like Go or StarCraft, it is often possible to create highly realistic simulations of the environment. Simulations allow agents to explore and learn without interacting with the real world, reducing the cost and risks associated with physical experimentation.
(2) Fast Iteration: Simulated environments enable fast and efficient iteration. Agents can undergo numerous trials in a short amount of time, accelerating the learning process. This rapid iteration is particularly advantageous for RL algorithms, allowing them to explore a large state space and discover effective strategies quickly. (3) Parallelization: Many RL algorithms can be parallelized, enabling multiple agents to explore different possibilities simultaneously.  (4) Low Resource Costs: Simulations typically require fewer resources than physical experiments. Running simulations is computationally less expensive than deploying real-world agents, making it more cost-effective to perform extensive trial-and-error in simulated environments. (5) Reproducibility: Simulated environments provide a high level of reproducibility. Researchers and practitioners can precisely replicate experiments, facilitating the comparison of different algorithms and strategies.  (6) Safety: In domains like StarCraft, trial-and-error in a simulated environment avoids the potential risks associated with deploying agents in the real world. RL agents can learn complex strategies and tactics without the concern for safety issues or unintended consequences.

Reviewer comment (Experiments):
All the tests are performed in very similar domains, which questions the applicability to other domains.

Author response:
The conducted tests are executed within the cloud computing infrastructure; however, it is noteworthy that no explicit reliance on specific system properties is made. The dataset is intentionally randomized across diverse systems, encompassing Virtual Machines (VMs) as computing systems, load balancers as networking systems, databases as storage systems, and Kubernetes as a container orchestration system. Each of these systems exhibits distinct time constants and seasonality patterns, as elucidated in the Appendix. For instance, the training data for Database Tuning (Figure 12) manifests random variability. In contrast, the training data for both VM right-sizing (Figure 10) and load balancing (Figure 11) adopt a triangular distribution with disparate temporal characteristics.

Reviewer comment (Experiments):
It is not clear how to select \epsilon. A high \epsilon means more computation, while a low \epsilon.

Author response:
$\epsilon$ is a user-defined parameter. A heightened value of  $\epsilon$ signifies a proclivity toward exploring a greater array of training data subsets and engaging in a more aggressive pursuit of an optimal training subset. Conversely, a diminished $\epsilon$ indicates a preference for the exploration of fewer training data subsets, thereby facilitating a more expeditious convergence. This dynamic reflects a discernible exploration-exploitation trade-off.

Reviewer comment (Experiments):
It seems to be applicable only to discrete domains with a small number of possible actions. Also, the tests are performed on deterministic environments. Such conditions seem very restrictive for real world or even simple, domains.

Author response:
The domain under consideration exhibits a continuous-state space paired with a discrete-action space. The diversity of potential action values is evident in the trajectories depicted in Figures 6 through 9 in the Appendix. The training set utilized for the environment model comprises 4, 5, 6, and 15 training actions, each accompanied by a corresponding state time series, encompassing 1440 data points. This configuration closely mirrors the intricacies of real-world systems.

Reviewer comment (Evaluation):
Evaluating the Shapley values is still computationally expensive, even with the proposed algorithm. It is not clear how much an agent gains with this approach. Once the values are known, the gain is quite clear, however, the authors do not report how expensive is to obtain such values.

Author response:
We have incorporated a discussion on computational complexity on Page 4, with specific reference to the discourse surrounding Equation 2. The computational complexity of Algorithm 1 is situated between $\mathcal{O}(2^n)$, denoting the worst-case scenario, and $\mathcal{O}(\epsilon)$, reflecting the best-case scenario. Combining these two extremes of $\mathcal{O}(2^n)$ and $\mathcal{O}(\epsilon)$, the performance of Algorithm 1  can be represented as a ratio of the exponential of global cut-off cardinality and the exponential of the total number of training data points subtracted from $1$, as shown in Equation 2.

Reviewer comment (Novelty):
The use of the Shapley value is not new in the literature. The main difference in this paper is to use it for the selection of actions.

Author response:
Our distinctive contribution involves the innovative application of the Shapley value in selecting training data for the environment model training of a Reinforcement Learning (RL) system. The key challenges stem from: (1) Combinatorial Explosion: RL environment often deals with large state and action spaces, leading to a combinatorial explosion of possible coalitions (combinations of agents) for which Shapley values need to be calculated. (2) Sequential Decision Making: RL involves sequential decision making, where the impact of an agent's action may not be immediately apparent but could unfold over time. This temporal aspect introduces complexities. (3) Model Uncertainty: RL often involves uncertainty, both in terms of the environment dynamics and the learned model.

Reviewer comment (Writing):
Some terms are not properly described in the paper, e.g., g(a_t), \phi.

Author response:
The manuscript underwent comprehensive proofreading procedures, and an additional section dedicated to Notation has been incorporated within the Appendix.

Reviewer comment (Writing):
The paper has several English errors that need to be corrected.

Author response:
The manuscript underwent comprehensive proofreading procedures, and an additional section dedicated to Notation has been incorporated within the Appendix.

Reviewer comment (Presentation):
The presentation of this work is very poor. It is difficult to understand this method directly without relevant background knowledge. One suggestion to improve readability is to add a Background and Notation section before the methodology section.

Author response:
* The authors express sincere gratitude for the valuable feedback received. In response to the limitations imposed by page constraints, a new subsection titled "Background and Notation" has been incorporated at the outset of the Methodology section. Furthermore, to effectively navigate the challenges posed by page limitations, a dedicated Notation section has been appended to the document's Appendix.

Reviewer comment (Writing):
The manuscript’s text description is not well-organized, as it has only one paragraph for almost every chapter, which makes it tiring for the reader. A suggestion is to divide the text into appropriate paragraphs.

Author response:
* We appreciate your observation regarding the paragraph division and understand your concern about the potential strain it may place on the reader. In response to your suggestion, we have revised the manuscript to enhance readability by incorporating more nuanced paragraphing, ensuring a smoother flow between ideas. We trust that these adjustments contribute positively to the overall reading experience with the specified constraints of page limit.

Reviewer comment (Presentation):
A label should be added to the display of pictures.

Author response:
* All figures have labels.

Reviewer comment (Writing):
The motivation of this paper is not explained well. The reviewer cannot gain deep insight from the current version of this manuscript. The reviewer believes this manuscript was hastily completed, although the author may have solved some interesting problems. The reviewer requests revision according to the provided comments before reviewing again.

Author response:
* The provided feedback has been valued, leading to a comprehensive revision of the manuscript. Our objective is to establish a principled selection metric for training data pertaining to the environment model within a reinforcement learning system. Action Shapley is distinguished among alternative selection methodologies such as LIME due to its advantages in sample efficiency, model agnosticity, interpretability, fairness, and stability.  
* While the use of the Shapley value has been documented in extant literature, our contribution lies in its novel application to the domain of training data selection for the environment model of a reinforcement learning (RL) system. The integration of Shapley value in this context presents several challenges: (1) Combinatorial Explosion: RL often deals with large state and action spaces, leading to a combinatorial explosion of possible coalitions (combinations of agents) for which Shapley values need to be calculated. This computational complexity can be prohibitive for exact Shapley value computation, especially in complex environments. (2) Sample Inefficiency: Estimating Shapley values requires evaluating the marginal contribution of each agent across all possible coalitions. In RL, obtaining accurate estimates of these marginal contributions may require a large number of samples, leading to sample inefficiency, which can be a significant drawback in terms of computational resources. (3) Sequential Decision Making: RL involves sequential decision making, where the impact of an agent's action may not be immediately apparent but could unfold over time. This temporal aspect introduces complexities in attributing contributions to individual agents, as their actions may interact in non-trivial ways. (4) Model Uncertainty: RL often involves uncertainty, both in terms of the environment dynamics and the learned model. Shapley values rely on a well-defined cooperative game, and uncertainties can introduce challenges in accurately modeling and attributing contributions. (5) Computational Resources: Calculating Shapley values can be computationally expensive, and for real-time applications or resource-constrained environments, this may pose a limitation.  
* We handle this problem using a randomized dynamic algorithm, acknowledging the trade-offs between accuracy and computational feasibility.  
* The manuscript underwent comprehensive proofreading procedures, and an additional section dedicated to Notation has been incorporated within the Appendix. The authors express gratitude for the thorough review conducted on the revised manuscript.