Original: Model-based Reinforcement Learning for Service Mesh Fault Resiliency in a Web Application-level
Revised: Model-based Reinforcement Learning for Service Mesh Fault Resiliency in a Web Application-level

Original: ABSTRACT
Revised: ABSTRACT

Original: Microservice-based architectures enable different aspects of web applications to be created and updated independently, even after deployment.
Revised: Microservice-based architectures enable different aspects of web applications to be created and updated independently, even after deployment.

Original: Associated technologies such as service mesh provide application-level fault resilience through attribute configurations that govern the behavior of request - response service – and the interactions among them – in the presence of failures.
Revised: Associated technologies such as service mesh provide application-level fault resilience through attribute configurations that govern the behavior of request - response service – and the interactions among them – in the presence of failures.

Original: While this provides tremendous flexibility, the configured values of these attributes – and the relationships among them – can significantly affect the performance and fault resilience of the overall application.
Revised: While this provides tremendous flexibility, the configured values of these attributes – and the relationships among them – can significantly affect the performance and fault resilience of the overall application.

Original: Furthermore, it is impossible to determine the best and worst combinations of attribute values with respect to fault resiliency via testing, due to the complexities of the underlying distributed system and the many possible attribute value combinations.
Revised: Furthermore, it is impossible to determine the best and worst combinations of attribute values with respect to fault resiliency via testing, due to the complexities of the underlying distributed system and the many possible attribute value combinations.

Original: In this paper, we present a model-based reinforcement learning workflow towards service mesh fault resiliency.
Revised: In this paper, we present a model-based reinforcement learning workflow towards service mesh fault resiliency.

Original: Our approach enables the prediction of the most significant fault resilience behaviors at a web application-level, scratching from single service to aggregated multi-service management with efficient agent collaborations
Revised: Our approach enables the prediction of the most significant fault resilience behaviors at a web application-level, scratching from single service to aggregated multi-service management with efficient agent collaborations

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: A key trend in web application development in recent years is the advent of microservices-based architectures, in which applications are composed of small microservices that communicate with one another via distributed system mechanisms.
Revised: A key trend in web application development in recent years is the advent of microservices-based architectures, in which applications are composed of small microservices that communicate with one another via distributed system mechanisms.

Original: Using open-source microservices technologies such as Kubernetes [1, 34, 48], developers can create and update different aspects of an application independently, even after deployment.
Revised: Using open-source microservices technologies such as Kubernetes [1, 34, 48], developers can create and update different aspects of an application independently, even after deployment.

Original: At the same time, to ensure that faults in individual microservices – or delays in communication among them – do not cascade into application-level failures, microservice-based architectures increasingly include service mesh technologies such as Istio [2, 13] and Linkerd [3, 32].
Revised: At the same time, to ensure that faults in individual microservices – or delays in communication among them – do not cascade into application-level failures, microservice-based architectures increasingly include service mesh technologies such as Istio [2, 13] and Linkerd [3, 32].

Original: These service meshes [12, 26, 36] contain associated "sidecars" [4] that monitor individual microservices for failures and delays, and perform associated actions to ensure application-level fault resilience; for example, bypassing problematic microservices upon consecutive errors, or ejecting them for a period of time [29, 37, 49].
Revised: These service meshes [12, 26, 36] contain associated "sidecars" [4] that monitor individual microservices for failures and delays, and perform associated actions to ensure application-level fault resilience; for example, bypassing problematic microservices upon consecutive errors, or ejecting them for a period of time [29, 37, 49].

Original: The number of consecutive errors or the length of the ejection time is configured through attributes in the service mesh.
Revised: The number of consecutive errors or the length of the ejection time is configured through attributes in the service mesh.

Original: The service mesh architecture is depicted in Figure 1.
Revised: The service mesh architecture is depicted in Figure 1.

Original: As Figure 2 shows [5], multiple services are aggregated and responsible for request - connection independently, which is challenging to be optimized when faults injected [19].
Revised: As Figure 2 shows [5], multiple services are aggregated and responsible for request - connection independently, which is challenging to be optimized when faults injected [19].

Original: Istio h ttpbin supported by service mesh is developed to control over web application behaviors, providing tremendous flexibility for communications.
Revised: Istio h ttpbin supported by service mesh is developed to control over web application behaviors, providing tremendous flexibility for communications.

Original: Nevertheless, the degree to which an web application is fault resilient is heavily dependent on the configured attribute values [18, 31, 47] and the relationships among them.
Revised: Nevertheless, the degree to which an web application is fault resilient is heavily dependent on the configured attribute values [18, 31, 47] and the relationships among them.

Original: Furthermore, it is impossible to determine the best and worst combinations of attributes via testing, due to the complexities of the underlying distributed system and the many possible attribute value combinations.
Revised: Furthermore, it is impossible to determine the best and worst combinations of attributes via testing, due to the complexities of the underlying distributed system and the many possible attribute value combinations.

Original: The emerging machines learning methods have been widely applied to many networking problems, however, there is no previous machine learning methods focused on exploring the service mesh resiliency.
Revised: The emerging machines learning methods have been widely applied to many networking problems, however, there is no previous machine learning methods focused on exploring the service mesh resiliency.

Original: In this paper, we present a model-based reinforcement learning approach towards service mesh ( h ttpbin) fault resiliency that we call SFR2L (Service Fault Resiliency with Reinforcement Learning).
Revised: In this paper, we present a model-based reinforcement learning approach towards service mesh ( h ttpbin) fault resiliency that we call SFR2L (Service Fault Resiliency with Reinforcement Learning).

Original: Our novel contributions are as followings:
Revised: Our novel contributions are as followings:

Original: 1) To the best of our knowledge, it is the first investigation on service mesh resiliency using machine learning methods, especially the first one to optimize aggregated services using model-based reinforcement learning, which enables
Revised: 1) To the best of our knowledge, it is the first investigation on service mesh resiliency using machine learning methods, especially the first one to optimize aggregated services using model-based reinforcement learning, which enables

Original: the prediction of the most significant fault resiliency applicationlevel behaviors.
Revised: the prediction of the most significant fault resiliency applicationlevel behaviors.

Original:  We develop a complete workflow to implement the flexible control over service mesh resiliency, including data collection, service modelling, policy learning for resiliency optimization and validation on simulation.
Revised: 2) We develop a complete workflow to implement the flexible control over service mesh resiliency, including data collection, service modelling, policy learning for resiliency optimization and validation on simulation.

Original: In terms of the policy learning, three cases are investigated: single agent for single service, multiple agents for single service and collaborative agents for multi-services, which consider most common optimizing scenarios in web applications.
Revised: In terms of the policy learning, three cases are investigated: single agent for single service, multiple agents for single service and collaborative agents for multi-services, which consider most common optimizing scenarios in web applications.

Original: There are five structured fault injection (circuit breaking pattern) datasets disclosed by us, covering from 50 toloading call settings.
Revised: There are five structured fault injection (circuit breaking pattern) datasets disclosed by us, covering from 50 toloading call settings.

Original: Each dataset is well-labelled and available at here .
Revised: Each dataset is well-labelled and available at here .

Original: We hope this work can make contributions to the fault resiliency community and help more scholars perform more impactful researches in this field.
Revised: We hope this work can make contributions to the fault resiliency community and help more scholars perform more impactful researches in this field

Original:  RELATED WORK 2. Service Modelling
Revised:  RELATED WORK 2. Service Modelling

Original: [50] present a model-based reinforcement learning approach for resource allocation in scientific workflow systems based on microservices.
Revised: [50] present a model-based reinforcement learning approach for resource allocation in scientific workflow systems based on microservices.

Original: While this paper does not address service meshes or fault resilience, our overall approach is inspired by their work.
Revised: While this paper does not address service meshes or fault resilience, our overall approach is inspired by their work.

Original: From a service resiliency perspective, two kinds of approaches have been proposed: systematic testing and formal modeling.
Revised: From a service resiliency perspective, two kinds of approaches have been proposed: systematic testing and formal modeling.

Original: [24] presents an infrastructure and approach for systematic testing of resiliency.
Revised: [24] presents an infrastructure and approach for systematic testing of resiliency.

Original: However, this work does not cover the selection of the tests to be run.
Revised: However, this work does not cover the selection of the tests to be run.

Original: Our work focuses on automating such a selection through machine learning.
Revised: Our work focuses on automating such a selection through machine learning.

Original: In terms of formal modeling, [41] presents the use of continuoustime Markov Chains (CTMCs) and formal verification to analyze tradeoffs in service resiliency mechanisms in simple client-service interactions.
Revised: In terms of formal modeling, [41] presents the use of continuoustime Markov Chains (CTMCs) and formal verification to analyze tradeoffs in service resiliency mechanisms in simple client-service interactions.

Original: Earlier work [27] also uses formal verification based on CTMCs, and analyzes multiple concurrent target services as well as steady-state availability measures including degraded service functionality.[Table]
Revised: Earlier work [27] also uses formal verification based on CTMCs, and analyzes multiple concurrent target services as well as steady-state availability measures including degraded service functionality.

Original: Microservices and their self-adaptation is an active area of research, and a comprehensive survey and taxonomy of recent work is given in [20].
Revised: Microservices and their self-adaptation is an active area of research, and a comprehensive survey and taxonomy of recent work is given in [20].

Original: However, as described in [20], there is a very limited work on application-level resilency, as well as very little work on using machine learning in the context of service self-adaptation. 
Revised: However, as described in [20], there is a very limited work on application-level resilency, as well as very little work on using machine learning in the context of service self-adaptation. [Table]

Original: 2. The Istio Service Mesh
Revised: 2. The Istio Service Mesh

Original: As described in Section 1, Istio is an open-source service mesh technology for distributed and microservice architectures.
Revised: As described in Section 1, Istio is an open-source service mesh technology for distributed and microservice architectures.

Original: Istio provides a transparent way to build applications.
Revised: Istio provides a transparent way to build applications.

Original: Istio’s traffic management features enable service monitoring and application-level fault resilience.
Revised: Istio’s traffic management features enable service monitoring and application-level fault resilience.

Original: In particular, Istio provides outlier detection and circuit breakers to realize fault resilience.
Revised: In particular, Istio provides outlier detection and circuit breakers to realize fault resilience.

Original: Outlier detection enables the capacity of services to be limited when they are behaving anomalously, or even to be ejected for a period of time.
Revised: Outlier detection enables the capacity of services to be limited when they are behaving anomalously, or even to be ejected for a period of time.

Original: Circuit breaking [6] is a capability that prevents service failures from cascading.
Revised: Circuit breaking [6] is a capability that prevents service failures from cascading.

Original: In particular, if a service A calls another service B, which does not respond within an acceptable time period, the call can be retried or even bypassed via the circuit breaker specification.
Revised: In particular, if a service A calls another service B, which does not respond within an acceptable time period, the call can be retried or even bypassed via the circuit breaker specification.

Original: As depicted in Figure 1, the sidecar proxy for service B monitors its response, and the sidecar for service A performs the specified circuit breaking.
Revised: As depicted in Figure 1, the sidecar proxy for service B monitors its response, and the sidecar for service A performs the specified circuit breaking.

Original: Istio also provides fault injection [7] and load testing [8] capabilities in order to test application fault recovery.
Revised: Istio also provides fault injection [7] and load testing [8] capabilities in order to test application fault recovery.

Original: Such testing is considered critical to perform prior to application deployment to gain confidence in the fault resilience of deployed applications.
Revised: Such testing is considered critical to perform prior to application deployment to gain confidence in the fault resilience of deployed applications.

Original: To realize these fault resilience mechanisms, Istio enables traffic rules to be configured for application deployment; these configurations govern the specific behaviors of outlier detection and circuit breaking.
Revised: To realize these fault resilience mechanisms, Istio enables traffic rules to be configured for application deployment; these configurations govern the specific behaviors of outlier detection and circuit breaking.

Original: Some of the attributes of these traffic rules are depicted in Table 1, and govern the number of requests and connections allowed to a service that may be behaving anomalously, the amount of time it may be ejected and at what rate and at what detection interval, and the number of consecutive errors after which a circuit breaker will be tripped.
Revised: Some of the attributes of these traffic rules are depicted in Table 1, and govern the number of requests and connections allowed to a service that may be behaving anomalously, the amount of time it may be ejected and at what rate and at what detection interval, and the number of consecutive errors after which a circuit breaker will be tripped.

Original: The total threads is the number of Istio worker threads, while the total calls is the number of requests to the application used in Istio load testing.
Revised: The total threads is the number of Istio worker threads, while the total calls is the number of requests to the application used in Istio load testing.

Original: Details of these attributes as well as other Istio traffic rule attributes are at [9]; configurations for fault injection and load testing are at [7, 8].
Revised: Details of these attributes as well as other Istio traffic rule attributes are at [9]; configurations for fault injection and load testing are at [7, 8].

Original: While configuration of these traffic rule attributes enables finegrained control of fault resilience policies and fault injection and load testing, the degree to which an application is fault resilient is heavily dependent on the attribute configurations and the relationships among them.
Revised: While configuration of these traffic rule attributes enables finegrained control of fault resilience policies and fault injection and load testing, the degree to which an application is fault resilient is heavily dependent on the attribute configurations and the relationships among them.

Original: Thus, a key challenge is to determine the most significant combination of attribute values, where the“best” values from a fault resilience perspective can be used for application deployment and the “worst” values can be used to drive fault injection and load testing.
Revised: Thus, a key challenge is to determine the most significant combination of attribute values, where the“best” values from a fault resilience perspective can be used for application deployment and the “worst” values can be used to drive fault injection and load testing.

Original: However, the determination of these most significant value combinations is highly complex due to the inter-dependencies among attributes, the failure behavior of the underlying services and the communication among them, as well as the complexities of the underlying distributed system.
Revised: However, the determination of these most significant value combinations is highly complex due to the inter-dependencies among attributes, the failure behavior of the underlying services and the communication among them, as well as the complexities of the underlying distributed system.

Original: Thus, it is impossible to determine the most significant attribute values via manual or automated testing due to sheer number of possible behaviors.
Revised: Thus, it is impossible to determine the most significant attribute values via manual or automated testing due to sheer number of possible behaviors.

Original:  OVERVIEW OF REINFORCEMENT LEARNING
Revised:  OVERVIEW OF REINFORCEMENT LEARNING

Original: In our context, rewards correspond to the degree of service meshbased application over time.
Revised: In our context, rewards correspond to the degree of service meshbased application over time.

Original: "Worst-case" rewards (penalties) can be used to determine configuration settings that are critical to test for resiliency in load testing prior to application deployment.
Revised: "Worst-case" rewards (penalties) can be used to determine configuration settings that are critical to test for resiliency in load testing prior to application deployment.

Original: For model-free methods to be used in our context, the agents must take real-time actions directly on the service mesh, and learn from the observed behavior.
Revised: For model-free methods to be used in our context, the agents must take real-time actions directly on the service mesh, and learn from the observed behavior.

Original: This necessitates implementation of algorithmic APIs in the Istio environment, likely to be very expensive and inefficient.
Revised: This necessitates implementation of algorithmic APIs in the Istio environment, likely to be very expensive and inefficient.

Original: On the other hand, model-based methods must be approached with care, as deterministic models are unlikely to yield good approximations of the transition states among different microservices.
Revised: On the other hand, model-based methods must be approached with care, as deterministic models are unlikely to yield good approximations of the transition states among different microservices.

Original: [10]firstlypropose to utilize deep neural network to generate
Revised: [10]firstlypropose to utilize deep neural network to generate

Original: Q-factor instead of storing large amount of reward-action pairs in harsh table.
Revised: Q-factor instead of storing large amount of reward-action pairs in harsh table.

Original: Following their works, [38] present an algorithm according to the deterministic policy gradient to execute over continuous action spaces.
Revised: Following their works, [38] present an algorithm according to the deterministic policy gradient to execute over continuous action spaces.

Original: For model-based reinforcement learning, [23, 30, 39, 43] demonstrate the theoretical basis of policy gradient for model-based interaction [15, 16, 46].
Revised: For model-based reinforcement learning, [23, 30, 39, 43] demonstrate the theoretical basis of policy gradient for model-based interaction [15, 16, 46].

Original: With regard to multi-agent reinforcement learning (MARL), [44] introduce the efficient MARL algorithm for parallel policy optimization.
Revised: With regard to multi-agent reinforcement learning (MARL), [44] introduce the efficient MARL algorithm for parallel policy optimization.

Original: [14] propose to deploy multi-agent to optimize the traffic controls and networks, which is an important application in actual networking practice.
Revised: [14] propose to deploy multi-agent to optimize the traffic controls and networks, which is an important application in actual networking practice.

Original: Communication/Collaboration is a common configuration in multi-agents system [17, 21, 28, 33, 53] and advantageous at executing more stable, efficient and better decision-makings [25, 35, 40, 45, 51, 52] using decentralized Q-network.
Revised: Communication/Collaboration is a common configuration in multi-agents system [17, 21, 28, 33, 53] and advantageous at executing more stable, efficient and better decision-makings [25, 35, 40, 45, 51, 52] using decentralized Q-network.

Original: Nevertheless, decentralized multiagents have weak performance in the case that only small datasets are available or there are fewer state vector features for policy learning.
Revised: Nevertheless, decentralized multiagents have weak performance in the case that only small datasets are available or there are fewer state vector features for policy learning.

Original: Regarding this, [22, 27] conduct the proof of cooperative multi-agent and illustrate that the model parameters can be shared by decentralized multiple agents and agent itself also preserve its own private network to make decisions, which is the groundwork for our multi-service management.
Revised: Regarding this, [22, 27] conduct the proof of cooperative multi-agent and illustrate that the model parameters can be shared by decentralized multiple agents and agent itself also preserve its own private network to make decisions, which is the groundwork for our multi-service management.

Original: Our simulation model is inspired by [50], which introduces good approximation capabilities in assisting policy learning for service-mesh based architecture, obtaining greater training and networking optimizing efficiency than the model-free approaches.
Revised: Our simulation model is inspired by [50], which introduces good approximation capabilities in assisting policy learning for service-mesh based architecture, obtaining greater training and networking optimizing efficiency than the model-free approaches.

Original:  WORKFLOW IMPLEMENTATION
Revised:  WORKFLOW IMPLEMENTATION

Original: Our implementation for exploring service mesh-based application fault resiliency is organized as followings (Figure 4 depicts the whole pipeline)
Revised: Our implementation for exploring service mesh-based application fault resiliency is organized as followings (Figure 4 depicts the whole pipeline)

Original: 1) Data collection from an Istio application; 2) Simulation model training and selection; 3) Model-based Reinforcement learning; 4) Validation on policy learning results.
Revised: 1) Data collection from an Istio application; 2) Simulation model training and selection; 3) Model-based Reinforcement learning; 4) Validation on policy learning results.

Original: 4. Data collection
Revised: 4. Data collection

Original: To this end, we collect substantial traces covering target parametric spaces of the Istio httpbin service for each dataset, varying the configuration settings of the traffic rules and fault injection and load testing settings from Table 1.
Revised: To this end, we collect substantial traces covering target parametric spaces of the Istio httpbin service for each dataset, varying the configuration settings of the traffic rules and fault injection and load testing settings from Table 1.

Original: All collected data points are based on the actual Istio application and well-labelled after data cleaning.
Revised: All collected data points are based on the actual Istio application and well-labelled after data cleaning.

Original: 4. Simulation Model
Revised: 4. Simulation Model

Original: In general, we apply multiple-layer perceptrons (MLP) to simulate the aggregated behaviors, enabling agents to interact with the environment and learn the best loading space given the traffic rule attributes.
Revised: In general, we apply multiple-layer perceptrons (MLP) to simulate the aggregated behaviors, enabling agents to interact with the environment and learn the best loading space given the traffic rule attributes.

Original: Figure 3 depicts the input-output relations for modelling application-level fault resiliency.
Revised: Figure 3 depicts the input-output relations for modelling application-level fault resiliency.

Original: The 9-dimensional input vector contains 7 deterministic traffic rules and 2 loading settings given in Table 1, and the output corresponds to the 2 learned response attributes.
Revised: The 9-dimensional input vector contains 7 deterministic traffic rules and 2 loading settings given in Table 1, and the output corresponds to the 2 learned response attributes.

Original: We train the simulation MLP model weights: W 𝑆𝑀 = 𝑙 (cid:206) 𝑙 = 1 ( Φ 𝑙 + b 𝑙 ), which is the well-trained 𝑙 -layer MLP ( Φ 𝑙 is the weights of 𝑙 -th layer and b 𝑙 is the corresponding bias).
Revised: We train the simulation MLP model weights: W 𝑆𝑀 = 𝑙 (cid:206) 𝑙 = 1 ( Φ 𝑙 + b 𝑙 ), which is the well-trained 𝑙 -layer MLP ( Φ 𝑙 is the weights of 𝑙 -th layer and b 𝑙 is the corresponding bias).

Original: When the inference error reaches minimum, we save the model weights for following agent-environment interactions.
Revised: When the inference error reaches minimum, we save the model weights for following agent-environment interactions.

Revised: learning agent(s), Y is the web application (APP) response vector with 2 features: QPS and failure rate.

Original: 4. Model-based Reinforcement Learning
Revised: 4. Model-based Reinforcement Learning

Original: We then use our simulation model as the basis for our model-based reinforcement learning (RL).
Revised: We then use our simulation model as the basis for our model-based reinforcement learning (RL).

Original: In our context, the environment is our simulation model (well-trained MLP), states correspond to traffic rule settings, actions determine the loading settings (the number of threads and loading calls).
Revised: In our context, the environment is our simulation model (well-trained MLP), states correspond to traffic rule settings, actions determine the loading settings (the number of threads and loading calls).

Original: We explore both single and multiple agents, the latter either working independently or dependently.
Revised: We explore both single and multiple agents, the latter either working independently or dependently.

Original: As depicted in Figure 4, our agents learn from responses of our simulation model of the Istio h ttpbin service to the perform actions.
Revised: As depicted in Figure 4, our agents learn from responses of our simulation model of the Istio h ttpbin service to the perform actions.

Original: In real-world web application, communications between servers and clients are normally aggreagted with multiple services (microservices) and holistic resiliency for all services are essential.
Revised: In real-world web application, communications between servers and clients are normally aggreagted with multiple services (microservices) and holistic resiliency for all services are essential.

Original: Take a simple shopping web application for example, it may contain check-out, product, sign-up/in/out, shopping cart and review service.
Revised: Take a simple shopping web application for example, it may contain check-out, product, sign-up/in/out, shopping cart and review service.

Original: These services experience very closed visiting volume due to the fact that visitors would go through all these steps to finish their shopping.
Revised: These services experience very closed visiting volume due to the fact that visitors would go through all these steps to finish their shopping.

Original: All services play individual roles but also have interdependencies to ensure the function of whole application.
Revised: All services play individual roles but also have interdependencies to ensure the function of whole application.

Original: So we extend our multi-agent collaboration into management of multiservice resiliency.
Revised: So we extend our multi-agent collaboration into management of multiservice resiliency.

Original: Section 5 describes our model-based reinforcement learning algorithm in much more detail.
Revised: Section 5 describes our model-based reinforcement learning algorithm in much more detail.

Original: 4. Validation
Revised: 4. Validation

Original: Finally, we train the reinforcement learning algorithms to do policy learning and obtain optimized loading decisions given different traffic rule and loading setting combinations.
Revised: Finally, we train the reinforcement learning algorithms to do policy learning and obtain optimized loading decisions given different traffic rule and loading setting combinations.

Original: All decisions with the highest reward are validated in actual Istio to evaluate policy learning.
Revised: All decisions with the highest reward are validated in actual Istio to evaluate policy learning.

Original:  REINFORCEMENT LEARNING PARADIGM
Revised:  REINFORCEMENT LEARNING PARADIGM

Original: In the beginning, we present some preliminaries on RL and how states, actions, and rewards are defined in the context of simulation model.
Revised: In the beginning, we present some preliminaries on RL and how states, actions, and rewards are defined in the context of simulation model.

Original: Based on the basic setting, we illustrate how to apply single agent to single service.
Revised: Based on the basic setting, we illustrate how to apply single agent to single service.

Original: Then we discuss the deployment of multiagent reinforcement learning to address the complex parametric space optimization according to their collaborative relationships.
Revised: Then we discuss the deployment of multiagent reinforcement learning to address the complex parametric space optimization according to their collaborative relationships.

Original: Finally, the multi-service holistic resiliency optimization is illustrated using the combination of decentralized learning and centralized learning paradigm.
Revised: Finally, the multi-service holistic resiliency optimization is illustrated using the combination of decentralized learning and centralized learning paradigm.

Original: We define 𝑡 as the index of timestamp that RL agent interacts with the simulation model (SM), 𝑠 ( 𝑡 ) as the state vector for RL agent(s) at 𝑡 , 𝑟 ( 𝑡 ) as the reward for 𝑡 -th action 𝑎 ( 𝑡 ) , i ( 𝑡 ) as the input vector for SM, o ( t ) as the application responses emulated by the
Revised: We define 𝑡 as the index of timestamp that RL agent interacts with the simulation model (SM), 𝑠 ( 𝑡 ) as the state vector for RL agent(s) at 𝑡 , 𝑟 ( 𝑡 ) as the reward for 𝑡 -th action 𝑎 ( 𝑡 ) , i ( 𝑡 ) as the input vector for SM, o ( t ) as the application responses emulated by the

Original: SM.
Revised: SM.

Original: W 𝑆𝑀 is the well-trained simulation model weights.
Revised: W 𝑆𝑀 is the well-trained simulation model weights.

Original: 𝑎 ( 𝑡 ) and s ( 𝑡 ) are concatenated as SM input vector 𝑖 ( 𝑡 ) to trigger o ( t ) 
Revised: 𝑎 ( 𝑡 ) and s ( 𝑡 ) are concatenated as SM input vector 𝑖 ( 𝑡 ) to trigger o ( t ) i ( t )

Original: [Table] [Equation]
Revised: = { s ( t ) ; 𝑎 ( 𝑡 )} .

Original:  ; is to vertically concatenate vectors.
Revised: (1) ; is to vertically concatenate vectors.

Original: For o ( t ) o ( t ) =
Revised: For o ( t ) o ( t ) =

Original: W 𝑆𝑀 i ( t ) 
Revised: W 𝑆𝑀 i ( t ) .

Original: [Equation]

Original: We would like to explore the borderline of normal operation of services regarding fault injection, which means higher failure rate  (as Table 2 shows)
Revised: We would like to explore the borderline of normal operation of services regarding fault injection, which means higher failure rate [Table] [Table] (as Table 2 shows)

Original: 𝑃 503 ( 𝑡 ) but not the full failure rate is preferred .
Revised: 𝑃 503 ( 𝑡 ) but not the full failure rate is preferred .

Original: At the same time, higher QPS ( 𝑞 ( 𝑡 ) ) is beneficial to improve the networking efficiency.
Revised: At the same time, higher QPS ( 𝑞 ( 𝑡 ) ) is beneficial to improve the networking efficiency.

Original: So the definition of 503 reward is [Equation]
Revised: So the definition of 503 reward is 𝑟 503 ( 𝑡 ) =

Revised: 𝑞 ( 𝑡 ) · 𝑃 503 ( 𝑡 ) .

Original: 5. Single Agent for Single Service
Revised: 5. Single Agent for Single Service

Original: Firstly, we demonstrate the simplest case that only one agent and one simulation model interact with each other.
Revised: Firstly, we demonstrate the simplest case that only one agent and one simulation model interact with each other.

Original: In this case, only one kind of action (number of threads or calls) is decided by 𝐴𝑔 ( 𝑡 ) .
Revised: In this case, only one kind of action (number of threads or calls) is decided by 𝐴𝑔 ( 𝑡 ) .

Original: Given a preset traffic rule s ( t ) , agent 𝐴𝑔 ( 𝑡 ) takes it as input state and make actions 𝑎 ( 𝑡 ) .
Revised: Given a preset traffic rule s ( t ) , agent 𝐴𝑔 ( 𝑡 ) takes it as input state and make actions 𝑎 ( 𝑡 ) .

Original: After that, i ( t ) integrates both input (traffic rules + one loading testing) and output (optimized loading testing) of 𝐴𝑔 ( 𝑡 ) and triggers application responses (QPS + failure rate) o ( t ) .
Revised: After that, i ( t ) integrates both input (traffic rules + one loading testing) and output (optimized loading testing) of 𝐴𝑔 ( 𝑡 ) and triggers application responses (QPS + failure rate) o ( t ) .

Original: Finally, 𝑟 ( 𝑡 ) is formulated from o ( t ) and update the Q-network of 𝐴𝑔 ( 𝑡 ) .
Revised: Finally, 𝑟 ( 𝑡 ) is formulated from o ( t ) and update the Q-network of 𝐴𝑔 ( 𝑡 ) .

Original: The policy of the single agent is 𝜋 𝜃 ( 𝑡 ) =
Revised: The policy of the single agent is 𝜋 𝜃 ( 𝑡 ) =

Original: 𝑎 ( 𝑡 ) , and the Q-factor is 𝑄 ( s ,𝑎 ) =
Revised: 𝑎 ( 𝑡 ) , and the Q-factor is 𝑄 ( s ,𝑎 ) =

Original: 𝐸 [ 𝑟 ( 𝑡 + 1 ) ,𝑟 ( 𝑡 + 2 ) , ... | 𝑆 ( 𝑡 ) = s ,𝐴 𝑡 = 𝑎 ] , RL model is 𝑚 ( s ,𝑎 ) =
Revised: 𝐸 [ 𝑟 ( 𝑡 + 1 ) ,𝑟 ( 𝑡 + 2 ) , ... | 𝑆 ( 𝑡 ) = s ,𝐴 𝑡 = 𝑎 ] , RL model is 𝑚 ( s ,𝑎 ) =

Original: 𝐸 [ 𝑆 ( 𝑡 + 1 )| 𝑆 ( 𝑡 ) = s ,𝐴 𝑡 = 𝑎 ] .
Revised: 𝐸 [ 𝑆 ( 𝑡 + 1 )| 𝑆 ( 𝑡 ) = s ,𝐴 𝑡 = 𝑎 ] .

Original: Our goal is to maximize the performance function 𝐽 ( 𝜃 ) =
Revised: Our goal is to maximize the performance function 𝐽 ( 𝜃 ) =

Original: 𝐸 [ 𝑟 ( 1 ) + 𝛼𝑟 ( 2 ) + 𝛼 2 𝑟 ( 3 ) + ... | 𝜋 ( 𝜃 )] , where 𝛼 is discounted coefficient.
Revised: 𝐸 [ 𝑟 ( 1 ) + 𝛼𝑟 ( 2 ) + 𝛼 2 𝑟 ( 3 ) + ... | 𝜋 ( 𝜃 )] , where 𝛼 is discounted coefficient.

Original: In the implementation, the agent will search through all the actions for a given state and select the state-action pair with the highest corresponding Q-factor using
Revised: In the implementation, the agent will search through all the actions for a given state and select the state-action pair with the highest corresponding Q-factor using

Original: Bellman Equation [42].
Revised: Bellman Equation [42].

Original: The update of Q-network depends on the square error between the Q-factor that estimated by agents and actual rewards received from SM.
Revised: The update of Q-network depends on the square error between the Q-factor that estimated by agents and actual rewards received from SM.

Original: The policy gradient for long term
Revised: The policy gradient for long term

Original: [Equation]
Revised: [Equation]

Original: where 𝛾 is the sequential action-state pair trace in time order: { s ( 0 ) ,𝑎 ( 0 ) , s ( 1 ) ,𝑎 ( 1 ) , ..., s ( 𝑡 − 1 ) ,𝑎 ( 𝑡 − 1 )} , 𝑅 ( 𝛾 ) is the reward function across the trace and ∇ 𝐽 ( 𝜃 ) is the gradient used for network update.
Revised: where 𝛾 is the sequential action-state pair trace in time order: { s ( 0 ) ,𝑎 ( 0 ) , s ( 1 ) ,𝑎 ( 1 ) , ..., s ( 𝑡 − 1 ) ,𝑎 ( 𝑡 − 1 )} , 𝑅 ( 𝛾 ) is the reward function across the trace and ∇ 𝐽 ( 𝜃 ) is the gradient used for network update.

Original: The single agent for single service is summarized in Algorithm 1.
Revised: The single agent for single service is summarized in Algorithm 1.

Revised: Input: s s

Revised: Round 1 to Round 𝑡 do

Original: 5. Multi-agents for Single Service
Revised: 5. Multi-agents for Single Service

Original: Section 5.1 illustrates the basic case for policy learning.
Revised: Section 5.1 illustrates the basic case for policy learning.

Original: However, there are two decided loading settings to tune the resiliency together.
Revised: However, there are two decided loading settings to tune the resiliency together.

Original: Following the previous settings, we extend the scenario into multi-agent interactions and define two kinds of collaborative relationships between two agents, which are responsible for two loading decisions, respectively.
Revised: Following the previous settings, we extend the scenario into multi-agent interactions and define two kinds of collaborative relationships between two agents, which are responsible for two loading decisions, respectively.

Original: Denote 𝑎 1 ( 𝑡 ) as the action taken by 𝐴𝑔 1 ( 𝑡 ) , 𝑎 2 ( 𝑡 ) as the action taken by 𝐴𝑔 2 ( 𝑡 ) .
Revised: Denote 𝑎 1 ( 𝑡 ) as the action taken by 𝐴𝑔 1 ( 𝑡 ) , 𝑎 2 ( 𝑡 ) as the action taken by 𝐴𝑔 2 ( 𝑡 ) .

Original: s 1 ( 𝑡 ) , s 2 ( 𝑡 ) are respective state vectors, i ( t ) , o ( t ) , 𝑟 ( 𝑡 ) maintain the same definition as before.
Revised: s 1 ( 𝑡 ) , s 2 ( 𝑡 ) are respective state vectors, i ( t ) , o ( t ) , 𝑟 ( 𝑡 ) maintain the same definition as before.

Original: Two agents share the same reward for ∇ 𝐽 ( 𝜃 1 ) , ∇ 𝐽 ( 𝜃 2 ) ,𝜃 1 ,𝜃are RL Q-network parameters.
Revised: Two agents share the same reward for ∇ 𝐽 ( 𝜃 1 ) , ∇ 𝐽 ( 𝜃 2 ) ,𝜃 1 ,𝜃are RL Q-network parameters.

Original: 5.2.1 Independent Collaboration.
Revised: 5.2.1 Independent Collaboration.

Original: The definition on "Independent" (Int.) originates from state vectors for two agents.
Revised: The definition on "Independent" (Int.) originates from state vectors for two agents.

Original: In this scenario, 𝐴𝑔 1 ( 𝑡 ) and 𝐴𝑔 2 ( 𝑡 ) take the same state vector s ( 𝑡 ) with traffic rules only and make actions in parallel: 𝜋 𝜃 1 ( 𝑡 ) =
Revised: In this scenario, 𝐴𝑔 1 ( 𝑡 ) and 𝐴𝑔 2 ( 𝑡 ) take the same state vector s ( 𝑡 ) with traffic rules only and make actions in parallel: 𝜋 𝜃 1 ( 𝑡 ) =

Original: 𝑎 1 ( 𝑡 ) and 𝜋 𝜃 2 ( 𝑡 ) =
Revised: 𝑎 1 ( 𝑡 ) and 𝜋 𝜃 2 ( 𝑡 ) =

Original: 𝑎 2 ( 𝑡 ) .
Revised: 𝑎 2 ( 𝑡 ) .

Original: After both actions are made, the i ( t ) is generated as i ( 𝑡 ) = { s ( 𝑡 ) ;
Revised: After both actions are made, the i ( t ) is generated as i ( 𝑡 ) = { s ( 𝑡 ) ;

Original: 𝑎 1 ( 𝑡 ) ; 𝑎 2 ( 𝑡 )} .
Revised: 𝑎 1 ( 𝑡 ) ; 𝑎 2 ( 𝑡 ) } .

Revised: [Equation]

Original: After obtaining i ( 𝑡 ) , the following interaction with SM aligns with Section 4.1.
Revised: After obtaining i ( 𝑡 ) , the following interaction with SM aligns with Section 4.1.

Original: 5.2.2 Dependent Collaboration.
Revised: 5.2.2 Dependent Collaboration.

Original: The definition on "Dependent" (Det.) also drives from the state vector for two agents.
Revised: The definition on "Dependent" (Det.) also drives from the state vector for two agents.

Original: Two agents are executed in order and the state for latter agent combines the action with the state of former agent to take the second action.
Revised: Two agents are executed in order and the state for latter agent combines the action with the state of former agent to take the second action.

Original: Assume 𝐴𝑔 1 ( 𝑡 ) is the first executed agent, 𝐴𝑔 2 ( 𝑡 ) is the second executed agent, state vector s 2 ( 𝑡 ) relies on 𝑎 1 ( 𝑡 ) and s 1 ( 𝑡 ) s 2 ( 𝑡 )
Revised: Assume 𝐴𝑔 1 ( 𝑡 ) is the first executed agent, 𝐴𝑔 2 ( 𝑡 ) is the second executed agent, state vector s 2 ( 𝑡 ) relies on 𝑎 1 ( 𝑡 ) and s 1 ( 𝑡 ) s 2 ( 𝑡 )

Original: = { s 1 ( 𝑡 ) ; 𝑎 1 ( 𝑡 )} .
Revised: = { s 1 ( 𝑡 ) ; 𝑎 1 ( 𝑡 )} .

Original: Similarly, the application response is i ( t ) =
Revised: Similarly, the application response is i ( t ) =

Original: { s 2 ( t ) ; 𝑎 2 ( 𝑡 )} .
Revised: { s 2 ( t ) ; 𝑎 2 ( 𝑡 )} .

Original: We summarize the multi-agent algorithm in Algorithm 2.
Revised: We summarize the multi-agent algorithm in Algorithm 2.

Original: T ( 𝑡 ) is denoted as traffic rule vector at timestamp 𝑡 .
Revised: T ( 𝑡 ) is denoted as traffic rule vector at timestamp 𝑡 .

Original: All types of agent interdenpedencies are listed in Table 3.
Revised: All types of agent interdenpedencies are listed in Table 3.

Original: 5. Communicative Multi-agents for Multi-services
Revised: Algorithm

Original: In light of real-world web applications subject to collaboration of all aggregated services to fulfill functions, communications are necessary to know how other services operate in the same stream.
Revised: Input: T ( 1 ) , ..., T ( 𝑡 ) from Round 1 to Round 𝑡 1 for T ( 𝑡 ) do if Mode =

Original: At this part, we extend our exploration that multiple services are optimized by multi-agents. However, decentralized only Q-network mandates the iterative decision execution for information sharing, which leads to serious communication latency when high volume of requests.
Revised: Int.

Original: As a consequence, we enable individual RL agents to efficiently communicate in the beginning phase so that all policy 𝑡 = 𝑡 + 1. are the first 7 listed in Table 1.
Revised: then 𝑎 1 ( 𝑡 ) ← 𝜋 𝜃 1 ( 𝑡 ) , 𝑎 2 ( 𝑡 ) ← 𝜋 𝜃 2 ( 𝑡 ) separately; Generate i ( t ) as Equation (

Original: Thread&Call represent two agents take actions independently, Thread-Call or CallThread represent the actions are decided in order. The former one takes action firstly and passes it to the latter one.
Revised: ; if Mode = Det. then are the two agents take actions independently, Thread-Call or CallThread represent the actions are decided in order takes action firstly and passes it to the latter one.

Original: Agent
Revised: [Table]

Original: State
Revised: State

Original: Action (in order)
Revised: [Table] (in order)

Original: Thread Agent 7 Traffic rules + Calls
Revised: [Table] Agent 7 Traffic rules + Calls

Original: Threads
Revised: Threads

Original: Call Agent 7 Traffic Rules + Threads
Revised: Call Agent 7 Traffic Rules + Threads

Original: Calls
Revised: Calls

Original: Thread&Call 7 Traffic Rules for both
Revised: Thread&Call 7 Traffic Rules for both

Original: Threads, Calls
Revised: Threads, Calls

Original: Thread-Call 7 Rules-7 Rules + Threads Threads, Calls Call-Thread 7 Rules-7 Rules + Calls 
Revised: Thread-Call 7 Rules-7 Rules + Threads Threads, Calls Call-Thread 7 Rules-7 Rules + Calls Calls, Threads

Revised: 5. Communicative Multi-agents for Multi-services

Original: Calls, Threads learnings can obtain better summed rewards with low time complexity.
Revised: In light of real-world web applications subject to collaboration of all aggregated services to fulfill functions, communications are necessary to know how other services operate in the same stream. At this part, we extend our exploration that multiple services are optimized by multi-agents. However, decentralized only Q-network mandates the iterative decision execution for information sharing, which leads to serious communication latency when high volume of requests. As a consequence, we enable individual RL agents to efficiently communicate in the beginning phase so that all policy learnings can obtain better summed rewards with low time complexity.

Original: [10,22] propose the theoretical foundation of homogeneous agent cooperation through sharing their model parameters, allowing model to be trained with experiences of all agents, which makes the algorithm data efficient.
Revised: [10,22] propose the theoretical foundation of homogeneous agent cooperation through sharing their model parameters, allowing model to be trained with experiences of all agents, which makes the algorithm data efficient.

Original: Inspired by their work, we design that partial learning is centralized while the left learning is decentralized.
Revised: Inspired by their work, we design that partial learning is centralized while the left learning is decentralized.

Original: The agents take different actions via searching through their own action spaces, receive rewards and update two split networks.
Revised: The agents take different actions via searching through their own action spaces, receive rewards and update two split networks.

Revised: firstly, then go through individual PNet and search through their own action space to receive rewards

Original: We define the sharable network as SNet with input layer and its weight is 𝜃 𝑠 , decentralized network (or say private) as PNet with hidden and output layer and their weights are 𝜃 𝑝 1 , ...,𝜃 𝑝𝑛 ,𝑖 is the index of service.
Revised: We define the sharable network as SNet with input layer and its weight is 𝜃 𝑠 , decentralized network (or say private) as PNet with hidden and output layer and their weights are 𝜃 𝑝 1 , ...,𝜃 𝑝𝑛 ,𝑖 is the index of service.

Original: As Figure 4 shows, all states from all agents pass
Revised: As Figure 4 shows, all states from all agents pass

Original: SNet firstly and go through their own PNet for decision-makings.
Revised: Input T 1 ( 1 ) , ... 𝑡 n from Round Round

Original: For the purpose of optimizing the holistic "worst-case" resiliency, the reward is defined as
Revised: For the purpose of optimizing the holistic "worst-case" resiliency, the reward is defined as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where 𝛽 is the coefficient.
Revised: where 𝛽 is the coefficient.

Original: After rewards are generated for each one, the respective PNet will be updated by corresponding rewards and Q-factor pair, SNet will be updated by all pairs from all service agents.
Revised: After rewards are generated for each one, the respective PNet will be updated by corresponding rewards and Q-factor pair, SNet will be updated by all pairs from all service agents.

Original: As a consequence, the policy of each agent is relevant to 𝜃 𝑝𝑛firstly, then go through individual PNet and search through their own action space to receive rewards. Input: T 1 ( 1 ) , ..., T n ( 1 ) ..., T 1 ( 𝑡 ) , ..., T n ( 𝑡 ) from Round Round 𝑡 for 𝑛 services do and 𝜋 𝜃 𝑠 ,𝜃 𝑝𝑛 ( 𝑡 ) =
Revised: As a consequence, the policy of each agent is relevant to 𝜃 𝑝𝑛 and 𝜋 𝜃 𝑠 ,𝜃 𝑝𝑛 ( 𝑡 ) =

Original: 𝑎 𝑛 ( 𝑡 ) .
Revised: 𝑎 𝑛 ( 𝑡 ) .

Original: The long-term policy gradient for PNet 𝑛 
Revised: The long-term policy gradient for PNet 𝑛 ∇ 𝐽 ( 𝜃 𝑝𝑛 ) =

Revised: 𝐸 𝛾 𝑛 [∇ 𝜃 𝑝𝑛 𝑡 − 1 ∑︁ 𝑡 =𝑙𝑜𝑔𝜋 ( 𝑎 𝑛 ( 𝑡 )| ms n

Original: [Equation]
Revised: [Equation]

Original:  where ms n ( 𝑡 ) is the output vector of SNet and the input of PNet.
Revised: 𝑡 [Equation] [Equation] (9) where ms n ( 𝑡 ) is the output vector of SNet and the input of PNet.

Original: The prediction function ˆ 𝑓 𝜃 𝑠 of SNet is represented by 
Revised: The prediction function ˆ 𝑓 𝜃 𝑠 of SNet is represented by ˆ s ( 𝑡 + 1 )

Original: [Equation]
Revised: = ˆ 𝑓 𝜃 𝑠

Original:  Updating 𝜃 𝑠 is to find the MSE minimizer of predicted ˆ s ( 𝑡 + 1 ) and s ( 𝑡 +1 ) 𝜃 𝑠 = argmin
Revised: ( s n ( 𝑡 ) , ms n ( 𝑡 )) . Updating 𝜃 𝑠 is to find the MSE minimizer of predicted ˆ s ( 𝑡 + 1 ) and s ( 𝑡 +

Original: 𝜃 𝑠 (cid:205)

Original: D (cid:13)(cid:13)(cid:13) s n ( 𝑡 + 1 ) −

Original: [Equation]
Revised: [Equation]

Original: 𝑓 𝜃 𝑠 ( s ( 𝑡 ) , ms n ( 𝑡 )) (cid:13)(cid:13)(cid:13)| D | , (11) where training data s n ( 𝑡 ) , ms n ( 𝑡 ) , s n ( 𝑡 + 1 ) ∈ D for all agents.
Revised: where training data s n ( 𝑡 ) , ms n ( 𝑡 ) , s n ( 𝑡 + 1 ) ∈ D for all agents.

Original: As shown in figure 5, the agent can firstly efficiently communicate based on shared experienced and then optimize their own decisions combining with individual case and holistic circumstances.
Revised: As shown in figure 5, the agent can firstly efficiently communicate based on shared experienced and then optimize their own decisions combining with individual case and holistic circumstances.

Original: If both kinds of actions are decided by agents, only the same type of agents communicate over all services (call agents only communicate with other call agents, thread agent only communicate with other agents etc.
Revised: If both kinds of actions are decided by agents, only the same type of agents communicate over all services (call agents only communicate with other call agents, thread agent only communicate with other agents etc.

Original: The collaborative relationship for agents within one service have accordance with Section 5.2).
Revised: The collaborative relationship for agents within one service have accordance with Section 5.2).

Original: The learning paradigm for multi-services is summarized in Algorithm 3 and visualized in Figure 6.
Revised: The learning paradigm for multi-services is summarized in Algorithm 3 and visualized in Figure 6.

Original: The implementation of action space searching and PNet refreshment can be executed in parallel, only the SNet updates require iterative steps.
Revised: The implementation of action space searching and PNet refreshment can be executed in parallel, only the SNet updates require iterative steps.

Original:  EXPERIMENTS 6. Simulation Model Evaluation
Revised:  EXPERIMENTS 6. Simulation Model Evaluation

Original: [11] summarizes most common ways in modelling networking communications, among which Logistic Regression, Linear Ridge
Revised: [11] summarizes most common ways in modelling networking communications, among which Logistic Regression, Linear Ridge

Original: Regression and Support Vector Regression are highlighted due to their outperformed simulating abilities.
Revised: Regression and Support Vector Regression are highlighted due to their outperformed simulating abilities.

Original: We select these three methods as baseline simulation models to fight against 5 layerMLP and see which emulates the application response (in our case study, the Istio httpbin service) best.
Revised: We select these three methods as baseline simulation models to fight against 5 layerMLP and see which emulates the application response (in our case study, the Istio httpbin service) best.

Original: We collect 5 groups of structured datasets and split them into 8: as training and testing set, respectively.
Revised: We collect 5 groups of structured datasets and split them into 8: as training and testing set, respectively.

Original: For MLP, the input layer has 9 neurons, 3 hidden layers have 512 neurons and the output layer has 2 neurons.
Revised: For MLP, the input layer has 9 neurons, 3 hidden layers have 512 neurons and the output layer has 2 neurons.

Original: The learning rate varies from 1 − 6 to 1 − 5 .
Revised: The learning rate varies from 1 − 6 to 1 − 5 .

Original: The range for the traffic rule and thread and call settings we used in our experiments is in Table 4.
Revised: The range for the traffic rule and thread and call settings we used in our experiments is in Table 4.

Original: Four types of models are trained and MLP has the best performance according to Table 4, hence we use it in our approach.
Revised: Four types of models are trained and MLP has the best performance according to Table 4, hence we use it in our approach.

Original: 6. Policy Evaluation
Revised: 6. Policy Evaluation

Original: We explore the ”worst-case” rewards, as defined in Section 5.
Revised: We explore the ”worst-case” rewards, as defined in Section 5.

Original: "worstcase" rewards (penalties) provide insight into configuration settings that are critical to use in load testing prior to application deployment.
Revised: "worstcase" rewards (penalties) provide insight into configuration settings that are critical to use in load testing prior to application deployment.

Original: The learning ability is characterized by how much RL algorithm outperform baselines under the same context, whose metrics is the maximum rolling mean ratio of cumulative reward obtained in RL to baseline in last 25 epochs.
Revised: The learning ability is characterized by how much RL algorithm outperform baselines under the same context, whose metrics is the maximum rolling mean ratio of cumulative reward obtained in RL to baseline in last 25 epochs.

Revised: In simulation session, we implement the simulation model to interact with RL agent(s) and record activities at the epoch with the highest reward ratio.

Revised: Then data points at the best epoch work as the loading and traffic parameters to trip the actual service response and obtain the validated reward ratio.

Revised: For all experiments, we analyze single agent and multiple/collaborative agent model-based reinforcement learning using our algorithm and implementation from Section 4.

Revised: Every experiment is performed for 3 times and we present the mean values of 3 results.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Val.
Revised: Val.

Original: Single for Single
Revised: Single for Single

Original: Call 1.
Revised: Call 1.

Original: Thread 2.
Revised: Thread 2.

Original: Multi for Single
Revised: Multi for Single

Original: Thread&Call 2.
Revised: Thread&Call 2.

Original: Thread-Call 2.
Revised: Thread-Call 2.

Original: Call-Thread 2.
Revised: Call-Thread 2.

Original: Multi for Multi
Revised: Multi for Multi

Original: Call* 1.
Revised: Call* 1.

Original: Thread* 2.
Revised: Thread* 2.

Original: Thread&Call* 2.
Revised: Thread&Call* 2.

Original: Thread-Call* 2. Call-Thread* 2.
Revised: Thread-Call* 2. Call-Thread* 2.

Original: In simulation session, we implement the simulation model to interact with RL agent(s) and record activities at the epoch with the highest reward ratio.

Original: Then data points at the best epoch work as the loading and traffic parameters to trip the actual service response and obtain the validated reward ratio.

Original: For all experiments, we analyze single agent and multiple/collaborative agent model-based reinforcement learning using our algorithm and implementation from Section 4.

Original: Every experiment is performed for 3 times and we present the mean values of 3 results.

Original: We compare the selection of configuration setting values generated by our algorithm to baselines in Table 5.
Revised: We compare the selection of configuration setting values generated by our algorithm to baselines in Table 5.

Original: In terms of RL experiment, we use our deep model-based reinforcement learning algorithm to identify the configuration settings of threads and calls over time, over variations of the other traffic rule settings as per Table 1.
Revised: In terms of RL experiment, we use our deep model-based reinforcement learning algorithm to identify the configuration settings of threads and calls over time, over variations of the other traffic rule settings as per Table 1.

Original: Collaborative agent experiment includes either call or thread selection and both thread and call selection with 5 service aggregated in applications.
Revised: Collaborative agent experiment includes either call or thread selection and both thread and call selection with 5 service aggregated in applications.

Original: For baseline experiment, there is no exact relevant machine learning methods addressing fault resiliency, so we randomly and evenly pick the action and see reward difference gained by non-monitoring and monitoring (RL) groups.
Revised: For baseline experiment, there is no exact relevant machine learning methods addressing fault resiliency, so we randomly and evenly pick the action and see reward difference gained by non-monitoring and monitoring (RL) groups.

Original: 6. Result Analysis
Revised: 6. Result Analysis

Original: From Table 5, it is clear that model-based RL algorithm is able to outperform baselines in most cases and learning effects are validated in actual Istio API.
Revised: From Table 5, it is clear that model-based RL algorithm is able to outperform baselines in most cases and learning effects are validated in actual Istio API.

Original: Although there are errors between simulations and validations, the ratio trends are almostly aligned.
Revised: Although there are errors between simulations and validations, the ratio trends are almostly aligned.

Original: The factor of thread/call has different influence significance on the policy learning: thread is more significant for S1 and call is more significant for
Revised: The factor of thread/call has different influence significance on the policy learning: thread is more significant for S1 and call is more significant for

Original: S2 - S5.
Revised: S2 - S5.

Original: To be extended, multi-agents learning abilities are closed to single agent when either factor is very trivial to fault resiliency, such as thread in S2, S5 and call in S1.
Revised: To be extended, multi-agents learning abilities are closed to single agent when either factor is very trivial to fault resiliency, such as thread in S2, S5 and call in S1.

Original: The difference derives from the parameter range setting in simulation model training, and our approach is also beneficial to help clarify the significant factor for the service with specific traffic volume.
Revised: The difference derives from the parameter range setting in simulation model training, and our approach is also beneficial to help clarify the significant factor for the service with specific traffic volume.

Original: In terms of single service cases, most of multi-agents work better than single agent decisions, which proves that complex parameter interdependence optimization can fulfill the potential of policy learning.
Revised: In terms of single service cases, most of multi-agents work better than single agent decisions, which proves that complex parameter interdependence optimization can fulfill the potential of policy learning.

Original: For instance, Thread&Call agents gain 27% higher rewards than Call only (3.45 to 2.71) agent in simulation and 69% higher rewards (2.80 to 1.77) in validation in S2.
Revised: For instance, Thread&Call agents gain 27% higher rewards than Call only (3.45 to 2.71) agent in simulation and 69% higher rewards (2.80 to 1.77) in validation in S2.

Original: What is more important, multi-agents usually have higher validation accuracy than single agent .
Revised: What is more important, multi-agents usually have higher validation accuracy than single agent .

Original: Take S1 as an example, Thread only agent has 2.
Revised: Take S1 as an example, Thread only agent has 2.

Original: reward ratio in simulation and 1.63 in validation (36% higher), but
Revised: reward ratio in simulation and 1.63 in validation (36% higher), but

Original: Thread&Call agent has closed 2.26 reward ratio and more accurate 2.15 validated ratio (5% higher).
Revised: Thread&Call agent has closed 2.26 reward ratio and more accurate 2.15 validated ratio (5% higher).

Original: Similarly, Call only agent has 2.
Revised: Similarly, Call only agent has 2.

Original: in simulation and 1.77 in validation (53% higher), Thread&Call has 3.45 in simulation and 2.80 in validation (23% higher) in S2.
Revised: in simulation and 1.77 in validation (53% higher), Thread&Call has 3.45 in simulation and 2.80 in validation (23% higher) in S2.

Revised: We can see that learnings are fast and then keep volatile for all cases, but multi-services with communications have more stable trends against single service.

Revised: The best epoch has been reached before 150th epoch.

Original: Regardingthedifferencebetweensingleserviceandmulti-services, it is apparent that the reward ratios obtained by all 5 services are higher than the corresponding single case and have more stable learning trends against single service, which strongly supports the importance of collaborative decision making for aggregated services resiliency, especially they experience closed loading volumes in real world.
Revised: Regardingthedifferencebetweensingleserviceandmulti-services, it is apparent that the reward ratios obtained by all 5 services are higher than the corresponding single case and have more stable learning trends against single service, which strongly supports the importance of collaborative decision making for aggregated services resiliency, especially they experience closed loading volumes in real world.

Original: For S1, S2, multi-services outperform single-service in corresponding cases for both simulation and validation.
Revised: For S1, S2, multi-services outperform single-service in corresponding cases for both simulation and validation.

Original: For S3, S and S5, even if aggregated services behave closely to corresponding single service in simulation, the validating reward ratios are much higher.
Revised: For S3, S and S5, even if aggregated services behave closely to corresponding single service in simulation, the validating reward ratios are much higher.

Original: For example, we could observe that even if the reward ratios of Thread only and Call only agent in single and multiple services cases are very closed (1.13 to 1.18, 1.75 to 1.77) in S3 simulation, but
Revised: For example, we could observe that even if the reward ratios of Thread only and Call only agent in single and multiple services cases are very closed (1.13 to 1.18, 1.75 to 1.77) in S3 simulation, but

Original: S3 validating results reveal that collaborative services have better results (1.06 to 0.98, 1.43 to 1.31).
Revised: S3 validating results reveal that collaborative services have better results (1.06 to 0.98, 1.43 to 1.31).

Original: In terms of MLP approximation accuracy, Table 4 indicates that
Revised: In terms of MLP approximation accuracy, Table 4 indicates that

Original: MLP has variant emulating capabilities on the behavior of service resiliency, among which S1, S2 have the lowest MSE values (0.13, 0.17).
Revised: MLP has variant emulating capabilities on the behavior of service resiliency, among which S1, S2 have the lowest MSE values (0.13, 0.17).

Original: However, the difference between simulation and validation results do not strongly related to the simulating MSE value, modelbased RL still provides optimized decisions on loading setting for
Revised: However, the difference between simulation and validation results do not strongly related to the simulating MSE value, modelbased RL still provides optimized decisions on loading setting for

Original: S3 and S4 although their MLP simulating MSE values (0.52, 0.63) are relatively higher out of all groups.
Revised: S3 and S4 although their MLP simulating MSE values (0.52, 0.63) are relatively higher out of all groups.

Original: For model-based learning, the simulating model is not required to describe behaviors very precisely, in other words, we just need to know the estimated reactions of services given a specific context to enable continuous agent-environment interaction and policy learning.
Revised: For model-based learning, the simulating model is not required to describe behaviors very precisely, in other words, we just need to know the estimated reactions of services given a specific context to enable continuous agent-environment interaction and policy learning.

Original: We think the experiments are also enlightening to apply model-based RL to many other fields when the real-time online interaction is expensive or unrealistic.
Revised: We think the experiments are also enlightening to apply model-based RL to many other fields when the real-time online interaction is expensive or unrealistic.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: Our model-based reinforcement learning algorithm can help predict which values of the traffic rule settings and threads and calls yield rewards with respect to fault resiliency of the Istio h ttpbin service.
Revised: Our model-based reinforcement learning algorithm can help predict which values of the traffic rule settings and threads and calls yield rewards with respect to fault resiliency of the Istio h ttpbin service.

Original: We comprehensively investigate how model-based RL can help optimize the parametric spaces of single services, what kind of relationships between different variable agents are suitable given specific settings and how to employ service cluster using collaborative agents to communicate with each other efficiently.
Revised: We comprehensively investigate how model-based RL can help optimize the parametric spaces of single services, what kind of relationships between different variable agents are suitable given specific settings and how to employ service cluster using collaborative agents to communicate with each other efficiently.

Original: Inparticular,theconfigurationsettingsthatyield the“worst-case” rewards give insight into which combinations of configurations should be tested rigorously during load testing to ensure robust fault recovery, as these may significantly compromise applicationlevel fault resiliency.
Revised: Inparticular,theconfigurationsettingsthatyield the“worst-case” rewards give insight into which combinations of configurations should be tested rigorously during load testing to ensure robust fault recovery, as these may significantly compromise applicationlevel fault resiliency.

Original: Regarding the dynamic operating status of microservice networking, the validation on SFR2L is crucial to approve the performance of our designs.
Revised: Regarding the dynamic operating status of microservice networking, the validation on SFR2L is crucial to approve the performance of our designs.

Original: Our research is promising to the maintenance of serve mesh-based architecture in industry and construct an effective workflow to demonstrate the optimal state for a service or cluster of services.
Revised: Our research is promising to the maintenance of serve mesh-based architecture in industry and construct an effective workflow to demonstrate the optimal state for a service or cluster of services.

Original:  ADDITIONAL EXPERIMENT INFORMATION 8. Simulation Model Training
Revised:  ADDITIONAL EXPERIMENT INFORMATION 8. Simulation Model Training

Original: The parameter range setting for our data collection has the following criterion:
Revised: The parameter range setting for our data collection has the following criterion:

Original: The failure rate and success rate distribution is relatively equivalent for whole dataset. 2)
Revised: The failure rate and success rate distribution is relatively equivalent for whole dataset

Original: Some parameters are fixed to see better optimization on other parameters.
Revised: Some parameters are fixed to see better optimization on other parameters.

Original: Interval Time and Max Ejection Rate are constant in our experiments in terms of no complex topology within single httpbin service.
Revised: Interval Time and Max Ejection Rate are constant in our experiments in terms of no complex topology within single httpbin service.

Original: 3) Settings are closed to cases in industrialized practice.
Revised: 3) Settings are closed to cases in industrialized practice.

Original: The number of calls are set from 50 to 2000, which covers broad ranges of loading testing
Revised: The number of calls are set from 50 to 2000, which covers broad ranges of loading testing.

Original: 8. Training Details
Revised: 8. Training Details

Original: We conduct our experiment for 500 epochs, in which each epoch contains 1000 simulation interactions.
Revised: We conduct our experiment for 500 epochs, in which each epoch contains 1000 simulation interactions.

Original: For the network of single agent in Section 5.1 and Section 5.2, it contains a input layer and a output layer ,and another 2 hidden layers with 512 neurons.
Revised: For the network of single agent in Section 5.1 and Section 5.2, it contains a input layer and a output layer ,and another 2 hidden layers with 512 neurons.

Original: The learning rate is 5 × 10 − 5 , dimension of single agent state vectors are uniformly set as 8 (7 rules + 1 loading setting), dependent multiagent state vectors are set as 7 and 8 dimensions, independent multi-agent state vectors are set as 7 dimensions.
Revised: The learning rate is 5 × 10 − 5 , dimension of single agent state vectors are uniformly set as 8 (7 rules + 1 loading setting), dependent multiagent state vectors are set as 7 and 8 dimensions, independent multi-agent state vectors are set as 7 dimensions.

Original: According to
Revised: According to

Original: Section 5.3, SNet has 2-layer 512 neurons in the end, PNet has ahidden neuron in the front.
Revised: Section 5.3, SNet has 2-layer 512 neurons in the end, PNet has ahidden neuron in the front.

Original: The learning rate is set as 1 × 10 − 5 .
Revised: The learning rate is set as 1 × 10 − 5 .

Original: All experiments use Adam optimizer in Pytorch library for gradient descents.
Revised: All experiments use Adam optimizer in Pytorch library for gradient descents.

Original: Theaction spaces are constrained by the datasets that used for training simulation model.
Revised: Theaction spaces are constrained by the datasets that used for training simulation model.

Original: The available actions are listed in Table
Revised: The available actions are listed in Table

Original: 6. LIMITATION
Revised: 6. LIMITATION

Original: Communication latency and efficiencies in services within web applications are subject to many factors, such as the stability of internet stability, the volume of concurrent visiting and internal properties of networking (bandwidth etc.).
Revised: Communication latency and efficiencies in services within web applications are subject to many factors, such as the stability of internet stability, the volume of concurrent visiting and internal properties of networking (bandwidth etc.).

Original: So the validation accuracy for simulation rely on the real-time networking transmission circumstances.
Revised: So the validation accuracy for simulation rely on the real-time networking transmission circumstances.

Original: In the meanwhile, the data collection process is not able to precisely reflect the real-time networking operation, but the substantial traces that we collect are capable of emulating the general responses of fault injections.
Revised: In the meanwhile, the data collection process is not able to precisely reflect the real-time networking operation, but the substantial traces that we collect are capable of emulating the general responses of fault injections.