Original: Online Learning and Pricing for Network Revenue Management with Reusable Resources
Revised: Online Learning and Pricing for Network Revenue Management with Reusable Resources

Original: Abstract
Revised: Abstract

Original: We consider a price-based network revenue management problem with multiple products and multiple reusable resources.
Revised: We consider a price-based network revenue management problem with multiple products and multiple reusable resources.

Original: Each randomly arriving customer requests a product (service) that needs to occupy a sequence of reusable resources (servers).
Revised: Each randomly arriving customer requests a product (service) that needs to occupy a sequence of reusable resources (servers).

Original: We adopt an incomplete information setting where the firm does not know the price-demand function for each product and the goal is to dynamically set prices of all products to maximize the total expected revenue of serving customers.
Revised: We adopt an incomplete information setting where the ﬁrm does not know the price-demand function for each product and the goal is to dynamically set prices of all products to maximize the total expected revenue of serving customers.

Original: We propose novel batched bandit learning algorithms for finding near-optimal pricing policies, and show that they admit a near-optimal cumulative regret bound of ˜ O
Revised: We propose novel batched bandit learning algorithms for ﬁnding near-optimal pricing policies, and show that they admit a near-optimal cumulative regret bound of ˜ O

Original: ( J √ XT ) , where J , X , and T are the numbers of products, candidate prices, and service periods, respectively.
Revised: ( J √ XT ) , where J , X , and T are the numbers of products, candidate prices, and service periods, respectively.

Original: As part of our regret analysis, we develop the first finite-time mixing time analysis of an open network queueing system (i.e., the celebrated Jackson Network), which could be of independent interest.
Revised: As part of our regret analysis, we develop the ﬁrst ﬁnite-time mixing time analysis of an open network queueing system (i.e., the celebrated Jackson Network), which could be of independent interest.

Original: Our numerical studies show that the proposed approaches perform consistently well.
Revised: Our numerical studies show that the proposed approaches perform consistently well.

Original:  Introduction
Revised:  Introduction

Original: Network revenue management (NRM) (Gallego and Van Ryzin 1997) has attained remarkable results in a wide range of applications including airline management, electricity pricing, retailing, healthcare, and leisure (see, e g., Williamson 1992, Talluri and Van Ryzin 2006, Özer et al. 2012).
Revised: Network revenue management (NRM) (Gallego and Van Ryzin 1997) has attained remarkable results in a wide range of applications including airline management, electricity pricing, retailing, healthcare, and leisure (see, e g., Williamson 1992, Talluri and Van Ryzin 2006, Özer and Phillips 2012).

Original: The bulk of existing literature focuses primarily on canonical settings with perishable resources in which the product once sold is permanently removed from inventory (Besbes and Zeevi 2012, Ferreira et al.
Revised: The bulk of existing literature focuses primarily on canonical settings with perishable resources in which the product once sold is permanently removed from inventory (Besbes and Zeevi 2012, Ferreira et al.

Original: 2018, Chen and Shi 2019).
Revised: 2018, Chen and Shi 2019).

Original: Little progress has been made for NRM with reusable resources, mainly due to the complexity of analyzing the underlying stochastic systems (typically manifested as stochastic queueing networks).
Revised: Little progress has been made for NRM with reusable resources, mainly due to the complexity of analyzing the underlying stochastic systems (typically manifested as stochastic queueing networks).

Original: In this paper, we develop easy-to-implement online learning and pricing policies for NRM with reusable resources shared by multiple products and analyze their theoretical and empirical performances.
Revised: In this paper, we develop easy-to-implement online learning and pricing policies for NRM with reusable resources shared by multiple products and analyze their theoretical and empirical performances.

Original: 1.1 Problem Description
Revised: 1.1 Problem Description

Original: A service-providing firm is endowed with multiple types of reusable resources, each having a fixed capacity.
Revised: A service-providing ﬁrm is endowed with multiple types of reusable resources, each having a ﬁxed capacity.

Original: The firm sells multiple types of products (services), each requiring the use of a sequence of different types of reusable resources (servers).
Revised: The ﬁrm sells multiple types of products (services), each requiring the use of a sequence of different types of reusable resources (servers).

Original: (See, e.g., Figure 2.)
Revised: (See, e.g., Figure 2.)

Original: Customers randomly arrive according to a price-dependent multivariate Poisson process to request products.
Revised: Customers randomly arrive according to a price-dependent multivariate Poisson process to request products.

Original: Upon requesting one particular product, a customer enters the system by going through a sequence of product-specified reusable resources one by one.
Revised: Upon requesting one particular product, a customer enters the system by going through a sequence of product-speciﬁed reusable resources one by one.

Original: During the service process, if there is no available next type of resource upon finishing using the previous type in the sequence, she will join the queue in front of this next type of resource and wait until the next available unit.
Revised: During the service process, if there is no available next type of resource upon ﬁnishing using the previous type in the sequence, she will join the queue in front of this next type of resource and wait until the next available unit.

Original: Both the mean demand rate of each product and the average usage time of each resource depend on the posted price vector.
Revised: Both the mean demand rate of each product and the average usage time of each resource depend on the posted price vector.

Original: However, such dependence is unknown to the firm a priori .
Revised: However, such dependence is unknown to the ﬁrm a priori .

Original: We note that under a fixed price vector, the service system can be reduced to the celebrated Jackson Network (Jackson 1963, 1957).
Revised: We note that under a ﬁxed price vector, the service system can be reduced to the celebrated Jackson Network (Jackson 1963, 1957).

Original: The main objective of this paper is to design a “learning-while-doing” algorithm to dynamically post prices of different products, in order to maximize the expected total revenue collected over a finite time horizon.
Revised: The main objective of this paper is to design a “learning-while-doing” algorithm to dynamically post prices of different products, in order to maximize the expected total revenue collected over a ﬁnite time horizon.

Original: It is evident that NRM with reusable resources finds many important real-world applications, including cloud computing, hospitality industry, project management, and leisure (Lei and Jasin 2020).
Revised: It is evident that NRM with reusable resources ﬁnds many important real-world applications, including cloud computing, hospitality industry, project management, and leisure (Lei and Jasin 2020).

Original: For example, firms such as Amazon and Microsoft offer diverse cloud computing services, each requiring a different set of computational resources (virtual machines) for a certain duration of time to finish their computational jobs (see, e.g., Kaewpuang et al. 2013, Püschel et al. 2015).
Revised: For example, ﬁrms such as Amazon and Microsoft offer diverse cloud computing services, each requiring a different set of computational resources (virtual machines) for a certain duration of time to ﬁnish their computational jobs (see, e.g., Kaewpuang et al. 2013, Püschel et al. 2015).

Original: 1.2 Main Results and Contributions
Revised: 1.2 Main Results and Contributions

Original: We propose new batched bandit algorithms for NRM with reusable resources (RNRM).
Revised: We propose new batched bandit algorithms for NRM with reusable resources (RNRM).

Original: The performance measure is regret, which is the revenue difference between our learning algorithms and a clairvoyant optimal pricing policy under full information.
Revised: The performance measure is regret, which is the revenue difference between our learning algorithms and a clairvoyant optimal pricing policy under full information.

Original: We use J, X, T to denote the numbers of products, candidate prices, and service periods, respectively.
Revised: We use J, X, T to denote the numbers of products, candidate prices, and service periods, respectively.

Original: We prove in Theorems 1 and 2 that the cumulative regret is ˜ O
Revised: We prove in Theorems 1 and 2 that the cumulative regret is ˜ O

Original: ( J √ XT ) , which matches the lower bound along the dimensions of X and T up to a logarithmic factor.
Revised: ( J √ XT ) , which matches the lower bound along the dimensions of X and T up to a logarithmic factor.

Original: It remains open whether one could improve the dependence on J .
Revised: It remains open whether one could improve the dependence on J .

Original: Note that all prior studies on non-reusable settings (e.g., Miao and Wang 2021) show polynomial dependence on J at best.
Revised: Note that all prior studies on non-reusable settings (e.g., Miao and Wang 2021) show polynomial dependence on J at best.

Original: We test diverse instances to show numerical performance of our approaches.
Revised: We test diverse instances to show numerical performance of our approaches.

Original: The main contribution of our work is three-fold: (Modeling.)
Revised: The main contribution of our work is three-fold: (Modeling.)

Original: To the best of our knowledge, we are the first to consider the joint learning and optimization problem in NRM with reusable resources.
Revised: To the best of our knowledge, we are the ﬁrst to consider the joint learning and optimization problem in NRM with reusable resources.

Original: Prior literature on reusable resources only focused on single-product settings (see Jia et al.
Revised: Prior literature on reusable resources only focused on single-product settings (see Jia et al.

Original: (2020, 2022) and references therein).
Revised: (2020, 2022) and references therein).

Original: It is considerably more challenging to model and analyze the network setting where multiple products share multiple resources (with finite capacities), mainly due to the sequential use of resources (to finish up a service) and the evolving nature of available capacities in real-time.
Revised: It is considerably more challenging to model and analyze the network setting where multiple products share multiple resources (with ﬁnite capacities), mainly due to the sequential use of resources (to ﬁnish up a service) and the evolving nature of available capacities in real-time.

Original: We establish a connection between our model and the celebrated Jackson Network (Jackson 1963, 1957).
Revised: We establish a connection between our model and the celebrated Jackson Network (Jackson 1963, 1957).

Original: (Algorithm.)
Revised: (Algorithm.)

Original: We first develop a linear programming (LP) upper bound on the optimal objective value given full information.
Revised: We ﬁrst develop a linear programming (LP) upper bound on the optimal objective value given full information.

Original: The LP model provides a static pricing control policy.
Revised: The LP model provides a static pricing control policy.

Original: Therefore, we can cast our RNRM as a multi-armed bandits (MAB) problem where we select among candidate prices (analogs of arms) to minimize the cumulative regret.
Revised: Therefore, we can cast our RNRM as a multi-armed bandits (MAB) problem where we select among candidate prices (analogs of arms) to minimize the cumulative regret.

Original: There are two key challenges.
Revised: There are two key challenges.

Original: The first lies in the fact that whenever the system posts a new price, the arrival processes change immediately while the system still contains “old” customers from the previously posted price.
Revised: The ﬁrst lies in the fact that whenever the system posts a new price, the arrival processes change immediately while the system still contains “old” customers from the previously posted price.

Original: Therefore, it is difficult to analyze the transient system performance and keep track of the cumulative revenue in a dynamic pricing scheme.
Revised: Therefore, it is difﬁcult to analyze the transient system performance and keep track of the cumulative revenue in a dynamic pricing scheme.

Original: We resolve this challenge by proposing batched algorithms.
Revised: We resolve this challenge by proposing batched algorithms.

Original: Our algorithms separate the time horizon into successive batches and select a price for each batch.
Revised: Our algorithms separate the time horizon into successive batches and select a price for each batch.

Original: We further divide each batch into two intervals.
Revised: We further divide each batch into two intervals.

Original: In the first interval, the system completes serving the previous customers and reaches the steady-state under the new price.
Revised: In the ﬁrst interval, the system completes serving the previous customers and reaches the steady-state under the new price.

Original: In the second interval, the system maintains the steady-state (exploiting the new price).
Revised: In the second interval, the system maintains the steady-state (exploiting the new price).

Original: This batched framework not only allows for system stabilization (under new prices) but also allows for infrequent O (log T ) price changes, which is more implementable in practical settings (see Cheung et al. 2017).
Revised: This batched framework not only allows for system stabilization (under new prices) but also allows for infrequent O (log T ) price changes, which is more implementable in practical settings (see Cheung et al. 2017).

Original: The second challenge is more mathematical because we must ensure that the regret does not scale exponentially in the number of products or resources (given the combinatorial nature of the underlying graph).
Revised: The second challenge is more mathematical because we must ensure that the regret does not scale exponentially in the number of products or resources (given the combinatorial nature of the underlying graph).

Original: (Analysis.)
Revised: (Regret Analysis.)

Original: To cope with the first challenge mentioned above, we bound the loss of nonstationarity by bounding the length of the first interval of each batch.
Revised: To cope with the ﬁrst challenge mentioned above, we bound the loss of nonstationarity by bounding the length of the ﬁrst interval of each batch.

Original: We prove that the system can reach the new steady-state reasonably fast (see mixing times of Jackson Network in Propositions 3 and 4).
Revised: We prove that the system can reach the new steady-state reasonably fast (see mixing times of Jackson Network in Propositions 3 and 4).

Original: To the best of our knowledge, this is the first result giving a finite-time bound on mixing times of Jackson
Revised: To the best of our knowledge, this is the ﬁrst result giving a ﬁnite-time bound on mixing times of Jackson

Original: Networks.
Revised: Networks.

Original: To cope with the second challenge, we decompose the resulting Jackson Network into topological layers and present a novel nested coupling argument (based on the layers).
Revised: To cope with the second challenge, we decompose the resulting Jackson Network into topological layers and present a novel nested coupling argument (based on the layers).

Original: The coupling argument of Jia et al.
Revised: The coupling argument of Jia et al.

Original: (2020) in the single-resource single-product setting cannot be directly applied to our network setting.
Revised: (2020) in the single-resource single-product setting cannot be directly applied to our network setting.

Original: First, in the network of M/M/c queues, the arrival and waiting processes depend on previously visited resources, which is significantly more complex to analyze upon price changes.
Revised: First, in the network of M/M/c queues, the arrival and waiting processes depend on previously visited resources, which is signiﬁcantly more complex to analyze upon price changes.

Original: To resolve this challenge, we have to carefully construct virtual and alternative queues to explicitly account for such dependency.
Revised: To resolve this challenge, we have to carefully construct virtual and alternative queues to explicitly account for such dependency.

Original: On a high level, the difficulty lies in that the original queues have a time-varying rate from an initial rate to a stable rate upon any price change.
Revised: On a high level, the difﬁculty lies in that the original queues have a time-varying rate from an initial rate to a stable rate upon any price change.

Original: Given a carefully chosen time period τ , we first construct alternative queues that start from zero initial rate and ramp up directly to the stable rate at a timestamp before τ so that the customer distribution matches the original queues in period τ .
Revised: Given a carefully chosen time period τ , we ﬁrst construct alternative queues that start from zero initial rate and ramp up directly to the stable rate at a timestamp before τ so that the customer distribution matches the original queues in period τ .

Original: We then construct virtual queues that could clear all previously arrived customers at a specific time τ , and show that their mixing times provide an upper bound on the alternative queues and consequently the original queues.
Revised: We then construct virtual queues that could clear all previously arrived customers at a speciﬁc time τ , and show that their mixing times provide an upper bound on the alternative queues and consequently the original queues.

Original: Second, even ignoring the subtleties of time-varying rates (upon price changes), if one applies the coupling argument of Jia et al.
Revised: Second, even ignoring the subtleties of time-varying rates (upon price changes), if one applies the coupling argument of Jia et al.

Original: (2020) in a brute-force way to every resource in the network, the coupling probability would result in an undesirable exponential scaling in the number of products.
Revised: (2020) in a brute-force way to every resource in the network, the coupling probability would result in an undesirable exponential scaling in the number of products.

Original: We decompose the network into topological layers so that the resulting coupling probability only scales linearly in the number of products.
Revised: We decompose the network into topological layers so that the resulting coupling probability only scales linearly in the number of products.

Original: 1.3 Literature Review
Revised: 1.3 Literature Review

Original: NRM with Reusable Resources and Demand Learning.
Revised: NRM with Reusable Resources and Demand Learning.

Original: Network, demand learning, and reusable resources are three key elements of this study.
Revised: Network, demand learning, and reusable resources are three key elements of this study.

Original: We only review related literature that shares at least two of the three elements.
Revised: We only review related literature that shares at least two of the three elements.

Original: NRM with demand learning.
Revised: NRM with demand learning.

Original: Due to the easy accessibility of real-time sales data, there have been recent studies on developing learning and pricing algorithms with probably near-optimal regret bound for NRM (see, e.g., Besbes and Zeevi 2012, Chen et al.
Revised: Due to the easy accessibility of real-time sales data, there have been recent studies on developing learning and pricing algorithms with probably near-optimal regret bound for NRM (see, e.g., Besbes and Zeevi 2012, Chen et al.

Original: 2019, Chen and Shi 2019, Badanidiyuru et al.
Revised: 2019, Chen and Shi 2019, Badanidiyuru et al.

Original: 2013, Agrawal and Devanur 2016, Ferreira et al.2018).
Revised: 2018, Agrawal and Devanur 2016, Ferreira et al.

Original: All the aforementioned study considers the canonical settings (first defined by Gallego and Van Ryzin 1997), i.e., non-reusable resources.
Revised: All the aforementioned study considers the canonical settings (ﬁrst deﬁned by Gallego and Van Ryzin 1997), i.e., non-reusable resources.

Original: NRM with reusable resources.
Revised: NRM with reusable resources.

Original: The only existing studies on RNRM are Owen and Simchi-Levi (2018),
Revised: The only existing studies on RNRM are Owen and Simchi-Levi (2018),

Original: Rusmevichientong et al.
Revised: Rusmevichientong et al.

Original: and Lei and Jasin (2020).
Revised: and Lei and Jasin (2020).

Original: Lei and Jasin (2020) considered a deterministic service time assumption, and developed an asymptotically near-optimal real-time pricing control algorithm.
Revised: Lei and Jasin (2020) considered a deterministic service time assumption, and developed an asymptotically near-optimal real-time pricing control algorithm.

Original: Owen and Simchi-Levi (2018) considered a joint assortment and pricing problem under a multi-resource setting and provided constant factor performance guarantees.
Revised: Owen and Simchi-Levi (2018) considered a joint assortment and pricing problem under a multi-resource setting and provided constant factor performance guarantees.

Original: Rusmevichientong et al.
Revised: Rusmevichientong et al.

Original: (2020) considered a dynamic assortment and pricing problem with reusable products and proposed a provably near-optimal policy based on approximate dynamic programming.
Revised: (2020) considered a dynamic assortment and pricing problem with reusable products and proposed a provably near-optimal policy based on approximate dynamic programming.

Original: Demand learning with reusable resources.
Revised: Demand learning with reusable resources.

Original: The aforementioned literature on reusable resources all assumed known distributional information of the underlying model.
Revised: The aforementioned literature on reusable resources all assumed known distributional information of the underlying model.

Original: There has been little work considering demand learning for service systems with reusable resources due to complex underlying system dynamics.
Revised: There has been little work considering demand learning for service systems with reusable resources due to complex underlying system dynamics.

Original: The only studies that considered demand learning for reusable resources are Chen et al.
Revised: The only studies that considered demand learning for reusable resources are Chen et al.

Original: (2020), Jia et al.
Revised: (2020), Jia et al.

Original: (2020, 2022), and all considered only the single resource setting.
Revised: (2020, 2022), and all considered only the single resource setting.

Original: Chen et al.
Revised: Chen et al.

Original: (2020) proposed an online learning framework for a dynamic pricing and capacity sizing problem in a GI/GI/1 queue and Jia et al.
Revised: (2020) proposed an online learning framework for a dynamic pricing and capacity sizing problem in a GI/GI/1 queue and Jia et al.

Original: (2020, 2022) developed MAB-based algorithms for optimal dynamic pricing in an M/M/c queue for the single product single resource setting.
Revised: (2020, 2022) developed MAB-based algorithms for optimal dynamic pricing in an M/M/c queue for the single product single resource setting.

Original: To the best of our knowledge, we are the first to thoroughly study all three elements.
Revised: To the best of our knowledge, we are the ﬁrst to thoroughly study all three elements.

Original: Transient Analysis of Jackson Networks.
Revised: Transient Analysis of Jackson Networks.

Original: Jackson Network has received much attention among the queueing networks because its steady-state distribution is particularly simple to compute as a product-form solution (Jackson 1963, 1957).
Revised: Jackson Network has received much attention among the queueing networks because its steady-state distribution is particularly simple to compute as a product-form solution (Jackson 1963, 1957).

Original: The queue lengths across the nodes in Jackson
Revised: The queue lengths across the nodes in Jackson

Original: Networks, i.e., a network of M/M/c queues, during the transient state can be bounded via techniques of stochastic dominance (see Massey 1986, 1984, 1987).
Revised: Networks, i.e., a network of M/M/c queues, during the transient state can be bounded via techniques of stochastic dominance (see Massey 1986, 1984, 1987).

Original: However, how the network reacts to system changes remains an open question (see, e.g., Mohamed et al. 2003, 2005).
Revised: However, how the network reacts to system changes remains an open question (see, e.g., Mohamed et al. 2003, 2005).

Original: Kelton and Law (1985) pointed out that a queueing system needs to run “long-enough” for removing the effects brought by the initial starting point and then reach an ensured “steady-state” portion of the run.
Revised: Kelton and Law (1985) pointed out that a queueing system needs to run “long-enough” for removing the effects brought by the initial starting point and then reach an ensured “steady-state” portion of the run.

Original: The only work that explicitly developed a finite-time bound on mixing times of a queueing system is a recent paper by Jia et al.
Revised: The only work that explicitly developed a ﬁnite-time bound on mixing times of a queueing system is a recent paper by Jia et al.

Original: In contrast, our paper gives the first finite-time high probability bound on mixing times of Jackson Networks (a network consisting of M/M/c queues) upon action changes.
Revised: In contrast, our paper gives the ﬁrst ﬁnite-time high probability bound on mixing times of Jackson Networks (a network consisting of M/M/c queues) upon action changes.

Original: Batched Bandit Approaches.
Revised: Batched Bandit Approaches.

Original: UCB2 (Auer et al. 2002) and Improved-UCB (Auer and Ortner 2010) were implicitly M -batch policies with M = Θ(log( T )) .
Revised: UCB2 (Auer et al. 2002) and Improved-UCB (Auer and Ortner 2010) were implicitly M -batch policies with M = Θ(log( T )) .

Original: Later, Perchet et al.
Revised: Later, Perchet et al.

Original: studied batched
Revised: studied batched

Original: MAB in the two-armed case under a static batch design and Gao et al.
Revised: MAB in the two-armed case under a static batch design and Gao et al.

Original: (2019) extended the algorithms to the general multiple-armed case and adaptively determined batch size.
Revised: (2019) extended the algorithms to the general multiple-armed case and adaptively determined batch size.

Original: There is also rich literature focusing on batched MAB problems with switching cost.
Revised: There is also rich literature focusing on batched MAB problems with switching cost.

Original: For example, Cesa-Bianchi et al.
Revised: For example, Cesa-Bianchi et al.

Original: (2013) considered uniform switching cost in the objective and Simchi-Levi and Xu (2019) treated the finite switching cost budget as a hard constraint.
Revised: (2013) considered uniform switching cost in the objective and Simchi-Levi and Xu (2019) treated the ﬁnite switching cost budget as a hard constraint.

Original: In contrast, we construct the batched action framework to control the loss of the nonstationarity (resulted from the transient time after each action change).
Revised: In contrast, we construct the batched action framework to control the loss of the nonstationarity (resulted from the transient time after each action change).

Original:  Problem Formulation
Revised:  Problem Formulation

Original: Consider a firm that manages I types of reusable resources to provide J types of products over a finite time horizon T .
Revised: Consider a ﬁrm that manages I types of reusable resources to provide J types of products over a ﬁnite time horizon T .

Original: Denote the capacity of resource type i ∈ I as c i .
Revised: Denote the capacity of resource type i ∈ I as c i .

Original: At the beginning of each period, the firm posts a price x = { x j : j ∈ J } for all products from a given price candidate set
Revised: At the beginning of each period, the ﬁrm posts a price x = { x j : j ∈ J } for all products from a given price candidate set

Original: X ⊂ R |J| .
Revised: X ⊂ R |J| .

Original: Let I =
Revised: Let I =

Original: |I| , J = |J | , and X = |X| .
Revised: |I| , J = |J | , and X = |X| .

Original: Each arriving customer requests one product j ∈ J , which is processed through an ordered sequence of resources.
Revised: Each arriving customer requests one product j ∈ J , which is processed through an ordered sequence of resources.

Original: Customers arrive following a price-dependent multivariate Poisson process with a total rate α ( x ) , which is unknown to the decision maker a priori .
Revised: Customers arrive following a price-dependent multivariate Poisson process with a total rate α ( x ) , which is unknown to the decision maker a priori .

Original: The service time of resource i follows an exponential distribution regardless of which product type it belongs to, which also depends on posted price x .
Revised: The service time of resource i follows an exponential distribution regardless of which product type it belongs to, which also depends on posted price x .

Original: If there is not enough capacity for the current resource, the customer will join the queue of this resource until being served.
Revised: If there is not enough capacity for the current resource, the customer will join the queue of this resource until being served.

Original: The resources are released to process the next customers upon finishing serving the current ones.
Revised: The resources are released to process the next customers upon ﬁnishing serving the current ones.

Original: We consider two popular payment models, the lump sum payment model (presented in the main manuscript) and the service duration-dependent payment model (presented in Appendix F).
Revised: We consider two popular payment models, the lump sum payment model (presented in the main manuscript) and the service duration-dependent payment model (presented in Appendix F).

Original: • ( Lump sum payment model.)
Revised: • ( Lump sum payment model.)

Original: A customer of product j ∈ J pays a lump sum
Revised: A customer of product j ∈ J pays a lump sum

Original: x j to the firm.
Revised: x j to the ﬁrm.

Original: More elaborately, if a customer chooses product j , then the customer will pay x j to the firm regardless of how long the services over resources are.
Revised: More elaborately, if a customer chooses product j , then the customer will pay x j to the ﬁrm regardless of how long the services over resources are.

Original: In our illustrative example, each product can be quoted a fixed price (i.e., not service duration dependent), which could be seen as a preset “cloud solution package”.
Revised: In our illustrative example, each product can be quoted a ﬁxed price (i e., not service duration dependent), which could be seen as a preset “cloud solution package”.

Original: The customers may prefer or benefit from this fixed price (contract) from their own accounting and cash flow management.
Revised: The customers may prefer or beneﬁt from this ﬁxed price (contract) from their own accounting and cash ﬂow management.

Original: Under this model, even though the customers are not sensitive to service durations, the cloud platform is nevertheless sensitive to service durations since they affect how many products/services can be completed within a finite horizon.
Revised: Under this model, even though the customers are not sensitive to service durations, the cloud platform is nevertheless sensitive to service durations since they affect how many products/services can be completed within a ﬁnite horizon.

Original: • ( Service duration dependent payment model.)
Revised: • ( Service duration dependent payment model.)

Original: A customer of product j ∈ J pays the amount that is equal to the per-unit-time x j multiplied by the total service duration.
Revised: A customer of product j ∈ J pays the amount that is equal to the per-unit-time x j multiplied by the total service duration.

Original: This payment scheme is duration dependent, which is also ubiquitous, especially in cloud computing applications (e g., how much time is spent using EC2 in our illustrative example).
Revised: This payment scheme is duration dependent, which is also ubiquitous, especially in cloud computing applications (e.g., how much time is spent using EC2 in our illustrative example).

Original: In the interest of space, the model and analytical details are presented in Appendix F.
Revised: In the interest of space, the model and analytical details are presented in Appendix F.

Original: Consider a resource-consumption matrix A ji ∈ R
Revised: Consider a resource-consumption matrix A ji ∈ R

Original: J × I , where element a ji = 1 indicates that service type j requires to be processed on server type i .
Revised: J × I , where element a ji = 1 indicates that service type j requires to be processed on server type i .

Original: We use an I -dimensional vector, y ∈ { 0 , 1 , . . .
Revised: We use an I -dimensional vector, y ∈ { 0 , 1 , . . .

Original: , ∞} I to denote the number of customers across different types of resources.
Revised: , ∞} I to denote the number of customers across different types of resources.

Original: The firm aims at finding a pricing policy as π : { ( y , t ) : y ∈ { 0 , 1 , . . .
Revised: The ﬁrm aims at ﬁnding a pricing policy as π : { ( y , t ) : y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T } → X , where the firm selects price π
Revised: , T } → X , where the ﬁrm selects price π

Original: ( y , t ) ∈ X for period t when there are y customers in the system at the beginning of period t , to maximize the expected cumulative revenue Y π .
Revised: ( y , t ) ∈ X for period t when there are y customers in the system at the beginning of period t , to maximize the expected cumulative revenue Y π .

Original: Assumption 1.
Revised: Assumption 1.

Original: (“Turn-off” price.)
Revised: (“Turn-off” price.)

Original: For each product j ∈ J , there exists a “turn-off” price ¯ p j < ∞ such that, if the posted price is larger than this turn-off price, then the demand for this product is zero.
Revised: For each product j ∈ J , there exists a “turn-off” price ¯ p j < ∞ such that the demand for this product is zero when the posted price is larger than this turn-off price.

Original: Assumption 1 is widely assumed in NRM studies (see, e.g., Lei and Jasin 2020).
Revised: Assumption 1 is widely assumed in NRM studies (see, e.g., Lei and Jasin 2020).

Original: This turn-off price can be used to effectively turn off the demand for undesired products whenever needed.
Revised: This turn-off price can be used to effectively turn off the demand for undesired products whenever needed.

Original: Therefore, for each candidate price x ∈ X , we can further decompose the products into turn-on product set
Revised: Therefore, for each candidate price x ∈ X , we can further decompose the products into turn-on product set

Original: J on ( x ) and a turn-off product set J off ( x ) , where the latter can be empty.
Revised: J on ( x ) and a turn-off product set J off ( x ) , where the latter can be empty.

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists an order of server types I such that the processing sequence of servers i 1 , i 2 , . .
Revised: There exists an order of server types I such that the processing sequence of servers i 1 , i 2 , . .

Original: , i n always has i 1 <
Revised: , i n always has i 1 <

Original: i 2 < . .
Revised: i 2 < . .

Original: < i n for any product j ∈ J .
Revised: < i n for any product j ∈ J .

Original: One can understand Assumption 2 via viewing a network.
Revised: One can understand Assumption 2 via viewing a network.

Original: Consider a graph with resources as nodes and draw a directed arc from node i to i ′ if any product requires successive processing on resources i and i ′ .
Revised: Consider a graph with resources as nodes and draw a directed arc from node i to i (cid:48) if any product requires successive processing on resources i and i (cid:48) .

Original: If the system satisfies Assumption 2, then this graph is acyclic, i.e., no cycles, and there exists at least one topological order such that all arcs are from a smaller index to a larger index.
Revised: If the system satisﬁes Assumption 2, then this graph is acyclic, i.e., no cycles, and there exists at least one topological order such that all arcs are from a smaller index to a larger index.

Original: Without loss of generality, we consider one such topological order when referring to resource i in the rest part of this paper.
Revised: Without loss of generality, we consider one such topological order when referring to resource i in the rest part of this paper.

Original: To better utilize this network representation, we also add a source node s to represent customers entering the system and a sink node t to represent customers exiting the system.
Revised: To better utilize this network representation, we also add a source node s to represent customers entering the system and a sink node t to represent customers exiting the system.

Original: Thus, the consumption matrix can be extended to A ji ∈ R J ×
Revised: Thus, the consumption matrix can be extended to A ji ∈ R J ×

Original: ( I +2) and a js = a jt = 1 for all products.
Revised: ( I +2) and a js = a jt = 1 for all products.

Original: Assumption 3.
Revised: Assumption 3.

Original: (Product-form demand.)
Revised: (Product-form demand.)

Original: Any given price vector x induces a (Markovian) routing matrix P ( x ) where its ( i, i ′ )
Revised: Any given price vector x induces a (Markovian) routing matrix P ( x ) where its ( i, i (cid:48) )

Original: th entry p i,i ′ represents the product demand portion who will consume resource i ′ next out of those who consume resource i .
Revised: th entry p i,i (cid:48) represents the product demand portion who will consume resource i (cid:48) next out of those who consume resource i .

Original: Then the demand rate for any product j ∈ J , represented by a sequence of resources s, i 1 , i 2 , . .
Revised: Then the demand rate for any product j ∈ J , represented by a sequence of resources s, i 1 , i 2 , . .

Original: , i n , t , is given by
Revised: , i n , t , is given by

Original: [Equation]
Revised: [Equation]

Original: which means that the demand rate for product j is equal to the total demand rate α ( x ) times a sequence of routing probabilities of P ( x ) , under posted price vector x .
Revised: which means that the demand rate for product j is equal to the total demand rate α ( x ) times a sequence of routing probabilities of P ( x ) , under posted price vector x .

Original: We use Figure 2 in the Appendix to demonstrate a concrete example.
Revised: We use Figure 2 in the Appendix to demonstrate a concrete example.

Original: With Assumptions 2 and 3, the service system under any posted price x ∈ X can be reduced to a Jackson Network consisting with I number of M/M/c queues (when the system only contains customers arriving under this price, i.e., after finishing serving other customers arriving under the previously posted prices).
Revised: With Assumptions 2 and 3, the service system under any posted price x ∈ X can be reduced to a Jackson Network consisting with I number of M/M/c queues (when the system only contains customers arriving under this price, i.e., after ﬁnishing serving other customers arriving under the previously posted prices).

Original: The main idea is that there exists a mapping x
Revised: The main idea is that there exists a mapping x

Original: (cid:55)→ α ( x ) , P ( x ) that gives rise to product demand pattern.
Revised: (cid:55)→ α ( x ) , P ( x ) that gives rise to product demand pattern.

Original: Note that our assumptions are mild in the sense that this mapping can be nonparametric.
Revised: Note that our assumptions are mild in the sense that this mapping can be nonparametric.

Original: The decision maker does not know this underlying mapping and needs to learn it while maximizing revenue on the fly.
Revised: The decision maker does not know this underlying mapping and needs to learn it while maximizing revenue on the ﬂy.

Original: Given any price x , (i) on one hand, this price affects the total demand α ( x ) entering the system.
Revised: Given any price x , (i) on one hand, this price affects the total demand α ( x ) entering the system.

Original: (ii)
Revised: (ii)

Original: On the other hand, this price affects the internal competition between products, which is reflected in the server routing matrix P ( x ) .
Revised: On the other hand, this price affects the internal competition between products, which is reﬂected in the server routing matrix P ( x ) .

Original: As a result, α ( x ) and P ( x ) define a Jackson Network well and the demand rate α j ( x ) of each product type is also determined (use vector ⃗α x ∈ R J to denote demand of products).
Revised: As a result, α ( x ) and P ( x ) deﬁne a Jackson Network well and the demand rate α j ( x ) of each product type is also determined (use vector (cid:126)α x ∈ R J to denote demand of products).

Original: We use λ i ( x ) and µ i ( x ) to denote the demand and service rates of resource type i .
Revised: We use λ i ( x ) and µ i ( x ) to denote the demand and service rates of resource type i .

Original: Assumption 4.
Revised: Assumption 4.

Original: (Stability.)
Revised: (Stability.)

Original: The queueing system of each resource is stable under any price, i.e., c i µ i ( x ) >
Revised: The queueing system of each resource is stable under any price, i.e., c i µ i ( x ) >

Original: λ i ( x ) .
Revised: λ i ( x ) .

Original: Therefore, the Jackson Network is stable as well.
Revised: Therefore, the Jackson Network is stable as well.

Original: Assumption 4 ensures that the underlying Jackson Network is stable under any candidate price (i.e., the queue length will not grow indefinitely).
Revised: Assumption 4 ensures that the underlying Jackson Network is stable under any candidate price (i.e., the queue length will not grow indeﬁnitely).

Original: This assumption can be readily satisfied by putting appropriate lower bounds on the price vector.
Revised: This assumption can be readily satisﬁed by putting appropriate lower bounds on the price vector.

Original: Assumption 5.
Revised: Assumption 5.

Original: (Change of rates.)
Revised: (Change of rates.)

Original: For any node, consider the arrival and service rates ( λ 1 , µ 1 ) and ( λ 2 , µ 2 ) under x 1 , x 2 ∈ X where λ 1 ≥ λ 2 , also denote the utilization factor by ρ 1 = λ 1 /µ 1 and ρ 2 = λ 2 /µ 2 , respectively, then 1) λ 1 − λ 2 ≤ [Equation] and 2) if µ [Equation]
Revised: For any node, consider the arrival and service rates ( λ 1 , µ 1 ) and ( λ 2 , µ 2 ) under x 1 , x 2 ∈ X where λ 1 ≥ λ 2 , also denote the utilization factor by ρ 1 = λ 1 /µ 1 and ρ 2 = λ 2 /µ 2 , respectively, then 1) λ 1 − λ 2 ≤ ρ 1 − 3 e log ρ 1 and 2) if µ 1 > µ 2 , then ρ 1 ≥ ρ 22 .

Original: > µ 2 , then ρ

Original: [Equation]

Original: ≥ ρ 22 .

Original: This assumption is used to justify Lemmas 3 and 4 (used in the proof of Propositions 3 and 4) and the original proofs are in Jia et al.
Revised: This assumption is used to justify Lemmas 3 and 4 (used in the proof of Propositions 3 and 4) and the original proofs are in Jia et al.

Original: These conditions are mild and easy to validate in practice in the following sense.
Revised: These conditions are mild and easy to validate in practice in the following sense.

Original: The difference in arrival rates between two prices is bounded (by a large constant).
Revised: The difference in arrival rates between two prices is bounded (by a large constant).

Original: This can be easily satisfied by interpolating intermediate prices in the candidate price set X . 2)
Revised: This can be easily satisﬁed by interpolating intermediate prices in the candidate price set X . 2)

Original: If both the service and arrival rate under a price is higher than or equal to another price, the utilization factor in the former case must be greater than the latter squared.
Revised: If both the service and arrival rate under a price is higher than or equal to another price, the utilization factor in the former case must be greater than the latter squared.

Original: A simple sufficient condition to guarantee ρ 1 ≥ ρ 22 is ρ 1 ≥ ρ 2 , which reduces to λ 1 /λ 2 ≥ µ 1 /µ 2 , and thus the practitioners only need to validate that the ratio of change in arrival rates exceeds the ratio of change in service rates.
Revised: A simple sufﬁcient condition to guarantee ρ 1 ≥ ρ 22 is ρ 1 ≥ ρ 2 , which reduces to λ 1 /λ 2 ≥ µ 1 /µ 2 , and thus the practitioners only need to validate that the ratio of change in arrival rates exceeds the ratio of change in service rates.

Original: Assumption 6.
Revised: Assumption 6.

Original: We assume log T ≥ 4 (to ensure sufficient times for mixing).
Revised: We assume log T ≥ 4 (to ensure sufﬁcient time for mixing).

Original:  Regret, Relaxed Regret, and Static Price Benchmark
Revised:  Regret, Relaxed Regret, and Static Price Benchmark

Original: Under the full information scenario, where the firm knows the underlying arrival rate α j ( x ) of all products j ∈ J under any candidate price x ∈ X .
Revised: Under the full information scenario, where the ﬁrm knows the underlying arrival rate α j ( x ) of all products j ∈ J under any candidate price x ∈ X .

Original: We denote a state-dependent optimal policy as π ∗ ,
Revised: We denote a state-dependent optimal policy as π ∗ ,

Original: where π ∗ = argmax π Y π , and let Y ∗ =
Revised: where π ∗ = argmax π Y π , and let Y ∗ =

Original: Y π ∗ .
Revised: Y π ∗ .

Original: Therefore, the regret of any pricing policy π can be defined as Regret ( π, T ) =
Revised: Therefore, the regret of any pricing policy π can be deﬁned as Regret ( π, T ) =

Original: Y ∗ − Y π .
Revised: Y ∗ − Y π .

Original: In short, we aim at finding heuristic pricing policies that lead to a small regret.
Revised: In short, we aim at ﬁnding heuristic pricing policies that lead to a small regret.

Original: However, finding the state-dependent optimal pricing policy π ∗ is computationally intractable even under the full information scenario.
Revised: However, ﬁnding the state-dependent optimal pricing policy π ∗ is computationally intractable even under the full information scenario.

Original: Firstly, it requires solving a dynamic program with an infinite number of potential states, in the form of ( y , t ) , which suffers from the curse of dimensionality.
Revised: Firstly, it requires solving a dynamic program with an inﬁnite number of potential states, in the form of ( y , t ) , which suffers from the curse of dimensionality.

Original: Moreover, the transient dynamics of a queueing network are complex and hard to analyze especially when the offered price changes over periods.
Revised: Moreover, the transient dynamics of a queueing network are complex and hard to analyze especially when the offered price changes over periods.

Original: Thus, it is impossible to obtain π ∗ and Y ∗ .
Revised: Thus, it is impossible to obtain π ∗ and Y ∗ .

Original: To tackle this problem, we develop an upper bound on the expected revenue achievable by the state-dependent optimal expected revenue (see a similar approach in Jia et al. 2020).
Revised: To tackle this problem, we develop an upper bound on the expected revenue achievable by the state-dependent optimal expected revenue (see a similar approach in Jia et al. 2020).

Original: An LP  upper bound.
Revised: An LP based upper bound.

Original: Consider continuous decision variables π xy t  for y ∈ { 0 , 1 , . .
Revised: Consider continuous decision variables π xy t , a y t , and Y y t for y ∈ { 0 , 1 , . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T, x ∈ X , a y t for y ∈ { 0 , 1 , . . .

Original: ,∞} I , t = 1 , . . . , T , and Y y t for y ∈ { 0 , 1 , . . . , ∞} I , t = 1 , . . . , T .
Revised: , T , and x ∈ X .

Original: Wecan formulate a linear program (LP) as follows.
Revised: We formulate a linear program (LP) as follows.

Revised: We also discuss the sufﬁciency of using ( y , t ) as the state variable in Appendix C.

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: The LP solution provides an upper bound on the optimal revenue, i.e., Y ∗ ≤ Y LP , and Y LP = T · ⃗α T ˜ x
Revised: The LP solution provides an upper bound on the optimal revenue, i.e., Y ∗ ≤ Y LP , and Y LP = T · (cid:126)α T ˜ x

Original: ˜ x where ˜ x = arg max x ∈X ⃗α T x x .
Revised: ˜ x where ˜ x = arg max x ∈X (cid:126)α T x x .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: Consider an arbitrary policy ˆ π .
Revised: Consider an arbitrary policy ˆ π .

Original: Let variable ˆ a y t take the value of probability that the system has y customers at the beginning of period t for y ∈ { 0 , 1 , . . .
Revised: Let variable ˆ a y t take the value of probability that the system has y customers at the beginning of period t for y ∈ { 0 , 1 , . . .

Original: , ∞} I and t = 1 , . . .
Revised: , ∞} I and t = 1 , . . .

Original: , T .
Revised: , T .

Original: Let variable ˆ π xy t = 1 if the policy ˆ π chooses price x at state ( y , t ) , and 0 otherwise.
Revised: Let variable ˆ π xy t = 1 if the policy ˆ π chooses price x at state ( y , t ) , and 0 otherwise.

Original: Let variable ˆ Y y t take the value of expected revenue collected before the end of period T from the customers who arrive during period t , when the system has y customers at the beginning of period t .
Revised: Let variable ˆ Y y t take the value of expected revenue collected before the end of period T from the customers who arrive during period t , when the system has y customers at the beginning of period t .

Original: Then we can show that the decision variables ( π, a, Y ) associated with any admissible policy must satisfy the constraints of the linear program (1), and the expected revenue of this admissible policy is exactly the corresponding objective value in (1).
Revised: Then we can show that the decision variables ( π, a, Y ) associated with any admissible policy must satisfy the constraints of the linear program (1), and the expected revenue of this admissible policy is exactly the corresponding objective value in (1).

Original: We can readily check that T · ⃗α T ˜ x
Revised: We can readily check that T · (cid:126)α T ˜ x

Original: ˜ x is an upper bound of Y LP .
Revised: ˜ x is an upper bound of Y LP .

Original: We can construct a solution which can obtain this upper bound and thus Y LP = T · ⃗α T ˜ x
Revised: We can construct a solution which can obtain this upper bound and thus Y LP = T · (cid:126)α T ˜ x

Original: ˜ x .
Revised: ˜ x .

Original: Definition 1.
Revised: Deﬁnition 1.

Original: The relaxed regret is defined as Regret ( π, T ) =
Revised: The relaxed regret is deﬁned as Regret ( π, T ) =

Original: Y LP − Y π .
Revised: Y LP − Y π .

Original: Based on Proposition 1, we have for any policy π , the corresponding relaxed regret is a valid upper bound of its regret, i.e., Regret ( π, T ) ≥ Regret ( π, T ) , and it suffices to bound the relaxed regret.
Revised: Based on Proposition 1, we have for any policy π , the corresponding relaxed regret is a valid upper bound of its regret, i.e., Regret ( π, T ) ≥ Regret ( π, T ) , and it sufﬁces to bound the relaxed regret.

Original:  Batched Bandit Algorithms
Revised:  Batched Bandit Algorithms

Original: The basis for implementing an Upper Confidence Bound (UCB) type of algorithm is identifying an upper confidence bound of the expected collected revenue of each candidate price.
Revised: The basis for implementing an Upper Conﬁdence Bound (UCB) type of algorithm is identifying an upper conﬁdence bound of the expected collected revenue of each candidate price.

Original: We start with the steady-state revenue rate.
Revised: We start with the steady-state revenue rate.

Original: We can show that this steady-state revenue rate can be used to compute a lower bound of the expected revenue during consecutive periods (see proof of Theorem 1).
Revised: We can show that this steady-state revenue rate can be used to compute a lower bound of the expected revenue during consecutive periods (see proof of Theorem 1).

Original: Therefore, it is equivalent to deriving an upper confidence bound of the steady-state revenue rate based on historical sales information.
Revised: Therefore, it is equivalent to deriving an upper conﬁdence bound of the steady-state revenue rate based on historical sales information.

Original: Jia et al.
Revised: Jia et al.

Original: (2020) established a concentration bound of the steady-state revenue rate under the single product setting.
Revised: (2020) established a concentration bound of the steady-state revenue rate under the single product setting.

Original: Here we develop a concentration bound for the multi-product multi-resource setting.
Revised: Here we develop a concentration bound for the multi-product multi-resource setting.

Original: We use n j ( x ) to denote the number of arrived customers requesting product j under price x .
Revised: We use n j ( x ) to denote the number of arrived customers requesting product j under price x .

Original: We use ˆ d jk ( x ) to denote the observed time interval between the arrival time of customer k and the arrival time of customer k − 1 who are requesting product j under the price x .
Revised: We use ˆ d jk ( x ) to denote the observed time interval between the arrival time of customer k and the arrival time of customer k − 1 who are requesting product j under the price x .

Original: We use ¯ d j ( x ) to denote the empirical mean of the inter-arrival time and thus ¯ d j ( x ) = (cid:80) n j ( x ) k
Revised: We use ¯ d j ( x ) to denote the empirical mean of the inter-arrival time and thus ¯ d j ( x ) = (cid:80) n j ( x ) k

Original: =1 ˆ d jk ( x ) /n
Revised: =1 ˆ d jk ( x ) /n

Original: j ( x ) , for j ∈ J .
Revised: j ( x ) , for j ∈ J .

Original: Proposition 2.
Revised: Proposition 2.

Original: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( J 14 T ) for j ∈ J on ( x ) , then we have
Revised: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( J 14 T ) for j ∈ J on ( x ) , then we have

Original: [Equation]
Revised: [Equation]

Original: The proof of Proposition 2 is based on the concentration inequality for sub-exponential random variables (see, e.g., Boucheron et al.
Revised: The proof of Proposition 2 is based on the concentration inequality for sub-exponential random variables (see, e.g., Boucheron et al.

Original: 2013, Jia et al. 2020)
Revised: 2013, Jia et al. 2020)

Original: and is provided in Appendix C. Note that ⃗α T x x is the steady-state revenue rate and denote it by r ( ⃗α x , x ) .
Revised: and is provided in Appendix C. Note that (cid:126)α T x x is the steady-state revenue rate and denote it by r ( (cid:126)α x , x ) .

Original: Proposition 2 says that with a certain amount of arrival time observations of turn-on products under a candidate price, we can derive a valid upper bound of the steady-state revenue rate with a high probability guarantee.
Revised: Proposition 2 says that with a certain amount of arrival time observations of turn-on products under a candidate price, we can derive a valid upper bound of the steady-state revenue rate with a high probability guarantee.

Original: Hence, we define the corresponding upper and lower confidence bounds of the stationary revenue rate associated with price x by the end of period t as follows.
Revised: Hence, we deﬁne the corresponding upper and lower conﬁdence bounds of the stationary revenue rate associated with price x by the end of period t as follows.

Original: Definition 2. Define
Revised: Deﬁnition

Original: [Equation]
Revised: [Equation]

Original: √ n j ( x ) as the confidence radius by the end of period t .
Revised: as the conﬁdence radius by the end of period t .

Original: The upper and lower confidence bounds of the revenue rate associated with price x are:
Revised: The upper and lower conﬁdence bounds of the revenue rate associated with price x are:

Original: [Equation]
Revised: [Equation]

Original: 4.1 RNRM-UCB
Revised: 4.1 RNRM-UCB

Original: With Definition 2, we can leverage the idea of UCB to solve RNRM.
Revised: With Deﬁnition 2, we can leverage the idea of UCB to solve RNRM.

Original: We separate T periods into two phases.
Revised: We separate T periods into two phases.

Original: The first phase is the Warm-up Phase, where we offer each candidate price one by one until we collect 8 log( J 1 4 T ) number of arrival time intervals for each turn-on product j ∈ J on ( x ) under each price.
Revised: The ﬁrst phase is the Warm-up Phase, where we offer each candidate price one by one until we collect 8 log( J 1 4 T ) number of arrival time intervals for each turn-on product j ∈ J on ( x ) under each price.

Original: Therefore, the total length of this Warm-up Phase is O
Revised: Therefore, the total length of this Warm-up Phase is O

Original: ( X log( J 14 T )) .
Revised: ( X log( J 14 T )) .

Original: After this
Revised: After this

Original: Warm-up Phase, Proposition 2 is valid to use.
Revised: Warm-up Phase, Proposition 2 is valid to use.

Original: The second phase is the Learning Phase.
Revised: The second phase is the Learning Phase.

Original: The UCB algorithm cannot be directly used because in order to estimate the expected revenue when offering a price, we have to wait until the system reaches the steady-state.
Revised: The UCB algorithm cannot be directly used because in order to estimate the expected revenue when offering a price, we have to wait until the system reaches the steady-state.

Original: The key idea is to reduce the number of price changes and further reduce the intractable transient state performance.
Revised: The key idea is to reduce the number of price changes and further reduce the intractable transient state performance.

Original: We separate the Learning Phase into M number of batches.
Revised: We separate the Learning Phase into M number of batches.

Original: Define I m = 2 m , ∀ m = 1 , . . .
Revised: Deﬁne I m = 2 m , ∀ m = 1 , . . .

Original: , M .
Revised: , M .

Original: Each batch m contains I m Lτ periods, where τ = (log T ) 2 and L is the number of layers defined in Definition 3.
Revised: Each batch m contains I m Lτ periods, where τ = (log T ) 2 and L is the number of layers deﬁned in Deﬁnition 3.

Original: Therefore, the batch size is exponentially increasing.
Revised: Therefore, the batch size is exponentially increasing.

Original: We can then add m in the subscript of notation ( n mj ( x ) , ˆ d jk ( x ) , ¯ d mj ( x ) , Rad m ( x ) , U m ( x ) , L m ( x )) to represent their values by the end of batch m .
Revised: We can then add m in the subscript of notation ( n mj ( x ) , ˆ d jk ( x ) , ¯ d mj ( x ) , Rad m ( x ) , U m ( x ) , L m ( x )) to represent their values by the end of batch m .

Original: At the beginning of each batch m + 1 , RNRM-UCB chooses price x with the highest U m ( x ) and offers this price during batch m
Revised: At the beginning of each batch m + 1 , RNRM-UCB chooses price x with the highest U m ( x ) and offers this price during batch m

Original: + 1 .
Revised: + 1 .

Original: We present the algorithmic details in Algorithm 1.
Revised: We present the algorithmic details in Algorithm 1.

Original: Considering the firm holds prior information about the rates, then we can add the Bayesian assumption to this RNRM problem.
Revised: Considering the ﬁrm holds prior information about the rates, then we can add the Bayesian assumption to this RNRM problem.

Original: By the conjugate prior theory (see, e.g., Minton et al.
Revised: By the conjugate prior theory (see, e.g., Minton et al.

Original: 1962, Diaconis and
Revised: 1962, Diaconis and

Original: Ylvisaker 1979), we assume that RNRM-TS holds Gamma priors on α j ( x ) of product j ∈ J on ( x ) , ∀ x ∈ X and the posteriors are updated by the end of batch m, ∀ m = 1 , . . .
Revised: Ylvisaker 1979), we assume that RNRM-TS holds Gamma priors on α j ( x ) of product j ∈ J on ( x ) , ∀ x ∈ X and the posteriors are updated by the end of batch m, ∀ m = 1 , . . .

Original: , M .
Revised: , M .

Original: Specifically, we set the prior distribution of α j ( x ) as Gamma ( η α j ( x ) , β α j ( x ) ) .
Revised: Speciﬁcally, we set the prior distribution of α j ( x ) as Gamma ( η α j ( x ) , β α j ( x ) ) .

Original: After observing one arrival time interval ˆ d jk ( x ) of product j while offering price x , the posterior distribution of α j ( x ) is computed as Gamma ( η α j ( x ) + 1 , β α j ( x ) + ˆ d jk ( x )) .
Revised: After observing one arrival time interval ˆ d jk ( x ) of product j while offering price x , the posterior distribution of α j ( x ) is computed as Gamma ( η α j ( x ) + 1 , β α j ( x ) + ˆ d jk ( x )) .

Original: To implement RNRM-TS, at the beginning of batch m , we sample parameters ˆ α j ( x ) for product j ∈ J on ( x ) , ∀ x ∈ X , following the updated posterior distributions.
Revised: To implement RNRM-TS, at the beginning of batch m , we sample parameters ˆ α j ( x ) for product j ∈ J on ( x ) , ∀ x ∈ X , following the updated posterior distributions.

Original: Then we offer the price with the maximum sampled stationary network revenue rate r (ˆ ⃗α x , x ) = (cid:80) j ∈J on ( x ) x
Revised: Then we offer the price with the maximum sampled stationary network revenue rate r (ˆ (cid:126)α x , x ) = (cid:80) j ∈J on ( x ) x

Original: j ˆ α j ( x ) .
Revised: j ˆ α j ( x ) .

Original: We present the algorithmic details of RNRM-TS in Algorithm 2.
Revised: We present the algorithmic details of RNRM-TS in Algorithm 2.

Original:  time ˆ d jk ( x m ) for arriving customers of product j ∈ J on ( x m ) .
Revised: periods. Record inter-arrival time ˆ d jk ( x m ) for arriving customers of product j ∈ J on ( x m ) .

Original: Compute and update η α j ( x m ) , β α j ( x m ) , ∀ j ∈ J on ( x m ) .
Revised: Compute and update η α j ( x m ) , β α j ( x m ) , ∀ j ∈ J on ( x m ) .

Original: 18: end for
Revised: 18: end for

Original:  Performance Analysis 5.1 High Probability Bound for Mixing Time
Revised:  Performance Analysis 5.1 High Probability Bound for Mixing Time

Original: When the firm starts service from an empty state, i.e., zero customers in the network service system, or switches to another price from the currently offered price, the service (queueing network) system enters a transient state and needs a certain amount of time to reach the steady-state again (referred to as the mixing time).
Revised: When the ﬁrm starts service from an empty state, i.e., zero customers in the network service system, or switches to another price from the currently offered price, the service (queueing network) system enters a transient state and needs a certain amount of time to reach the steady-state again (referred to as the mixing time).

Original: Recall that the service system can be reduced to a Jackson Network under a specifically offered price vector, upon completion of existing customers under the previously posted price.
Revised: Recall that the service system can be reduced to a Jackson Network under a speciﬁcally offered price vector, upon completion of existing customers under the previously posted price.

Original: Therefore, we evaluate how quickly a Jackson Network can reach the steady-state.
Revised: Therefore, we evaluate how quickly a Jackson Network can reach the steady-state.

Original: To the best of our knowledge, this is the first result giving a finite-time high probability bound on mixing times of Jackson Network system (upon action changes).
Revised: To the best of our knowledge, this is the ﬁrst result giving a ﬁnite-time high probability bound on mixing times of Jackson Network system (upon action changes).

Original: Technically, we develop a new coupling argument between this target system and a virtual system starting from a state sampled from the steady-state distribution and maintaining the steady-state thereafter.
Revised: Technically, we develop a new coupling argument between this target system and a virtual system starting from a state sampled from the steady-state distribution and maintaining the steady-state thereafter.

Original: Poisson arrival and departure under steady-state.
Revised: Poisson arrival and departure under steady-state.

Original: Under the steady-state of an M/M/c queue (if it exists, i.e., ρ = λ/cµ < 1 ), then the process following which the customers leave the system is also a
Revised: Under the steady-state of an M/M/c queue (if it exists, i.e., ρ = λ/cµ < 1 ), then the process following which the customers leave the system is also a

Original: Poisson process with rate λ .
Revised: Poisson process with rate λ .

Original: Therefore, one can easily derive that the arrival as well as the departure processes of each type of resource (each node in the network representation) are Poisson processes with the same rate λ i under the steady-state (see the famous Burke’s Theorem Burke 1956).
Revised: Therefore, one can easily derive that the arrival as well as the departure processes of each type of resource (each node in the network representation) are Poisson processes with the same rate λ i under the steady-state (see the famous Burke’s Theorem Burke 1956).

Original: Jia et al.
Revised: Jia et al.

Original: (2020) is the only work for studying finite-time bounds on the mixing time for an M/M/c queue.
Revised: (2020) is the only work for studying ﬁnite-time bounds on the mixing time for an M/M/c queue.

Original: However, their results cannot be directly applied here because (i) the arrival process of each resource is not a well-defined Poisson process and (ii) the starting point of the later queues is not zero state nor a steady-state of old system parameters.
Revised: However, their results cannot be directly applied here because (i) the arrival process of each resource is not a well-deﬁned Poisson process and (ii) the starting point of the later queues is not zero state nor a steady-state of old system parameters.

Original: Even with these two challenges tackled, this approach will induce a mixing time which is linear on the number of resources.
Revised: Even with these two challenges tackled, this approach will induce a mixing time which is linear on the number of resources.

Original: This result is not desired for systems with multiple (especially large) resources.
Revised: This result is not desired for systems with multiple (especially large) resources.

Original: We develop a new mixing time analysis for Jackson Network.
Revised: We develop a new mixing time analysis for Jackson Network.

Original: We divide the resources of the service network into layers and derive a high-probability mixing time that is only linear on the number of layers.
Revised: We divide the resources of the service network into layers and derive a high-probability mixing time that is only linear on the number of layers.

Original: Definition 3.
Revised: Deﬁnition 3.

Original: We divide the nodes (types of servers) into layers indexed by the maximum number of previously occupied nodes (including the virtual source node) along the service routes of all products.
Revised: We divide the nodes (types of servers) into layers indexed by the maximum number of previously occupied nodes (including the virtual source node) along the service routes of all products.

Original: For example, Nodes 1 and 2 of the illustrative example (see Figure 2) is Layer-1 nodes, and Node 3 is a Layer-2 node.
Revised: For example, Nodes 1 and 2 of the illustrative example (see Figure 2) is Layer-1 nodes, and Node 3 is a Layer-2 node.

Original: We denote the number of layers by L and the set of Layer- l nodes as I l .
Revised: We denote the number of layers by L and the set of Layer- l nodes as I l .

Original: Proposition 3.
Revised: Proposition 3.

Original: (Coupling probability of a Jackson Network starting from empty state) Define A t to be the event in which the Jackson Network reaches the steady-state by time t .
Revised: (Coupling probability of a Jackson Network starting from empty state) Deﬁne A t to be the event in which the Jackson Network reaches the steady-state by time t .

Original: For t ≥ Lτ , where τ = (log T ) 2 , we have P ( A t ) ≥ 1 − 2 IT 2 .
Revised: For t ≥ Lτ , where τ = (log T ) 2 , we have P ( A t ) ≥ 1 − 2 IT 2 .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: We use the aforementioned Poisson arrival and departure property of the nodes in a Jackson Network to bound the mixing time of the nodes layer by layer.
Revised: We use the aforementioned Poisson arrival and departure property of the nodes in a Jackson Network to bound the mixing time of the nodes layer by layer.

Original: We define events A t ′ ,l = { all layer-l nodes reach the steady-state by time t ′ } and A it′
Revised: We deﬁne events A t (cid:48) ,l = { all layer-l nodes reach the steady-state by time t (cid:48) } and A it

Revised: [Equation]

Original: = { the node i reaches the steady-state by time t ′ } .
Revised: = { the node i reaches the steady-state by time t (cid:48) } .

Original: By treating the nodes in the same layer together,
Revised: By treating the nodes in the same layer together,

Original: [Equation]
Revised: [Equation]

Original: We then prove that P ( A lτ,l |A τ, 1 . . .
Revised: We then prove that P ( A lτ,l |A τ, 1 . . .

Original: A ( l − 1) τ,l − 1 ) ≥ 1 − (2 I l ) /T 2 , where I l = |I
Revised: A ( l − 1) τ,l − 1 ) ≥ 1 − (2 I l ) /T 2 , where I l = |I

Original: l | , by establishing a coupling argument between the target queue (with the help of an artificially constructed alternative queue) and a virtual queue starting from a state sampled from the steady-state distribution and maintaining the steady-state thereafter.
Revised: l | , by establishing a coupling argument between the target queue (with the help of an artiﬁcially constructed alternative queue) and a virtual queue starting from a state sampled from the steady-state distribution and maintaining the steady-state thereafter.

Original: The detailed proof is provided in Appendix E.
Revised: The detailed proof is provided in Appendix E.

Original: In addition, the transient state of the Jackson Network, as well as each resource queue (or each node), can be induced not only by starting from empty state but also by changing the price from the previously posted price.
Revised: In addition, the transient state of the Jackson Network, as well as each resource queue (or each node), can be induced not only by starting from empty state but also by changing the price from the previously posted price.

Original: It is possible that when a new price is posted, the customers who arrived under the previously posted price still remain in the system.
Revised: It is possible that when a new price is posted, the customers who arrived under the previously posted price still remain in the system.

Original: Therefore we also need to ensure that the coupling also occurs with high probability when price changes.
Revised: Therefore we also need to ensure that the coupling also occurs with high probability when price changes.

Original: Proposition 4.
Revised: Proposition 4.

Original: (Coupling probability of Jackson Network starting from steady-state of the previous price.)
Revised: (Coupling probability of Jackson Network starting from steady-state of the previous price.)

Original: Define B t to be the event in which the Jackson Network reaches the steady-state by time t .
Revised: Deﬁne B t to be the event in which the Jackson Network reaches the steady-state by time t .

Original: For t ≥ 2 Lτ , where τ = (log T ) 2 , we have P ( B t ) ≥ 1 − 4 IT 2 .
Revised: For t ≥ 2 Lτ , where τ = (log T ) 2 , we have P ( B t ) ≥ 1 − 4 IT 2 .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: This proposition is derived based on Proposition 3 and Assumption 5.
Revised: This proposition is derived based on Proposition 3 and Assumption 5.

Original: We define B t ′ ,l and B it ′ similarly and bound P ( B 2 lτ,l |B 2 τ, 1 . . .
Revised: We deﬁne B t (cid:48) ,l and B it (cid:48) similarly and bound P ( B 2 lτ,l |B 2 τ, 1 . . .

Original: B 2( l − 1) τ,l − 1 ) ≥ 1 − 4 I l T 2 by decomposing the scenarios by how the arrival and service rates are changing when considering each resource queue.
Revised: B 2( l − 1) τ,l − 1 ) ≥ 1 − 4 I l T 2 by decomposing the scenarios by how the arrival and service rates are changing when considering each resource queue.

Original: 5.2 Regret Bounds of RNRM-UCB and RNRM-TS
Revised: 5.2 Regret Bounds of RNRM-UCB and RNRM-TS

Original: The high probability bounds for mixing time of Jackson Network developed in §5.1 help bound the loss due to the nonstationarity, and thus we are ready to carry out the regret analysis of RNRM-UCB.
Revised: The high probability bounds for mixing time of Jackson Network developed in §5.1 help bound the loss due to the nonstationarity, and thus we are ready to carry out the regret analysis of RNRM-UCB.

Original: Theorem 1.
Revised: Theorem 1.

Original: The T -period cumulative regret of RNRM-UCB is bounded by ˜ O (cid:16) J √ XT (cid:17) .
Revised: The T -period cumulative regret of RNRM-UCB is bounded by ˜ O (cid:16) J √ XT (cid:17) .

Original: Note that the regret lower bound for nominal MAB (where J = 1 ) is Ω( √ XT ) (see, e.g., Theorem 2.1 in Slivkins (2019)), which suggests that our regret bound is tight, if the system size J is fixed.
Revised: Note that the regret lower bound for nominal MAB (where J = 1 ) is Ω( √ XT ) (see, e.g., Theorem 2.1 in Slivkins (2019)), which suggests that our regret bound is tight, if the system size J is ﬁxed.

Original: Proof Sketch.
Revised: Proof Sketch.

Original: The regret of the Warm-up phase is at most linear on the length of the Warm-up
Revised: The regret of the Warm-up phase is at most linear on the length of the Warm-up

Original: Phase, i.e., O ( X log( J 1 / 4 T )) .
Revised: Phase, i.e., O ( X log( J 1 / 4 T )) .

Original: We decompose the relaxed regret in the Learning Phase into two parts.
Revised: We decompose the relaxed regret in the Learning Phase into two parts.

Original: The first part is called loss of nonstationarity and it represents the revenue loss induced by the transient performance when changing the price.
Revised: The ﬁrst part is called loss of nonstationarity and it represents the revenue loss induced by the transient performance when changing the price.

Original: For batch m , we can derive that the expected revenue collected under the selected price x m as r ( ⃗α x m , x m )
Revised: For batch m , we can derive that the expected revenue collected under the selected price x m as r ( (cid:126)α x m , x m )

Original: I m Lτ − O
Revised: I m Lτ − O

Original: ( Lτ ) with Proposition 4.
Revised: ( Lτ ) with Proposition 4.

Original: The second part represents the loss of suboptimality , which is induced by not offering the optimal state-independent price in certain time periods.
Revised: The second part represents the loss of suboptimality , which is induced by not offering the optimal state-independent price in certain time periods.

Original: This part can be further rewritten as ∆( x m ) I m Lτ and ∆( x m ) =
Revised: This part can be further rewritten as ∆( x m ) I m Lτ and ∆( x m ) =

Original: ( r ( ⃗α ˜ x , ˜ x ) − U m (˜ x ))
Revised: ( r ( (cid:126)α ˜ x , ˜ x ) − U m (˜ x ))

Original: + ( U m ( x m ) − L m ( x m ))
Revised: + ( U m ( x m ) − L m ( x m ))

Original: + ( L m ( x m ) − r ( ⃗α x m , x m )) .
Revised: + ( L m ( x m ) − r ( (cid:126)α x m , x m )) .

Original: We bound these three terms separately based on Proposition 2 and Lemma 2.
Revised: We bound these three terms separately based on Proposition 2 and Lemma 2.

Original: Combine all results together, we reach a ˜ O ( J √ XT ) regret bound.
Revised: Combine all results together, we reach a ˜ O ( J √ XT ) regret bound.

Original: We provided detailed proof in Appendix E.
Revised: We provided detailed proof in Appendix E.

Original: We use Bayesian regret as the performance measure of RNRM-TS, which is typically used for the
Revised: We use Bayesian regret as the performance measure of RNRM-TS, which is typically used for the

Original: Bayesian setting and is computed as the expected regret of the instance in expectation over all problem instances.
Revised: Bayesian setting and is computed as the expected regret of the instance in expectation over all problem instances.

Original: Similarly, we define a relaxed Bayesian regret.
Revised: Similarly, we deﬁne a relaxed Bayesian regret.

Original: Definition 4.
Revised: Deﬁnition 4.

Original: Denoting all the events before period t by F t , ∀ t ≥ 0 , we define the relaxed Bayesian regret of policy π by period T as
Revised: Denoting all the events before period t by F t , ∀ t ≥ 0 , we deﬁne the relaxed Bayesian regret of policy π by period T as

Original: [Equation]
Revised: [Equation]

Original: The relaxed Bayesian regret bound of RNRM-TS is in the same order as RNRM-UCB (Appendix E).
Revised: The relaxed Bayesian regret bound of RNRM-TS is in the same order as RNRM-UCB (Appendix E).

Original: Theorem 2.
Revised: Theorem 2.

Original: The T -period cumulative Bayesian regret of RNRM-TS is bounded by ˜ O
Revised: The T -period cumulative Bayesian regret of RNRM-TS is bounded by ˜ O

Original: ( J √ XT ) .
Revised: ( J √ XT ) .

Original:  Numerical Results
Revised:  Numerical Results

Original: We consider T = 2000 , I = 2 with capacity c 1 =
Revised: We consider T = 2000 , I = 2 with capacity c 1 =

Original: c
Revised: c

Original: = 2 ,
Revised: = 2 ,

Original: J = 3 with product-resource orders { 1 , 2 , 1 → 2 } , and X = { [1 , 2 , 3] , [2 , 1 , 3] , [2 , 2 , 3] } .
Revised: J = 3 with product-resource orders { 1 , 2 , 1 → 2 } , and X = { [1 , 2 , 3] , [2 , 1 , 3] , [2 , 2 , 3] } .

Original: We set the underlying arrival rates of candidate prices as { [4 , 2 , 4] , [3 , 4 , 3] , [2 . 5 , 3 , 2 . 5] } and use uniform service rates [5 , 4 . 5] for all prices.
Revised: We set the underlying arrival rates of candidate prices as { [4 , 2 , 4] , [3 , 4 , 3] , [2 . 5 , 3 , 2 . 5] } and use uniform service rates [5 , 4 . 5] for all prices.

Original: We adopt the blind network revenue management algorithm proposed in Besbes and Zeevi (2012) to our setting as a benchmark.
Revised: We adopt the blind network revenue management algorithm proposed in Besbes and Zeevi (2012) to our setting as a benchmark.

Original: This benchmark algorithm tries each price for θT/X periods and then selects the price with the highest revenue for the rest of the periods.
Revised: This benchmark algorithm tries each price for θT/X periods and then selects the price with the highest revenue for the rest of the periods.

Original: We set θ as 0 .
Revised: We set θ as 0 .

Original: 1 , and 0 .
Revised: 1 , and 0 .

Original: in three
Revised: in three

Original: Besbes and Zeevi (2012) benchmarks, respectively (the same as the settings in Table 1 in the original paper Besbes and Zeevi 2012).
Revised: Besbes and Zeevi (2012) benchmarks, respectively (the same as the settings in Table 1 in the original paper Besbes and Zeevi 2012).

Original: To conclude, we implement five pricing algorithms: RNRM-UCB,
Revised: To conclude, we implement ﬁve pricing algorithms: RNRM-UCB,

Original: RNRM-TS, and three benchmark policies with different values of θ .
Revised: RNRM-TS, and three benchmark policies with different values of θ .

Original: Figure 1a shows the offered price over periods of each algorithm and Figure 1b depicts the cumulative time-average relaxed regret, i.e., (cid:80) tt ′
Revised: Figure 1a shows the offered price over periods of each algorithm and Figure 1b depicts the cumulative time-average relaxed regret, i.e., (cid:80) tt (cid:48)

Original: =1 ( Y LP t ′ − Y πt ′ ) /t .
Revised: =1 ( Y LP t (cid:48) − Y πt (cid:48) ) /t .

Original: For all algorithms (except “OPT”), the revenue is collected from the simulation where the system starts with no customers at time 0 .
Revised: For all algorithms (except “OPT”), the revenue is collected from the simulation where the system starts with no customers at time 0 .

Original: We can observe that both RNRM-UCB and RNRM-TS identify the best price (P1) through the learning process and the time average regret shrinks to zero very quickly.
Revised: We can observe that both RNRM-UCB and RNRM-TS identify the best price (P1) through the learning process and the time average regret shrinks to zero very quickly.

Original: [Figure]
Revised: [Figure]

Original:  Conclusion and Broader Impacts
Revised:  Conclusion and Broader Impacts

Original: We proposed two new batch bandit algorithms for RNRM problems with multiple products and multiple reusable resources.
Revised: We proposed two new batch bandit algorithms for RNRM problems with multiple products and multiple reusable resources.

Original: We established optimal regret upper bounds and demonstrated superior computational results.
Revised: We established optimal regret upper bounds and demonstrated superior computational results.

Original: This work solves an important practical problem and we believe that our approach will open many doors to learning more complex stochastic systems (especially in network settings).
Revised: This work solves an important practical problem and we believe that our approach will open many doors to learning more complex stochastic systems (especially in network settings).

Original: There are two immediate future research questions: (1) how we can accommodate continuous price space; and (2) how we can extend arrivals and services to general distributions.
Revised: There are two immediate future research questions: (1) how we can accommodate continuous price space; and (2) how we can extend arrivals and services to general distributions.

Original: Both directions would need significant methodological innovations to be developed.
Revised: Both directions would need signiﬁcant methodological innovations to be developed.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: The authors would like to thank anonymous reviewers for their detailed and helpful comments, which have helped improve the content and exposition of this paper.
Revised: The authors would like to thank anonymous reviewers for their detailed and helpful comments [Figure] which have helped improve the content and exposition of this paper

Original: This research is partially supported by an Amazon research award and Department of Energy award #DE-SC0018018.
Revised: This research is partially supported by an Amazon research award and Department of Energy award #DE-SC [Figure]

Original: B Illustrative Example
Revised: B Illustrative Example

Original: We refer to this simple illustrative example several times in the rest of the paper.
Revised: We refer to this simple illustrative example several times in the rest of the paper.

Original: Consider 4 types of resources and 4 types of products:
Revised: Consider 4 types of resources and 4 types of products:

Original: • Product No. 1 : ( s, 1 , 3 , t ) , i.e., SQL Database → EC • Product No. 2 : ( s, 1 , 3 , 4 , t ) , i.e., SQL Database → EC2 → SAP • Product No. 3 : ( s, 2 , 3 , t ) , i.e., NoSQL Database → EC • Product No. 4 : ( s, 2 , 3 , 4 , t ) , i.e., NoSQL Database → EC2 → SAP
Revised: • Product No. 1 : ( s, 1 , 3 , t ) , i.e., SQL Database → EC • Product No. 2 : ( s, 1 , 3 , 4 , t ) , i.e., SQL Database → EC2 → SAP • Product No. 3 : ( s, 2 , 3 , t ) , i e., NoSQL Database → EC • Product No. 4 : ( s, 2 , 3 , 4 , t ) , i.e., NoSQL Database → EC2 → SAP

Original: Typically, customers will choose between an SQL based or a NoSQL based database service to store and preprocess the data.
Revised: Typically, customers will choose between an SQL based or a NoSQL based database service to store and preprocess the data.

Original: After running algorithms on elastic cloud compute EC2, only a subset of customers may choose to further integrate SAP applications.
Revised: After running algorithms on elastic cloud compute EC2, only a subset of customers may choose to further integrate SAP applications.

Original: The network representation is given by
Revised: The network representation is given by

Original: Figure 2, and it is straightforward to check that this acyclic graph satisfies Assumption 2.
Revised: Figure 2, and it is straightforward to check that this acyclic graph satisﬁes Assumption 2.

Original: We assign a price vector x = ( x 1 , x 2 , x 3 , x 4 ) to these four products.
Revised: We assign a price vector x = ( x 1 , x 2 , x 3 , x 4 ) to these four products.

Original: Note that we do not assign prices to resources.
Revised: Note that we do not assign prices to resources.

Original: We only sell individual products (as bundles of resources).
Revised: We only sell individual products (as bundles of resources).

Original: Each posted price vector induces two quantities, namely, α ( x ) and P ( x ) .
Revised: Each posted price vector induces two quantities, namely, α ( x ) and P ( x ) .

Original: Here α ( x ) is the total arrival rate of all products given price vector x while P ( x ) is the so-called (Markovian) “routing matrix” that captures the demand flows for products.
Revised: Here α ( x ) is the total arrival rate of all products given price vector x while P ( x ) is the so-called (Markovian) “routing matrix” that captures the demand ﬂows for products.

Original: One can represent a product j ∈ J using an s - t path, i.e., ( s, i 1 , . .
Revised: One can represent a product j ∈ J using an s - t path, i.e., ( s, i 1 , . .

Original: , i n , t ) .
Revised: , i n , t ) .

Original: Assume that Assumption 3 holds.
Revised: Assume that Assumption 3 holds.

Original: Then we can write the demand rate for this particular product j as
Revised: Then we can write the demand rate for this particular product j as

Original: [Equation]
Revised: [Equation]

Original: where p i,i ′ are the ( i, i ′ )
Revised: where p i,i (cid:48) are the ( i, i (cid:48) )

Original: th entry of the routing probability matrix P ( x ) .
Revised: th entry of the routing probability matrix P ( x ) .

Original: Note that in our setting, the mapping x (cid:55)→ α ( x ) , P ( x ) is unknown to the decision maker.
Revised: Note that in our setting, the mapping x (cid:55)→ α ( x ) , P ( x ) is unknown to the decision maker.

Original: Thus, we need to offer different price vectors x to learn this mapping.
Revised: Thus, we need to offer different price vectors x to learn this mapping.

Original: And we also assume this mapping is nonparametric.
Revised: And we also assume this mapping is nonparametric.

Original: For the sake of illustration, we give the following linear mapping as an example.
Revised: For the sake of illustration, we give the following linear mapping as an example.

Original: [Equation]
Revised: [Equation]

Original: Suppose the price vector x = (1 , 2 , 2 , 5) .
Revised: Suppose the price vector x = (1 , 2 , 2 , 5) .

Original: In this case, the total arrival rate α ( x ) = 100 and the routing matrix P ( x ) is given by
Revised: In this case, the total arrival rate α ( x ) = 100 and the routing matrix P ( x ) is given by

Original: [Equation]
Revised: [Equation]

Original:    p

Original: [Equation]

Original: p

Original: [Equation]

Original: p

Original: [Equation]

Original: p

Original: [Equation]

Original: x

Original: [Equation]

Original: .p

Original: [Equation]

Original: p

Original: [Equation]

Original: p

Original: [Equation]

Original: , giving rise to
Revised: giving rise to

Original: Product Demand:
Revised: Product Demand:

Original: [Equation]
Revised: [Equation]

Original: Server Demand:
Revised: Demand:

Original: [Equation]
Revised: [Equation]

Original: Note that in this example, Assumption 3 also means that out of all the customers who have chosen
Revised: Note that in this example, Assumption 3 also means that out of all the customers who have chosen

Original: EC2, the proportion of choosing SAP is 0.3 and the proportion of not choosing SAP is 0.7 (regardless of database options).
Revised: EC2, the proportion of choosing SAP is 0.3 and the proportion of not choosing SAP is 0.7 (regardless of database options).

Original: C Technical Results in Section
Revised: C Technical Results in Section

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: We show that the decisions variables ( π, a, Y ) associated with any admissible policy must satisfy the constraints of the linear program (1), and the expected revenue of this admissible policy is exactly the corresponding objective value in (1).
Revised: We show that the decisions variables ( π, a, Y ) associated with any admissible policy must satisfy the constraints of the linear program (1), and the expected revenue of this admissible policy is exactly the corresponding objective value in (1).

Original: Consider an arbitrary policy ˆ π .
Revised: Consider an arbitrary policy ˆ π .

Original: We obtain the values of decision variables by the following steps.
Revised: We obtain the values of decision variables by the following steps.

Original: Variable ˆ a y t takes the value of probability that the system has y customers at the beginning of period t under the policy
Revised: Variable ˆ a y t takes the value of probability that the system has y customers at the beginning of period t under the policy

Original: ˆ π for y ∈ { 0 , 1 , . . .
Revised: ˆ π for y ∈ { 0 , 1 , . . .

Original: , ∞} I and t = 1 , . . .
Revised: , ∞} I and t = 1 , . . .

Original: , T .
Revised: , T .

Original: Therefore, constraints (1b) are immediately satisfied.
Revised: Therefore, constraints (1b) are immediately satisﬁed.

Original: Variable ˆ π xy t = 1 if the policy ˆ π chooses price x at state ( y , t ) , otherwise, ˆ π xy t = 0 , for y ∈ { 0 , 1 , . . .
Revised: Variable ˆ π xy t = 1 if the policy ˆ π chooses price x at state ( y , t ) , otherwise, ˆ π xy t = 0 , for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T , y ∈ X .
Revised: , T , y ∈ X .

Original: As a result, constraints (1c) hold because in the left-hand side we have one variable equal to 1 and others all zero.
Revised: As a result, constraints (1c) hold because in the left-hand side we have one variable equal to 1 and others all zero.

Original: Variable ˆ Y y t takes the value of expected revenue collected before the end of period T from the customers who arrive during period t , when the system has y customers at the beginning of period t , for y ∈ { 0 , 1 , . . .
Revised: Variable ˆ Y y t takes the value of expected revenue collected before the end of period T from the customers who arrive during period t , when the system has y customers at the beginning of period t , for y ∈ { 0 , 1 , . . .

Original: , ∞} I .
Revised: , ∞} I .

Original: Consider arbitrary y and t , we denote the price chosen at state ( y , t ) by the policy ˆ π as ˆ x .
Revised: Consider arbitrary y and t , we denote the price chosen at state ( y , t ) by the policy ˆ π as ˆ x .

Original: Then, by plugging in ˆ π ˆ xy t = 1 , in the right-hand side of constraints (1d), we have ⃗α T x x , where ⃗α T x denotes the average number of arrived customers for consuming products during period t and ˆ x is the revenue vector of these products.
Revised: Then, by plugging in ˆ π ˆ xy t = 1 , in the right-hand side of constraints (1d), we have (cid:126)α T x x , where (cid:126)α T x denotes the average number of arrived customers for consuming products during period t and ˆ x is the revenue vector of these products.

Original: Therefore, ⃗α T x x denotes the expected revenue collected from customers who arrive during period t (assuming all arrived customers are served successfully).
Revised: Therefore, (cid:126)α T x x denotes the expected revenue collected from customers who arrive during period t (assuming all arrived customers are served successfully).

Original: Consequently, we can conclude that the right-hand side of (1d) is as large as the left-hand side, because the right-hand side may also compute the revenue collected after the end of period T .
Revised: Consequently, we can conclude that the right-hand side of (1d) is as large as the left-hand side, because the right-hand side may also compute the revenue collected after the end of period T .

Original: Hence, constraints (1d) are also satisfied.
Revised: Hence, constraints (1d) are also satisﬁed.

Original: By the value assignment steps of ˆ a y t and ˆ Y y t , for y ∈ { 0 , 1 , . . .
Revised: By the value assignment steps of ˆ a y t and ˆ Y y t , for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T , one has that the corresponding LP objective value at solution (ˆ π, ˆ a, ˆ J ) is Y ˆ π .
Revised: , T , one has that the corresponding LP objective value at solution (ˆ π, ˆ a, ˆ J ) is Y ˆ π .

Original: Therefore, we have Y LP ≥ Y π , for any admissible policy π , and thus also for the optimal state-dependent policy π ∗ .
Revised: Therefore, we have Y LP ≥ Y π , for any admissible policy π , and thus also for the optimal state-dependent policy π ∗ .

Original: By defining ˜ x = argmax x ∈X ⃗α T x x , we find an optimal solution (˜ π, ˜ a, ˜ Y ) to the LP in (1) as follows.
Revised: By deﬁning ˜ x = argmax x ∈X (cid:126)α T x x , we ﬁnd an optimal solution (˜ π, ˜ a, ˜ Y ) to the LP in (1) as follows.

Original: [Equation]
Revised: [Equation]

Original: where y 

Revised: where y

Original: =
Revised: =

Original: [0] J .
Revised: [0] J .

Original: Then, we can compute Y LP = T · ⃗α T ˜ x
Revised: Then, we can compute Y LP = T · (cid:126)α T ˜ x

Original: ˜ x .
Revised: ˜ x .

Original: Note that ⃗α T x x is the stationary revenue rate of price x , denoted as r ( ⃗α x , x ) .
Revised: Note that (cid:126)α T x x is the stationary revenue rate of price x , denoted by r ( (cid:126)α x , x ) .

Original: Q.E.D.
Revised: Q.E.D.

Revised: Discussion over state variable deﬁnitions.

Revised: In LP (1), we use ( y , t ) as the state variable, where y reﬂects the number of customers who are consuming or in the waiting queue of each type of resource.

Original: D Technical Results in Section 4. Proof of Proposition 
Revised: With Proposition 1, we know that Y LP = T · (cid:126)α T ˜ x ˜ x is a valid upper bound of the state-dependent optimal policy with ( y , t ) the state variable, i.e., Y ∗ ≤ Y LP . One may consider a different (and more reﬁned) deﬁnition for the state variable.

Revised: Consider ( W, t ) as the new state variable with W ∈ R

Revised: I × J , where W ij represents the number of customers who are requesting product j and currently consuming or in the waiting queue of resource i .

Revised: We denote the optimal state-dependent policy under this new state variable deﬁnition by π ∗∗ and the corresponding expected revenue by Y ∗∗ .

Revised: Then we have two results.

Revised: First, it is evident that the optimal state-dependent policy under the original state variable deﬁnition is a feasible policy for the new state variable deﬁnition and thus Y ∗∗ ≥ Y ∗ .

Revised: Second, we still have Y ∗∗ ≤ Y LP .

Revised: To prove this, we can replace the state variables in decision variables in LP (1) to obtain a new linear program, denoted by LP2.

Revised: Then following the same analysis as in Proposition 1, we can readily show that the optimal objective value of LP2 is a valid upper bound of Y ∗∗ and

Revised: Y LP2 =

Revised: Y LP = T · (cid:126)α T ˜ x

Revised: ˜ x .

Revised: Therefore, we have Y ∗∗ ≤ Y LP2 =

Revised: Y LP .

Revised: In fact, under any valid state variable deﬁnition, Y LP provides an upper bound of the state-dependent optimal policy.

Revised: D Technical Results in Section 4. Proof

Revised: of Proposition 2.

Original: The proof is based on the concentration bound for sub-exponential random variables.
Revised: The proof is based on the concentration bound for sub-exponential random variables.

Original: For product j ∈ J off , their demand is off, i.e., α j ( x ) = 0 , j ∈ J off ( x ) .
Revised: For product j ∈ J off , their demand is off, i.e., α j ( x ) = 0 , j ∈ J off ( x ) .

Original: In this proof, we only consider one price and thus we use α j to represent α j ( x ) for notation simplicity.
Revised: In this proof, we only consider one price and thus we use α j to represent α j ( x ) for notation simplicity.

Original: We can rewrite the right-hand-side as:
Revised: We can rewrite the right-hand-side as:

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: We can derive that for one product j ∈ J on ( x ) :
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: We can derive that for one product j ∈ J on ( x ) :

Original: [Equation]
Revised: [Equation]

Original:  The exponential arrival time intervals are also sub-exponential random variables with parameters ( 4 α 2 j , 2 α j ) , denoted by SE ( 4 α 2 j , 2 α j ) , see Lemma 1 in the appendix.
Revised: 5b [Equation] The exponential arrival time intervals are also sub-exponential random variables with parameters ( 4 α 2 j , 2 α j ) , denoted by SE ( 4 α 2 j , 2 α j ) , see Lemma 1 in the appendix.

Original: Consider the concentration inequality for i.i.d.
Revised: Consider the concentration inequality for i

Revised: [Equation]

Original:  exponential variables ξ i , i = 1 , . ..
Revised: i.d. exponential variables ξ i , i = 1 , . .

Original: n with mean 1 /λ :
Revised: n with mean 1 /λ :

Original: [Equation]
Revised: [Equation]

Original: By the assumption n j ( x ) ≥ 8 log( J 14 T ) , we can compute
Revised: By the assumption n j ( x ) ≥ 8 log( J 1 4 T ) [Equation] we can compute

Original: [Equation]
Revised: [Equation]

Original: and thus we can apply the first inequality in (6) to (5).
Revised: and thus we can apply the ﬁrst inequality in (6) to (5).

Original: Therefore, we can derive
Revised: Therefore [Equation] we can derive

Original: P 
Revised: P 

Original: [Equation]
Revised: [Equation]

Original: Then, we can compute a lower bound of (4) as
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: |J on ( x ) | ≥ (cid:18) 1 − J · T(cid:19)
Revised: we can compute a lower bound of (4) as

Original: J ≥ 1 − 2 J J · T 4 = 1 − T 4 . Q.E.D.
Revised: [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: (Sub-exponential property, Lemma 3 in Jia et al.
Revised: (Sub-exponential property, Lemma 3 in Jia et al.

Original: If a random variable X follows an exponential distribution with mean 1 /λ , then the random variable X − 1 λ is ( 4 λ 2 , 2 λ ) -sub-exponential.
Revised: If a random variable X follows an exponential distribution with mean 1 /λ , then the random variable X − 1 λ is ( 4 λ 2 , 2 λ ) -sub-exponential.

Original: E Technical Results in Section
Revised: E Technical Results in Section

Original: Proof of Proposition 3.
Revised: Proof of Proposition 3.

Original: In this proof, we analyze the coupling time of nodes layer by layer.
Revised: In this proof, we analyze the coupling time of nodes layer by layer.

Original: We define events as follows (all starting from empty state). 
Revised: We deﬁne events as follows (all starting from empty state). [Equation]

Revised: the Jackson Network reaches the steady-state by time t }

Original: [Equation]
Revised: [Equation]

Revised: layer

Revised: [Equation]

Revised: the steady-state

Revised: [Equation]

Revised: time

Revised: [Equation]

Revised: the

Revised: [Equation]

Original:  First, we discuss the coupling analysis of an M/M/c queue, which is the basis for completing this proof.
Revised: [Equation] by time [Equation] First, we discuss the coupling analysis of an M/M/c queue, which is the basis for completing this proof.

Original: With the assumption log > 4 , the probability that an M/M/c queue reaches the steady-state from empty state after (log T ) 2 periods of time is bounded by 1 − 2 /T 2 , see Lemma 3 (Proposition 4 in Jia et al.
Revised: With the assumption log > 4 , the probability that an M/M/c queue reaches the steady-state from empty state after (log T ) 2 periods of time is bounded by 1 − 2 /T 2 , see Lemma 3 (Proposition 4 in Jia et al.

Original: These results are derived following the law of total probability, the first-order stochastic dominance between busy times of an M/M/c queue, and the concentration inequality for independent samples.
Revised: These results are derived following the law of total probability, the ﬁrst-order stochastic dominance between busy times of an M/M/c queue, and the concentration inequality for independent samples.

Original: We denote node i is a prerequisite node of i ′ , i < i ′ if there exists a product requires to use nodes i and i ′ .
Revised: We denote node i is a prerequisite node of i (cid:48) , i < i (cid:48) if there exists a product requires to use nodes i and i (cid:48) .

Original: Thus, it is easy to show that the prerequisite nodes of a Layer- l node must have smaller layer indices.
Revised: Thus, it is easy to show that the prerequisite nodes of a Layer- l node must have smaller layer indices.

Original: In our Jackson Network, node i ′ reaches its steady-state indicates that the arrival process of node i ′ is a stable Poisson process, and therefore it requires that all the prerequisite nodes also reach the steady-state.
Revised: In our Jackson Network, node i (cid:48) reaches its steady-state indicates that the arrival process of node i (cid:48) is a stable Poisson process, and therefore it requires that all the prerequisite nodes also reach the steady-state.

Original: In this proof, we analyze the nodes layer by layer.
Revised: In this proof, we analyze the nodes layer by layer.

Original: By treating the nodes in the same layer together, we can derive that
Revised: By treating the nodes in the same layer together, we can derive that

Original: [Equation]
Revised: [Equation]

Original: In short, we derive a lower bound of this mixing probability by requiring the Layer- l nodes to reach the steady-state within the first L × τ periods, conditional on that Layer- l ′ nodes also reach the steady-state within the first l ′ × τ periods for l ′ < l , or equivalently, all prerequisite nodes of Layer- l nodes reach the steady-state within ( l − 1) × τ periods.
Revised: In short, we derive a lower bound of this mixing probability by requiring the Layer- l nodes to reach the steady-state within the ﬁrst L × τ periods, conditional on that Layer- l (cid:48) nodes also reach the steady-state within the ﬁrst l (cid:48) × τ periods for l (cid:48) < l , or equivalently, all prerequisite nodes of Layer- l nodes reach the steady-state within ( l − 1) × τ periods.

Original: Layer-1 Nodes.
Revised: Layer-1 Nodes.

Original: In the first τ periods, by applying the probability bound for an M/M/c queue, we can easily conclude that all Layer-1 nodes (the correspondent M/M/c queues) reach the steady-state with probability 1 − 2 T 3 .
Revised: In the ﬁrst τ periods, by applying the probability bound for an M/M/c queue, we can easily conclude that all Layer-1 nodes (the correspondent M/M/c queues) reach the steady-state with probability 1 − 2 T 3 .

Original: Recall I 1 is the collection of all Layer-1 nodes and |I 1 | is the cardinality.
Revised: Recall I 1 is the collection of all Layer-1 nodes and |I 1 | is the cardinality.

Original: Therefore we can compute that the probability that all Layer-1 nodes reach the steady-state by first τ periods is
Revised: Therefore we can compute that the probability that all Layer-1 nodes reach the steady-state by ﬁrst τ periods is

Original: [Equation]
Revised: [Equation]

Original: because the queueing evolution of all Layer-1 nodes are independent of each other and thus we can simply multiply the probability together, in the first equality in the above expression.
Revised: because the queueing evolution of all Layer-1 nodes are independent of each other and thus we can simply multiply the probability together, in the ﬁrst equality in the above expression.

Original: Layer-2 Nodes.
Revised: Layer-2 Nodes.

Original: We then analyze the Layer-2 nodes.
Revised: We then analyze the Layer-2 nodes.

Original: Consider one arbitrary layer-2 node, denoted by i without loss of generality.
Revised: Consider one arbitrary layer-2 node, denoted by i without loss of generality.

Original: We then focus on the corresponding queue of this node i conditional on that all Layer-1 nodes have reached steady-state.
Revised: We then focus on the corresponding queue of this node i conditional on that all Layer-1 nodes have reached steady-state.

Original: We firstly show a virtual queue can reach steadystate within the next τ periods with probability at least 1 − 2 T 2 .
Revised: We ﬁrstly show a virtual queue can reach steadystate within the next τ periods with probability at least 1 − 2 T 2 .

Original: Then we show that an alternative queue of the actual queue takes a shorter time to reach steady-state and thus the actual queue can also reach steady-state within the next τ periods with probability at least 1 − 2 T 2 .
Revised: Then we show that an alternative queue of the actual queue takes a shorter time to reach steady-state and thus the actual queue can also reach steady-state within the next τ periods with probability at least 1 − 2 T 2 .

Original: • Virtual queue.
Revised: • Virtual queue.

Original: We consider a virtual queue that rejects all customers who arrived before τ periods.
Revised: We consider a virtual queue that rejects all customers who arrived before τ periods.

Original: Then by Lemma 3, this virtual queue reaches the 
Revised: Then by Lemma 3, this virtual queue reaches the steady-state with probability 1 − 2 T 2 after the next τ periods, i.e., after the ﬁrst 2 τ periods.

Original: • Alternative queue.
Revised: • Alternative queue.

Original: Denote the probability that the number of customers in Node i at the end of period τ is less than N by P i ( N ) , including the customers both using the server or in the waiting queue of node i .
Revised: Denote the probability that the number of customers in Node i at the end of period τ is less than N by P i ( N ) , including the customers both using the server or in the waiting queue of node i .

Original: The actual queue accepts customers during the first τ periods with a changing rate from an initial rate to the stable arrival rate λ i (conditional on that all Layer-1 nodes reach steady-state and thus provides stable Poisson arrivals for nodes with high layer indices).
Revised: The actual queue accepts customers during the ﬁrst τ periods with a changing rate from an initial rate to the stable arrival rate λ i (conditional on that all Layer-1 nodes reach steady-state and thus provides stable Poisson arrivals for nodes with high layer indices).

Original: Denote the demand for all products to the Jackson Network by α .
Revised: Denote the demand for all products to the Jackson Network by α .

Original: The stable arrival rate λ i = αp si + α
Revised: The stable arrival rate λ i = αp si + α

Original: (cid:80) i ′ :(
Revised: (cid:80) i (cid:48) :(

Original: i ′ ,i ∈ E ) p si ′ p i ′ i , where αp si is the initial rate, i.e., the demand rate of products that request node i as the first service, and αp si ′ p
Revised: i (cid:48) ,i ∈ E ) p si (cid:48) p i (cid:48) i , where αp si is the initial rate, i.e., the demand rate of products that request node i as the ﬁrst service, and αp si (cid:48)

Original:  i ′
Revised: p i (cid:48)

Original: i is the stable demand flowing from prerequisite node i ′ to node i .
Revised: i is the stable demand ﬂowing from prerequisite node i (cid:48) to node i .

Original: Consider an alternative queue that accepts customers at the stable rate λ i .
Revised: Consider an alternative queue that accepts customers at the stable rate λ i .

Original: If the queue starts from time 0 , then the probability that this virtual queue has customers with a size less than N at the end of period τ is smaller than P
Revised: If the queue starts from time 0 , then the probability that this virtual queue has customers with a size less than N at the end of period τ is smaller than P

Original: i ( N ) .
Revised: i ( N ) .

Original: This is because this alternative queue accepts more customers while the service times are all following the same distribution.
Revised: This is because this alternative queue accepts more customers while the service times are all following the same distribution.

Original: Then we postpone the starting point of this alternative queue to a time stamp between 0 and τ such that the probability that this alternative queue has customers with a size less than N at the end of period τ is the same as P i ( N ) .
Revised: Then we postpone the starting point of this alternative queue to a time stamp between 0 and τ such that the probability that this alternative queue has customers with a size less than N at the end of period τ is the same as P i ( N ) .

Original: Note that this timestamps exists, which can be easily proved by the memoryless property of the exponential service time assumptions.
Revised: Note that this timestamps exists, which can be easily proved by the memoryless property of the exponential service time assumptions.

Original: Then we have the following two results.
Revised: Then we have the following two results.

Original: First, this alternative queue has the same evolution path as the actual queue.
Revised: First, this alternative queue has the same evolution path as the actual queue.

Original: Second, because this alternative queue starts accepting customers earlier than time τ , and thus it will reach steady-state earlier than the virtual queue.
Revised: Second, because this alternative queue starts accepting customers earlier than time τ , and thus it will reach steady-state earlier than the virtual queue.

Original: We provide an illustration of the actual, alternative, and virtual queues in Figure 3.
Revised: We provide an illustration of the actual, alternative, and virtual queues in Figure 3.

Revised: The right part shows the probability that the queue has a different number of customers at time τ .

Original:  alternative and the actual queues and thus they are equivalent systems after time τ .
Revised: These probabilities are the same for the alternative and the actual queues and thus they are equivalent systems after time τ .

Original: To conclude, the actual queue can reach the steady-state with a higher probability after the first 2 τ periods.
Revised: To conclude, the actual queue can reach the steady-state with a higher probability after the ﬁrst 2 τ periods.

Original: Similarly, the queueing evolution of all Layer-2 nodes are independent of each other and thus we can simply multiply the probability together.
Revised: Similarly, the queueing evolution of all Layer-2 nodes are independent of each other and thus we can simply multiply the probability together.

Original: Therefore, we have that for all Layer-2 nodes
Revised: Therefore, we have that for all Layer-2 nodes

Original: [Equation]
Revised: [Equation]

Original: Layers with Higher Indices.
Revised: Layers with Higher Indices.

Original: Then we apply the same analysis steps of the Layer-2 nodes to nodes in layers with higher indices.
Revised: Then we apply the same analysis steps of the Layer-2 nodes to nodes in layers with higher indices.

Original: Conditional on the events that all nodes in layers with lower indices have reached the steady-state by the end of ( l − 1) τ periods, we can compute that all nodes of layer l can reach steady-state in the next τ period with a probability: 
Revised: Conditional on the events that all nodes in layers with lower indices have reached the steady-state by the end of ( l − 1) τ periods, we can compute that all nodes of layer l can reach steady-state in the next τ period with a probability: [Equation]

Original: [Equation]
Revised: [Equation]

Original: With the results in (9), (10), and (11), we are ready to bound the probability in (8):
Revised: With the results in (9), (10), and (11), we are ready to bound the probability in (8):

Original: [Equation]
Revised: [Equation]

Original: We refer to Figure 4 for a direct representation of the above proof sketch, which shows the mixing time analysis of an illustrative example.
Revised: We refer to Figure 4 for a direct representation of the above proof sketch, which shows the mixing time analysis of an illustrative example.

Original: [Figure]
Revised: [Figure]

Original: Proof of Proposition 4.
Revised: Proof of Proposition 4.

Original: Similar to the proof of Proposition 4, we analyze the mixing time of nodes layer by layer.
Revised: Similar to the proof of Proposition 4, we analyze the mixing time of nodes layer by layer.

Original: Before conducting the proof, we introduce an existing work for an M/M/c queue mixing time.
Revised: Before conducting the proof, we introduce an existing work for an M/M/c queue mixing time.

Original: We similarly define the following events when the Jackson Network starts to operate under a new price starting from the steady-state of the old price:
Revised: We similarly deﬁne the following events when the Jackson Network starts to operate under a new price starting from the steady-state of the old price:

Original: [Equation]
Revised: [Equation]

Original: Lemma 4 says that for one M/M/c queue where Assumption 5 holds, then the probability that the queue reaches a new steady-state from an old steady-state within 2 τ time periods after the price change is bounded by 1 − 4 /T 2 .
Revised: Lemma 4 says that for one M/M/c queue where Assumption 5 holds, then the probability that the queue reaches a new steady-state from an old steady-state within 2 τ time periods after the price change is bounded by 1 − 4 /T 2 .

Original: To derive this result, the authors decompose possible price changes into the following two cases.
Revised: To derive this result, the authors decompose possible price changes into the following two cases.

Original: The first is when the arrival rate changes from a higher value to a lower value and the second is vice versa.
Revised: The ﬁrst is when the arrival rate changes from a higher value to a lower value and the second is vice versa.

Original: For each case, the authors further consider two sub-cases: the unit service rate increase or decrease.
Revised: For each case, the authors further consider two sub-cases: the unit service rate increase or decrease.

Original: Consider the Jackson Network system in our problem.
Revised: Consider the Jackson Network system in our problem.

Original: We can bound the probability by:
Revised: We can bound the probability by:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The above inequality is derived following the same steps in (8) and we neglect the details here.
Revised: The above inequality is derived following the same steps in (8) and we neglect the details here.

Original: Assumption 5 allows us to apply Lemma 4 when we analyze one specific node, i.e., one specific
Revised: Assumption 5 allows us to apply Lemma 4 when we analyze one speciﬁc node, i.e., one speciﬁc

Original: M/M/c queue.
Revised: M/M/c queue.

Original: The structure of the following analysis is similar to the structure of the proof of
Revised: The structure of the following analysis is similar to the structure of the proof of

Original: Proposition 3, we first analyze Layer-1 nodes and then the nodes with higher layer indices.
Revised: Proposition 3, we ﬁrst analyze Layer-1 nodes and then the nodes with higher layer indices.

Original: For nodes with higher layer indices, we apply the aforementioned proposition to a virtual queue, and then show this virtual queue takes longer than an alternative system with the actual queue.
Revised: For nodes with higher layer indices, we apply the aforementioned proposition to a virtual queue, and then show this virtual queue takes longer than an alternative system with the actual queue.

Original: Layer-1 Nodes.
Revised: Layer-1 Nodes.

Original: In the first 2 τ periods, by applying the probability bound for an M/M/c queue with price change (Lemma 4), we can easily conclude that all Layer-1 nodes (the correspondent M/M/c queues) reach the steady-state with probability 1 − 4 T 2 .
Revised: In the ﬁrst 2 τ periods, by applying the probability bound for an M/M/c queue with price change (Lemma 4), we can easily conclude that all Layer-1 nodes (the correspondent M/M/c queues) reach the steady-state with probability 1 − 4 T 2 .

Original: Consider the probabilistic independency of the coupling process among Layer-1 nodes, the probability that all Layer-1 nodes reach the steady-state by the first 2 τ periods is
Revised: Consider the probabilistic independency of the coupling process among Layer-1 nodes, the probability that all Layer-1 nodes reach the steady-state by the ﬁrst 2 τ periods is

Original: [Equation]
Revised: [Equation]

Original: Layer-2 Nodes and Nodes with Higher Layer Indices.
Revised: Layer-2 Nodes and Nodes with Higher Layer Indices.

Original: We then analyze nodes with a layer indexor higher.
Revised: We then analyze nodes with a layer indexor higher.

Original: Consider one arbitrary Layer-l node, denoted by i without loss of generality.
Revised: Consider one arbitrary Layer-l node, denoted by i without loss of generality.

Original: We then focus on the corresponding queue of this node i conditional on that all prerequisites nodes have reached steady-state.
Revised: We then focus on the corresponding queue of this node i conditional on that all prerequisites nodes have reached steady-state.

Original: We firstly show a virtual queue can reach steady-state within the next 2 τ periods with probability at least 1 − 4 T 2 .
Revised: We ﬁrstly show a virtual queue can reach steady-state within the next 2 τ periods with probability at least 1 − 4 T 2 .

Original: Then we show that an alternative queue of the actual queue takes a shorter time to reach steady-state and thus the actual queue can also reach steady-state within the next 2 τ periods with probability at least 1 − 4 T 2 .
Revised: Then we show that an alternative queue of the actual queue takes a shorter time to reach steady-state and thus the actual queue can also reach steady-state within the next 2 τ periods with probability at least 1 − 4 T 2 .

Original: • Virtual queue.
Revised: • Virtual queue.

Original: We consider a virtual queue that accepts customers under the old queueing parameters in the first 2( l − 1) τ periods and then accepts customers under the new queueing parameter in the next 2 τ periods.
Revised: We consider a virtual queue that accepts customers under the old queueing parameters in the ﬁrst 2( l − 1) τ periods and then accepts customers under the new queueing parameter in the next 2 τ periods.

Original: With Assumption 5, we can directly apply the results for one M/M/c queue to reach steady-state starting from a steady-state and conclude that this virtual queue reaches the steady-state with probability 1 − 4 T 2 after the next 2 τ periods, i.e., after the first 2 lτ periods.
Revised: With Assumption 5, we can directly apply the results for one M/M/c queue to reach steady-state starting from a steady-state and conclude that this virtual queue reaches the steady-state with probability 1 − 4 T 2 after the next 2 τ periods, i.e., after the ﬁrst 2 lτ periods.

Original: • Alternative queue.
Revised: • Alternative queue.

Original: Denote the probability that in Node i at the end of 2( l − 1) τ periods, the number of old customers is less than N o and the number of new customers is less than N n by P i ( N o , N n ) , including the customers both using the server or in the waiting queue of node i .
Revised: Denote the probability that in Node i at the end of 2( l − 1) τ periods, the number of old customers is less than N o and the number of new customers is less than N n by P i ( N o , N n ) , including the customers both using the server or in the waiting queue of node i .

Original: Consider an alternative queue that starts accepting new customers under the new queueing system parameter from time 0 and denote the corresponding probability for having less than N o number of old customers and less than N n new customers at the end of 2( l − 1) τ periods by P 
Revised: Consider an alternative queue that starts accepting new customers under the new queueing system parameter from time 0 and denote the corresponding probability for having less than N o number of old customers and less than N n new customers at the end of 2( l − 1) τ periods by P (cid:48)

Original: i ( N o , N n ) .
Revised: i ( N o , N n ) .

Original: By the analysis in Proposition 3, we know there is a mismatch between P i ( N o , N n ) .
Revised: By the analysis in Proposition 3, we know there is a mismatch between P i ( N o , N n ) .

Original: Then we postpone the starting point of this alternative queue to a time stamp between 0 and 2( L − 1)
Revised: Then we postpone the starting point of this alternative queue to a time stamp between 0 and 2( L − 1)

Original: τ such that P ′ i ( N o , N n ) =
Revised: τ such that P (cid:48) i ( N o , N n ) =

Original: P i ( N o , N n ) .
Revised: P i ( N o , N n ) .

Original: Note that these timestamps always exist, which can be easily proved by the memoryless property of the exponential service time assumptions because we can always find this stamp backward from any customer state at time 2( l − 1) τ .
Revised: Note that these timestamps always exist, which can be easily proved by the memoryless property of the exponential service time assumptions because we can always ﬁnd this stamp backward from any customer state at time 2( l − 1) τ .

Original: Then we have the following two results.
Revised: Then we have the following two results.

Original: First, this alternative queue has the same evolution path as the actual queue.
Revised: First, this alternative queue has the same evolution path as the actual queue.

Original: Second, because this alternative queue starts accepting new customers earlier than time 2( l − 1) τ , and thus it will reach steady-state earlier than the virtual queue.
Revised: Second, because this alternative queue starts accepting new customers earlier than time 2( l − 1) τ , and thus it will reach steady-state earlier than the virtual queue.

Original: To conclude, the actual queue can reach the steady-state with a higher probability after the first 2 τ periods.
Revised: To conclude, the actual queue can reach the steady-state with a higher probability after the ﬁrst 2 τ periods.

Original: Therefore, conditional on the events that all nodes in layers with lower indices have reached the steady-state by the end of 2( l − 1) τ periods, we can compute that all nodes of layer l can reach steady-state in the next 2 τ period with a probability:[Equation]
Revised: Therefore, conditional on the events that all nodes in layers with lower indices have reached the steady-state by the end of 2( l − 1) τ periods, we can compute that all nodes of layer l can reach steady-state in the next 2 τ period with a probability:

Original: [Equation]
Revised: [Equation]

Original: With the results in (13) and (14), we are ready to bound the probability in (12):
Revised: With the results in (13) and (14), we are ready to bound the probability in (12):

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: We bound the relaxed regret during the Warm-up and Learning Phases separately.
Revised: We bound the relaxed regret during the Warm-up and Learning Phases separately.

Original: We further decompose the relaxed regret during the Learning Phase into two parts.
Revised: We further decompose the relaxed regret during the Learning Phase into two parts.

Original: The first part is the regret of suboptimality, which is induced by not offering the state-independent optimal price in certain time periods.
Revised: The ﬁrst part is the regret of suboptimality, which is induced by not offering the state-independent optimal price in certain time periods.

Original: We bound the loss due to suboptimality with the help of Proposition 2.
Revised: We bound the loss due to suboptimality with the help of Proposition 2.

Original: The second part is the regret of nonstationarity, which represents the revenue loss induced by changing offered prices.
Revised: The second part is the regret of nonstationarity, which represents the revenue loss induced by changing offered prices.

Original: We bound the loss due to nonstationarity with the help of Propositions 3 and 4.
Revised: We bound the loss due to nonstationarity with the help of Propositions 3 and 4.

Original: The structure of this proof is similar to that of Theorem 1 in Jia et al.
Revised: The structure of this proof is similar to that of Theorem 1 in Jia et al.

Original: (2020), where the authors consider a single reusable product setting.
Revised: (2020), where the authors consider a single reusable product setting.

Original: Regret of the Warm-up Phase.
Revised: Regret of the Warm-up Phase.

Original: The difference between Y LPWarm-up and Y RNRM-UCBWarm-up is at most linear on the length of the Warm-up Phase, which is O ( X log( J 1 / 4 T )) .
Revised: The difference between Y LPWarm-up and Y RNRM-UCBWarm-up is at most linear on the length of the Warm-up Phase, which is O ( X log( J 1 / 4 T )) .

Original: Therefore we can directly have
Revised: Therefore we can directly have

Original: Regret
Revised: Regret

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Regret of nonstationarity during the Learning Phase.
Revised: Regret of nonstationarity during the Learning Phase.

Original: We denote the offered price of batch m in
Revised: We denote the offered price of batch m in

Original: RNRM-UCB as x m .
Revised: RNRM-UCB as x m .

Original: Therefore, the expected revenue collected in batch m is
Revised: Therefore, the expected revenue collected in batch m is

Original: [Equation]
Revised: [Equation]

Original: where Y RNRM-UCB m, 0 ∼ 2 Lτ ( x m ) denotes the expected revenue collected in the first 2 Lτ periods and
Revised: where Y RNRM-UCB m, 0 ∼ 2 Lτ ( x m ) denotes the expected revenue collected in the ﬁrst 2 Lτ periods and

Original: Y RNRM-UCB m, 2 Lτ ∼
Revised: Y RNRM-UCB m, 2 Lτ ∼

Original: I m Lτ ( x m ) denotes the expected revenue collected from (2 Lτ + 1) th period to the end of batch m .
Revised: I m Lτ ( x m ) denotes the expected revenue collected from (2 Lτ + 1) th period to the end of batch m .

Original: We define event C 2 Lτ as that the network service system reaches the steady-state under the new price within the first 2 Lτ periods and define ¯ C 2 Lτ as the complementary event of C 2 Lτ .
Revised: We deﬁne event C 2 Lτ as that the network service system reaches the steady-state under the new price within the ﬁrst 2 Lτ periods and deﬁne ¯ C 2 Lτ as the complementary event of C 2 Lτ .

Original: By
Revised: By

Original: Propositions 3 and 4, we have P [ C 2 Lτ ] ≥ 1 − 4 I/T 2 .
Revised: Propositions 3 and 4, we have P [ C 2 Lτ ] ≥ 1 − 4 I/T 2 .

Original: Consequently, we define Y RNRM-UCB m, C 2 Lτ ( x m ) to be the expected value of Y RNRM-UCB m, 2 Lτ ∼
Revised: Consequently, we deﬁne Y RNRM-UCB m, C 2 Lτ ( x m ) to be the expected value of Y RNRM-UCB m, 2 Lτ ∼

Original: I m Lτ ( x m ) conditional on C 2 Lτ and Y RNRM-UCB m, ¯ C 2 Lτ ( x m ) similarly.
Revised: I m Lτ ( x m ) conditional on C 2 Lτ and Y RNRM-UCB m, ¯ C 2 Lτ ( x m ) similarly.

Original: As a result, we can express the expected revenue during batch m as:
Revised: As a result, we can express the expected revenue during batch m as

Original: [Equation]
Revised: [Equation]

Original: Regret of suboptimality during the Learning Phase.
Revised: Regret of suboptimality during the Learning Phase.

Original: Recall the upper and lower confidence bounds of the revenue rate defined in Definition 2.
Revised: Recall the upper and lower conﬁdence bounds of the revenue rate deﬁned in Deﬁnition 2.

Original: It is clear that
Revised: It is clear that

Original: [Equation]
Revised: [Equation]

Original: For batch m
Revised: For batch m

Original: [Equation]
Revised: [Equation]

Original: we can compute the difference in the revenue rates between the optimal price and selected price x m as
Revised: we can compute the difference in the revenue rates between the optimal price and selected price x m as

Original: [Equation]
Revised: [Equation]

Original: 17a
Revised: 17a

Revised: [Equation]

Revised: [Equation]

Revised: 17c

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds because U m ( x m ) ≥ U m (˜ x ) by the algorithm construction.
Revised: where the last inequality holds because U m ( x m ) ≥ U m (˜ x ) by the algorithm construction.

Original: Then the cumulative regret during the Learning Phase is
Revised: Then the cumulative regret during the Learning Phase is

Original: [Equation]
Revised: [Equation]

Original: By (17)
Revised: By (17)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We first analyze the middle term of (19).
Revised: We ﬁrst analyze the middle term of (19).

Original: Let M x denote the batches choosing price x and T x the number of periods choosing price x , i.e., T x = (cid:80) m
Revised: Let M x denote the batches choosing price x and T x the number of periods choosing price x , i.e., T x = (cid:80) m

Original: ∈M x
Revised: ∈M x

Original: I m Lτ .
Revised: I m Lτ .

Original: Then,
Revised: Then,

Revised: M

Original: [Equation]
Revised: [Equation]

Original:  where (20c) by Lemma 2, (20e) is derived by the number of the batches, and (20f) follows Jensen’s inequality.
Revised: 20f [Equation] where (20c) by Lemma 2, (20e) is derived by the number of the batches, and (20f) follows Jensen’s inequality.

Original: The number of batches is log 
Revised: The number of batches is log 2 (cid:16) 1 + T − 8 X log ( J 1 / 4 T ) 2 L log( T )

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: = O (cid:0) log( T ) (cid:1) .

Original: Below we analyze the first term of (19), and the same logic applies to the third term, of which we omit the details here.
Revised: Below we analyze the ﬁrst term of (19), and the same logic applies to the third term, of which we omit the details here.

Original: [Equation]
Revised: [Equation]

Revised: 21e

Original:  where (21e) is because U m (˜ x ) > 0 by Definition 2, and (21f) follows Proposition 2.
Revised: [Equation] ≤ O (cid:18) [Equation] (cid:19) , (21f) where (21e) is because U m (˜ x ) > 0 by Deﬁnition 2, and (21f) follows Proposition 2.

Original: Given that O (cid:0) n Mj ( x ) (cid:1) =
Revised: Given that O (cid:0) n Mj ( x ) (cid:1) =

Original: O ( T x ) , for any product j ∈ J on ( x ) , we have (cid:80) Mm =
Revised: O ( T x ) , for any product j ∈ J on ( x ) , we have (cid:80) Mm =

Original: 1 O ( Lτ ) =
Revised: 1 O ( Lτ ) =

Original: O (cid:0) L (log( T )) 3 (cid:1) .
Revised: O (cid:0) L (log( T )) 3 (cid:1) .

Original: Combining the three terms together,
Revised: Combining the three terms together,

Original: Regret
Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: Total Regret of RNRM-UCB algorithm.
Revised: Total Regret of RNRM-UCB algorithm.

Original: By adding up the relaxed regret during the Warm-up and Learning Phases, we have the regret of RNRM-UCB algorithm as (typically we always have
Revised: By adding up the relaxed regret during the Warm-up and Learning Phases, we have the regret of RNRM-UCB algorithm as (typically we always have

Revised: T (cid:29) X ):

Original: [Equation]
Revised: [Equation]

Original: Q.E
Revised: D

Original: [Equation]
Revised: [Equation]

Original: D.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: The analysis here is similar to that in the proof of Theorem 1.
Revised: The analysis here is similar to that in the proof of Theorem 1.

Original: We first bound the Bayesian regret of the Warm-up Phase and then for the Learning Phase.
Revised: We ﬁrst bound the Bayesian regret of the Warm-up Phase and then for the Learning Phase.

Original: Bayesian regret in the Warm-up Phase.
Revised: Bayesian regret in the Warm-up Phase.

Original: Similarly, we can bound the Bayesian regret in the
Revised: Similarly, we can bound the Bayesian regret in the

Original: Warm-up Phase as
Revised: Warm-up Phase as

Original: [Equation]
Revised: [Equation]

Original: Bayesian regret in the Learning Phase.
Revised: Bayesian regret in the Learning Phase.

Original: In batch m , the chosen price x m and the state-independent optimal price ˜ x are identically distributed conditional on F m , which indicates that E (cid:2)
Revised: In batch m , the chosen price x m and the state-independent optimal price ˜ x are identically distributed conditional on F m , which indicates that E (cid:2)

Original: U m ( x m ) −
Revised: U m ( x m ) −

Original: U m (˜ x ) |F m (cid:3) = 
(e [Equation]
Revised: U m (˜ x ) |F m (cid:3) =

Original: g., see the proof of Lemma 3.7 in Slivkins (2019)).
Revised: (e.g., see the proof of Lemma 3.7 in Slivkins (2019)).

Original: Thus, the Bayesian regret in batch m is:
Revised: Thus, the Bayesian regret in batch m is:

Original: BR m

Original: [Equation]
Revised: [Equation]

Original: 25a [Equation] where (25g) is derived following the analysis of nonstationarity in (16).
Revised: where (25g) is derived following the analysis of nonstationarity in (16).

Original: Derived from Proposition 2, for some γ > 0 , two properties hold (see, e
Revised: Derived from Proposition 2, for some γ > 0 , two properties hold (see, e

Original: [Equation]
Revised: [Equation]

Original: g., Slivkins 2019):
Revised: g., Slivkins 2019):

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Following these two properties of the stationary revenue rate in (26)
Revised: Following these two properties of the stationary revenue rate in (26)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: E

Original: [Equation]
Revised: [Equation]

Original: and similarly,
Revised: and similarly,

Original: [Equation]
Revised: [Equation]

Original: Summing over all batches m, ∀ m = 1 , . .
Revised: Summing over all batches m, ∀ m = 1 , . .

Original: , M , we have the cumulative relaxed Bayesian regret by the end of the Learning Phase as:
Revised: , M , we have the cumulative relaxed Bayesian regret by the end of the Learning Phase as:

Original: [Equation]
Revised: [Equation]

Original: where (29f) is by Lemma 2 and (29g) is by Jensen’s inequality.
Revised: where (29f) is by Lemma 2 and (29g) is by Jensen’s inequality.

Original: Regret of RNRM-TS algorithm.
Revised: Regret of RNRM-TS algorithm.

Original: By adding up the relaxed regret during the Warm-up and Learning
Revised: By adding up the relaxed regret during the Warm-up and Learning

Original: Phases, we have the regret of RNRM-TS algorithm as:
Revised: Phases, we have the regret of RNRM-TS algorithm as:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: For any price x ∈ X ,
Revised: For any price x ∈ X ,

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: Plugging the definition of Rad m ( x ) , ∀ m = 1 , . .
Revised: Plugging the deﬁnition of Rad m ( x ) , ∀ m = 1 , . .

Original: , M , one has:
Revised: , M , one has:

Original: [Equation]
Revised: [Equation]

Original: where and (32b) is by n mj ( x )
Revised: where and (32b) is by n mj ( x )

Original: =
Revised: =

Original: O ( I m Lτ ) , ∀ j ∈ J on ( x ) and (32e) is by Jensen’s inequality.
Revised: O ( I m Lτ ) , ∀ j ∈ J on ( x ) and (32e) is by Jensen’s inequality.

Original: Lemma 3.
Revised: Lemma 3.

Original: (Coupling probability of M/M/c queue starting from empty state, Proposition 4 in
Revised: (Coupling probability of M/M/c queue starting from empty state, Proposition 4 in

Original: Jia et al. 2020)
Revised: Jia et al. 2020)

Original: For τ = (log T ) 2 , if an M/M/c queue starts from empty state at time 0 , then the probability that the system reaches the steady-state within τ is bounded by 1 − 2 T 2 .
Revised: For τ = (log T ) 2 , if an M/M/c queue starts from empty state at time 0 , then the probability that the system reaches the steady-state within τ is bounded by 1 − 2 T 2 .

Original: Lemma 4.
Revised: Lemma 4.

Original: (Coupling probability of M/M/c queue when price changes, Proposition 5 in Jia et al.
Revised: (Coupling probability of M/M/c queue when price changes, Proposition 5 in Jia et al.

Original: For τ = (log T ) 2 , if an M/M/c queue starts from steady-state and changes system parameter at time 0 , then the probability that the system reaches the steady-state within 2 τ is bounded by 1 − 4 T 2 .
Revised: For τ = (log T ) 2 , if an M/M/c queue starts from steady-state and changes system parameter at time 0 , then the probability that the system reaches the steady-state within 2 τ is bounded by 1 − 4 T 2 .

Original: F Service Duration Dependent Payment Model
Revised: F Service Duration Dependent Payment Model

Original: We consider a new variant payment model where the customer pays the product’s per-unit-time price multiplied by its total service duration.
Revised: We consider a new variant payment model where the customer pays the product’s per-unit-time price multiplied by its total service duration.

Original: The firm then needs to decide the per-unit-time price for each product j ∈ J .
Revised: The ﬁrm then needs to decide the per-unit-time price for each product j ∈ J .

Original: Under this payment setting, we modify the definition of the steady-state revenue rate, derive a new concentration bound of historical observations and actual system parameters, and then correspondingly develop a new UCB for the revenue rate of each price.
Revised: Under this payment setting, we modify the deﬁnition of the steady-state revenue rate, derive a new concentration bound of historical observations and actual system parameters, and then correspondingly develop a new UCB for the revenue rate of each price.

Original: With those modified definitions, we present a new RNRM-UCB variant and show that it achieves the same regret bound.
Revised: With those modiﬁed deﬁnitions, we present a new RNRM-UCB variant and show that it achieves the same regret bound.

Original: Data collection.
Revised: Data collection.

Original: Similar to n j , ˆ d jk ( x ) , and ¯ d j ( x ) , we also collect service time observations over resources.
Revised: Similar to n j , ˆ d jk ( x ) , and ¯ d j ( x ) , we also collect service time observations over resources.

Original: We use n ri ( x ) to denote the number of customers who successfully consume resource i under price x .
Revised: We use n ri ( x ) to denote the number of customers who successfully consume resource i under price x .

Original: We use ˆ g ik ( x ) to denote the service time of customer k over resource i under the price x .
Revised: We use ˆ g ik ( x ) to denote the service time of customer k over resource i under the price x .

Original: We use ¯ g i ( x ) to denote the empirical mean of the service time and ¯ g i ( x ) =
Revised: We use ¯ g i ( x ) to denote the empirical mean of the service time and ¯ g i ( x ) =

Original: (cid:80) n ri ( x ) k
Revised: (cid:80) n ri ( x ) k

Original: =

Revised: =

Original: ˆ g ik ( x ) /n ri ( x ) , for
Revised: ˆ g ik ( x ) /n ri ( x ) , for

Original: i ∈
Revised: i ∈

Original: I .
Revised: I .

Original: We further assume that there is a known upper bound r max = max x,i,j α j /µ i , n min j ( x )
Revised: We further assume that there is a known upper bound r max = max x,i,j α j /µ i , n min j ( x )

Original: = min { n j ( x ) , min i : a ji =1 n si ( x ) } .
Revised: = min { n j ( x ) , min i : a ji =1 n si ( x ) } .

Original: Revenue rate.
Revised: Revenue rate.

Original: Under this duration-dependent payment model, we offer unit price x ∈ R J to products
Revised: Under this duration-dependent payment model, we offer unit price x ∈ R J to products

Original: J .
Revised: J .

Original: Customer k who requests to use product j pays the per-unit-time price multiplied by the total service duration, i.e., x j (cid:80) i : a ji
Revised: Customer k who requests to use product j pays the per-unit-time price multiplied by the total service duration, i.e., x j (cid:80) i : a ji

Original: =1 ˆ g ik ( x ) , to the firm.
Revised: =1 ˆ g ik ( x ) , to the ﬁrm.

Original: Different than the lump sum payment model, the steady-state revenue rate for this setting is r v ( x ) =
Revised: Different than the lump sum payment model, the steady-state revenue rate for this setting is r v ( x ) =

Original: (cid:80) j ∈J on ( x ) α j x j (cid:80) i : a ji =1 

Revised: (cid:80) j ∈J on ( x ) α j x j (cid:80) i : a ji =1

Original: /µ i ( x ) .
Revised: /µ i ( x ) .

Original: Proposition 5.
Revised: Proposition 5.

Original: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( L 14 J 14 T ) for j ∈ J on ( x ) and the number of service time observations for each resource n si ( x ) ≥ 8 log( L 1 4 J 1 4 T ) ∀ i ∈ { i ∈ I : (cid:80) j ∈J on ( x ) a ji ≥ 1 } , then we have 
Revised: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( L 14 J 14 T ) for j ∈ J on ( x ) and the number of service time observations for each resource n si ( x ) ≥ 8 log( L 1 4 J 1 4 T ) ∀ i ∈ { i ∈ I : (cid:80) j ∈J on ( x ) a ji ≥ 1 } , then we have [Table] [Equation] [Table]

Revised: [Equation]

Original: P
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where r max = max x ,i,j α j ( x ) /µ i ( x ) , and n min j ( x ) = min { n j ( x ) , min i : a ji =1 n si ( x ) } , and L is the number of layers.
Revised: where r max = max x ,i,j α j ( x ) /µ i ( x ) , and n min j ( x ) = min { n j ( x ) , min i : a ji =1 n si ( x ) } , and L is the number of layers.

Original: Proof of Proposition
Revised: Proof of Proposition

Original: [Table]
Revised: [Table]

Original: The above probability can be bounded by the product of the following probabilities from below.
Revised: The above probability can be bounded by the product of the following probabilities from below.

Original: Firstly we can bound that for i
Revised: Firstly we can bound that for i

Original: [Table]
Revised: [Table]

Original: j with a ji = 1 :
Revised: j with a ji = 1 :

Original: [Table]
Revised: [Table]

Original: The first and second probabilities in the above equation can be bounded with the same technique as that we used in (7) with Lemma 1 and thus we have
Revised: The ﬁrst and second probabilities in the above equation can be bounded with the same technique as that we used in (7) with Lemma 1 and thus we have

Original: [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: P 
Revised: P 

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have the above probability can be bounded by (cid:18)
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: Then, for product j ∈ J on ( x ) , we  can further bound
Revised: we have the above probability can be bounded by

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: i : a
Revised: for product j ∈ J on ( x )

Original: [Equation]
Revised: [Equation]

Original: P 
Revised: we can further bound

Original: [Equation]
Revised: [Equation]

Original: because (cid:80) i ∈I a ji ≤ L ∀ j ∈ J .
Revised: because (cid:80) i ∈I a ji ≤ L ∀ j ∈ J .

Original: Now we are ready to derive the concentration bound.
Revised: Now we are ready to derive the concentration bound.

Original: [Table]
Revised: [Table]

Original: According to the above proposition
Revised: According to the above proposition

Original: [Table]
Revised: [Table]

Original: we can define the confidence radius as follows.
Revised: we can deﬁne the conﬁdence radius as follows.

Original: (We use the superscript v to denote this duration-dependent payment model.)
Revised: (We use the superscript v to denote this duration-dependent payment model.)

Original: [Equation]
Revised: [Equation]

Original: Therefore, the upper and lower confidence bound of the revenue rate associated with each price is 
Revised: Therefore, the upper and lower conﬁdence bound of the revenue rate associated with each price is U vt

Original: [Equation]
Revised: [Equation]

Original: vt
Revised: x [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Algorithm design.
Revised: Algorithm design.

Original: The algorithmic framework remains unchanged except for two modifications.
Revised: The algorithmic framework remains unchanged except for two modiﬁcations.

Original: (i)
Revised: (i)

Original: In Step 6, the algorithm also requires the number of service time observations to be at least 8 log( L 14 J 14 T ) .
Revised: In Step 6, the algorithm also requires the number of service time observations to be at least 8 log( L 14 J 14 T ) .

Original: (ii)
Revised: (ii)

Original: In Steps 13 and 16, the algorithm uses the new UCB definition.
Revised: In Steps 13 and 16, the algorithm uses the new UCB deﬁnition. Algorithm

Original:  Offer x m in batch m , i.e., for I m Lτ periods.
Revised: = argmax x [Equation] 1 ( x ) . Offer x m in batch m , i.e., for I m Lτ periods.

Original: Record inter-arrival time ˆ d jk ( x m ) for arriving customers of product j ∈ J on ( x m ) .
Revised: Record inter-arrival time ˆ d jk ( x m ) for arriving customers of product j ∈ J on ( x m ) .

Original: Compute and update U vm ( x m ) .
Revised: Compute and update U vm ( x m ) .

Original: 17: end for
Revised: 17: end for

Original: Regret analysis.
Revised: Regret analysis.

Original: We change the constraints (1d) in the LP benchmark to
Revised: We change the constraints (1d) in the LP benchmark to

Original: [Equation]
Revised: [Equation]

Original: Under this duration-dependent payment setting, we can follow the same analysis technique to show that this LP model also gives a valid upper bound of the optimal revenue.
Revised: Under this duration-dependent payment setting, we can follow the same analysis technique to show that this LP model also gives a valid upper bound of the optimal revenue.

Original: The results of Propositions 1, 3, and 4 and Theorem 1 remain the same for this duration-dependent payment model.
Revised: The results of Propositions 1, 3, and 4 and Theorem 1 remain the same for this duration-dependent payment model.