Original: Online Learning and Pricing for Network Revenue Management with Reusable Resources
Revised: Online Learning and Pricing for Network Revenue Management with Reusable Resources

Original: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: We consider a price-based network revenue management problem with multipleproducts and multiple reusable resources.
Revised: We consider a price-based network revenue management problem with multiple products and multiple reusable resources.

Original: Each randomly arriving customer requests a product (service) that needs to occupy a sequence of reusable resources(servers).
Revised: Each randomly arriving customer requests a product (service) that needs to occupy a sequence of reusable resources (servers).

Original: We adopt an incomplete information setting where the ﬁrm does notknow the price-demand function for each product and the goal is to dynamically setprices of all products to maximize the total expected revenue of serving customers.
Revised: We adopt an incomplete information setting where the firm does not know the price-demand function for each product and the goal is to dynamically set prices of all products to maximize the total expected revenue of serving customers.

Original: We propose novel batched bandit learning algorithms for ﬁnding near-optimalpricing policies, and show that they admit a near-optimal cumulative regret boundof ˜ O
Revised: We propose novel batched bandit learning algorithms for finding near-optimal pricing policies, and show that they admit a near-optimal cumulative regret bound of ˜ O

Original: ( J √ XT ) , where J , X , and T are the numbers of products, candidate prices,and service periods, respectively.
Revised: ( J √ XT ) , where J , X , and T are the numbers of products, candidate prices, and service periods, respectively.

Original: As part of our regret analysis, we develop theﬁrst ﬁnite-time mixing time analysis of an open network queueing system (i.e., the  Jackson Network), which could be of independent interest.
Revised: As part of our regret analysis, we develop the first finite-time mixing time analysis of an open network queueing system (i.e., the celebrated Jackson Network), which could be of independent interest.

Original: Our numerical studiesshow very promising results of the proposed approaches.
Revised: Our numerical studies show that the proposed approaches perform consistently well.

Original:  Introduction
Revised:  Introduction

Original: Network revenue management (NRM) (Gallego and Van Ryzin 1997) has attained remarkable resultsin a wide range of applications including airline management, electricity pricing, retailing, healthcare,and leisure (see, e.g., Williamson 1992, Talluri and Van Ryzin 2006, Özer et al. 2012).
Revised: Network revenue management (NRM) (Gallego and Van Ryzin 1997) has attained remarkable results in a wide range of applications including airline management, electricity pricing, retailing, healthcare, and leisure (see, e g., Williamson 1992, Talluri and Van Ryzin 2006, Özer et al. 2012).

Original: The bulkof existing literature focuses primarily on canonical settings with perishable resources in whichthe product once sold is permanently removed from inventory (Besbes and Zeevi 2012, Ferreiraet al.
Revised: The bulk of existing literature focuses primarily on canonical settings with perishable resources in which the product once sold is permanently removed from inventory (Besbes and Zeevi 2012, Ferreira et al.

Original: 2018, Chen and Shi 2019).
Revised: 2018, Chen and Shi 2019).

Original: Little progress has been made for NRM with reusable resources,mainly due to the complexity of analyzing the underlying stochastic systems (typically manifestedas stochastic queueing networks).
Revised: Little progress has been made for NRM with reusable resources, mainly due to the complexity of analyzing the underlying stochastic systems (typically manifested as stochastic queueing networks).

Original: In this paper, we develop easy-to-implement online learning andpricing policies for NRM with reusable resources shared by multiple products and analyze theirtheoretical and empirical performances.
Revised: In this paper, we develop easy-to-implement online learning and pricing policies for NRM with reusable resources shared by multiple products and analyze their theoretical and empirical performances.

Original: 1.1 Problem description
Revised: 1.1 Problem Description

Original: A service-providing ﬁrm is endowed with multiple types of reusable resources, each having a ﬁxedcapacity.
Revised: A service-providing firm is endowed with multiple types of reusable resources, each having a fixed capacity.

Original: The ﬁrm sells multiple types of products (services), each requiring the use of a sequenceof different types of reusable resources (servers).
Revised: The firm sells multiple types of products (services), each requiring the use of a sequence of different types of reusable resources (servers).

Original: (See, e.g., Figure 2.)
Revised: (See, e.g., Figure 2.)

Original: Customers randomly arriveaccording to a price-dependent multivariate Poisson process to request products.
Revised: Customers randomly arrive according to a price-dependent multivariate Poisson process to request products.

Original: Upon requesting oneparticular product, a customer enters the system by going through a sequence of product-speciﬁedreusable resources one by one.
Revised: Upon requesting one particular product, a customer enters the system by going through a sequence of product-specified reusable resources one by one.

Original: During the service process, if there is no available next type ofresource upon ﬁnishing using the previous type in the sequence, she will join the queue in front ofthis next type of resource and wait until the next available unit.
Revised: During the service process, if there is no available next type of resource upon finishing using the previous type in the sequence, she will join the queue in front of this next type of resource and wait until the next available unit.

Original: Both the mean demand rate of eachproduct and the average usage time of each resource depend on the posted price vector.
Revised: Both the mean demand rate of each product and the average usage time of each resource depend on the posted price vector.

Original: However,such dependence is unknown to the ﬁrm a priori .
Revised: However, such dependence is unknown to the firm a priori .

Original: We note that under a ﬁxed price vector, the servicesystem can be reduced to the celebrated Jackson Network (Jackson 1963, 1957).
Revised: We note that under a fixed price vector, the service system can be reduced to the celebrated Jackson Network (Jackson 1963, 1957).

Original: The main objectiveof this paper is to design a “learning-while-doing” algorithm to dynamically post prices of differentproducts, in order to maximize the expected total revenue collected over a ﬁnite time horizon
Revised: The main objective of this paper is to design a “learning-while-doing” algorithm to dynamically post prices of different products, in order to maximize the expected total revenue collected over a finite time horizon.

Original: It is evident that NRM with reusable resources ﬁnds many important real-world applications, includingcloud computing, hospitality industry, project management, and leisure (Lei and Jasin 2020).
Revised: It is evident that NRM with reusable resources finds many important real-world applications, including cloud computing, hospitality industry, project management, and leisure (Lei and Jasin 2020).

Original: Forexample, ﬁrms such as Amazon and Microsoft offer diverse cloud computing services, each requiringa different set of computational resources (virtual machines) for a certain duration of time to ﬁnishtheir computational jobs (see, e.g., Kaewpuang et al. 2013, Püschel et al. 2015).
Revised: For example, firms such as Amazon and Microsoft offer diverse cloud computing services, each requiring a different set of computational resources (virtual machines) for a certain duration of time to finish their computational jobs (see, e.g., Kaewpuang et al. 2013, Püschel et al. 2015).

Original: 1.2 Main results and contributions
Revised: 1.2 Main Results and Contributions

Original: We propose new batched bandit algorithms for NRM with reusable resources (RNRM).
Revised: We propose new batched bandit algorithms for NRM with reusable resources (RNRM).

Original: The performance measure is regret, which is the revenue difference between our learning algorithms and aclairvoyant optimal pricing policy under full information.
Revised: The performance measure is regret, which is the revenue difference between our learning algorithms and a clairvoyant optimal pricing policy under full information.

Original: We use J, X, T to denote the numbers ofproducts, candidate prices, and service periods, respectively.
Revised: We use J, X, T to denote the numbers of products, candidate prices, and service periods, respectively.

Original: We prove in Theorems 1 and 2 that thecumulative regret is ˜ O
Revised: We prove in Theorems 1 and 2 that the cumulative regret is ˜ O

Original: ( J √ XT ) , which matches the lower bound along the dimensions of X and Tup to a logarithmic factor.
Revised: ( J √ XT ) , which matches the lower bound along the dimensions of X and T up to a logarithmic factor.

Original: It remains open whether one could improve the dependence on J sinceall prior studies  (e.g., Miao and Wang 2021) show polynomial dependence on J at best (albeit for anon-reusable setting).
Revised: It remains open whether one could improve the dependence on J . Note that all prior studies on non-reusable settings (e.g., Miao and Wang 2021) show polynomial dependence on J at best.

Original: We test diverse instances to show numerical performance of our approaches.
Revised: We test diverse instances to show numerical performance of our approaches.

Original: The main contribution of our work is three-fold:(Modeling)
Revised: The main contribution of our work is three-fold: (Modeling.)

Original: We are the ﬁrst to consider the joint learning and optimization problem in NRM withreusable resources.
Revised: To the best of our knowledge, we are the first to consider the joint learning and optimization problem in NRM with reusable resources.

Original: Prior literature on reusable resources only focused on single-product settings(see Jia et al.
Revised: Prior literature on reusable resources only focused on single-product settings (see Jia et al.

Original: (2020, 2022) and references therein).
Revised: (2020, 2022) and references therein).

Original: It is considerably more challenging to model andanalyze the network setting where multiple products share multiple resources (with ﬁnite capacities),mainly due to the sequential use of resources (to ﬁnish up a service) and the evolving nature ofavailable capacities in real-time.
Revised: It is considerably more challenging to model and analyze the network setting where multiple products share multiple resources (with finite capacities), mainly due to the sequential use of resources (to finish up a service) and the evolving nature of available capacities in real-time.

Original: We establish a connection between our model and the celebrated Jackson Network (Jackson 1963, 1957).(Algorithm)
Revised: We establish a connection between our model and the celebrated Jackson Network (Jackson 1963, 1957). (Algorithm.)

Original: We ﬁrst develop a linear programming (LP) upper bound on the optimal objective valuegiven full information.
Revised: We first develop a linear programming (LP) upper bound on the optimal objective value given full information.

Original: The LP model provides a static pricing control policy.
Revised: The LP model provides a static pricing control policy.

Original: Therefore, we cancast our RNRM as a multi-armed bandits (MAB) problem where we select among candidate prices(analogs of arms) to minimize the cumulative regret.
Revised: Therefore, we can cast our RNRM as a multi-armed bandits (MAB) problem where we select among candidate prices (analogs of arms) to minimize the cumulative regret.

Original: There are two key challenges.
Revised: There are two key challenges.

Original: The ﬁrst liesin the fact that whenever the system posts a new price, the arrival processes change immediatelywhile the system still contains “old” customers from the previously posted price.
Revised: The first lies in the fact that whenever the system posts a new price, the arrival processes change immediately while the system still contains “old” customers from the previously posted price.

Original: Therefore, itis difﬁcult to analyze the transient system performance and keep track of the cumulative revenuein a dynamic pricing scheme.
Revised: Therefore, it is difficult to analyze the transient system performance and keep track of the cumulative revenue in a dynamic pricing scheme.

Original: We resolve this challenge by proposing batched algorithms.
Revised: We resolve this challenge by proposing batched algorithms.

Original: Ouralgorithms separate the time horizon into successive batches and select a price for each batch.
Revised: Our algorithms separate the time horizon into successive batches and select a price for each batch.

Original: Wefurther divide each batch into two intervals.
Revised: We further divide each batch into two intervals.

Original: In the ﬁrst interval, the system completes serving theprevious customers and reaches the steady-state under the new price.
Revised: In the first interval, the system completes serving the previous customers and reaches the steady-state under the new price.

Original: In the second interval, thesystem maintains the steady-state (exploiting the new price).
Revised: In the second interval, the system maintains the steady-state (exploiting the new price).

Original: This batched framework not only allowsfor system stabilization (under new prices) but also allows for infrequent O (log T ) price changes,which is more implementable in practical settings (see Cheung et al. 2017).
Revised: This batched framework not only allows for system stabilization (under new prices) but also allows for infrequent O (log T ) price changes, which is more implementable in practical settings (see Cheung et al. 2017).

Original: The second challenge ismore mathematical as we need to ensure that the regret does not scale exponentially in the number ofproducts or resources (given the combinatorial nature of the underlying graph).(Analysis) To cope with the ﬁrst challenge mentioned above, we bound the loss of nonstationarity bybounding the length of the ﬁrst interval of each batch.
Revised: The second challenge is more mathematical because we must ensure that the regret does not scale exponentially in the number of products or resources (given the combinatorial nature of the underlying graph). (Analysis.) To cope with the first challenge mentioned above, we bound the loss of nonstationarity by bounding the length of the first interval of each batch.

Original: We prove that the system can reach the newsteady-state reasonably fast (see mixing times of Jackson Network in Propositions 3 and 4).
Revised: We prove that the system can reach the new steady-state reasonably fast (see mixing times of Jackson Network in Propositions 3 and 4).

Original: To thebest of our knowledge, this is the ﬁrst result giving a ﬁnite-time bound on mixing times of Jackson
Revised: To the best of our knowledge, this is the first result giving a finite-time bound on mixing times of Jackson

Original: Networks.
Revised: Networks.

Original: To cope with the second challenge, we decompose the resulting Jackson Network intotopological layers and present a novel nested coupling argument (based on the layers).
Revised: To cope with the second challenge, we decompose the resulting Jackson Network into topological layers and present a novel nested coupling argument (based on the layers).

Original: The couplingargument of Jia et al.
Revised: The coupling argument of Jia et al.

Original: (2020) in the single-resource single-product setting cannot be directly applied toour network setting.
Revised: (2020) in the single-resource single-product setting cannot be directly applied to our network setting.

Original: First, in the network of M/M/c queues, the arrival and waiting processes dependon previously visited nodes, which is signiﬁcantly more complex to analyze upon price changes.
Revised: First, in the network of M/M/c queues, the arrival and waiting processes depend on previously visited resources, which is significantly more complex to analyze upon price changes.

Original: To resolve this challenge, we have to carefully construct virtual and alternative queues to explicitlyaccount for such dependency.
Revised: To resolve this challenge, we have to carefully construct virtual and alternative queues to explicitly account for such dependency.

Original: On a high level, the difﬁculty lies in that the original queues have atime-varying rate from an initial rate to the stable rate upon any price change.
Revised: On a high level, the difficulty lies in that the original queues have a time-varying rate from an initial rate to a stable rate upon any price change.

Original: One could constructvirtual queues that could clear all previously arrived customers at a speciﬁc time  τ ,but it is far toostrong for the coupling analysis. Instead, we  construct alternative queues that start from zero initialrate and ramp up directly to the stable rate at a time stamp before τ so that the customer distributionmatches the original queues in period τ . 
Revised: Given a carefully chosen time period τ , we first construct alternative queues that start from zero initial rate and ramp up directly to the stable rate at a timestamp before τ so that the customer distribution matches the original queues in period τ . We then construct virtual queues that could clear all previously arrived customers at a specific time τ , and show that their mixing times provide an upper bound on the alternative queues and consequently the original queues.

Original: Second, even ignoring the subtleties of time-varying rates(upon price changes), if one applied the coupling argument of Jia et al.
Revised: Second, even ignoring the subtleties of time-varying rates (upon price changes), if one applies the coupling argument of Jia et al.

Original: (2020) in a brute-force way to every node in the network, the coupling probability would result in a undesirable exponential scalingin the number of products.
Revised: (2020) in a brute-force way to every resource in the network, the coupling probability would result in an undesirable exponential scaling in the number of products.

Original: We decompose the network into topological layers so that the resultingcoupling probability only scales linearly in the number of products.
Revised: We decompose the network into topological layers so that the resulting coupling probability only scales linearly in the number of products.

Original: 1.3 Literature review
Revised: 1.3 Literature Review

Original: NRM with Reusable Resources and Demand Learning.
Revised: NRM with Reusable Resources and Demand Learning.

Original: Network, demand learning, and reusableresources are three key elements of this study.
Revised: Network, demand learning, and reusable resources are three key elements of this study.

Original: We only review related literature that shares at leasttwo of the three elements.
Revised: We only review related literature that shares at least two of the three elements.

Original: NRM with demand learning.
Revised: NRM with demand learning.

Original: Due to  easy accessibility of real-time sales data, there have been recentstudies on developing learning and pricing algorithms with probably near-optimal regret bound under NRMsettings (see, e.g., Besbes and Zeevi 2012, Chen et al.
Revised: Due to the easy accessibility of real-time sales data, there have been recent studies on developing learning and pricing algorithms with probably near-optimal regret bound for NRM (see, e.g., Besbes and Zeevi 2012, Chen et al.

Original: 2019, Chen and Shi 2019, Badanidiyuruet al.
Revised: 2019, Chen and Shi 2019, Badanidiyuru et al.

Original: 2013, Agrawal and Devanur 2016, Ferreira et al. 2018).
Revised: 2013, Agrawal and Devanur 2016, Ferreira et al. 2018).

Original: All the aforementioned study considersthe canonical settings (ﬁrst deﬁned by Gallego and Van Ryzin 1997), i.e., perishable (non-reusable)resources.
Revised: All the aforementioned study considers the canonical settings (first defined by Gallego and Van Ryzin 1997), i.e., non-reusable resources.

Original: NRM with reusable resources.
Revised: NRM with reusable resources.

Original: The only existing studies on RNRM are Owen and Simchi-Levi (2018),
Revised: The only existing studies on RNRM are Owen and Simchi-Levi (2018),

Original: Rusmevichientong et al.
Revised: Rusmevichientong et al.

Original: and Lei and Jasin (2020).
Revised: and Lei and Jasin (2020).

Original: Lei and Jasin (2020) considered a deterministic service time assumption, and developed an asymptotically near-optimal real-time pricing controlalgorithm.
Revised: Lei and Jasin (2020) considered a deterministic service time assumption, and developed an asymptotically near-optimal real-time pricing control algorithm.

Original: Owen and Simchi-Levi (2018) considered a joint assortment and pricing problem under amulti-resource setting and provided constant factor performance guarantees.
Revised: Owen and Simchi-Levi (2018) considered a joint assortment and pricing problem under a multi-resource setting and provided constant factor performance guarantees.

Original: Rusmevichientong et al.(2020) considered a dynamic assortment and pricing problem with reusable products and proposed aprovably near-optimal policy based on approximate dynamic programming.
Revised: Rusmevichientong et al. (2020) considered a dynamic assortment and pricing problem with reusable products and proposed a provably near-optimal policy based on approximate dynamic programming.

Original: Demand learning with reusable resources.
Revised: Demand learning with reusable resources.

Original: The aforementioned literature on reusable resourcesall assumed known distributional information of the underlying model.
Revised: The aforementioned literature on reusable resources all assumed known distributional information of the underlying model.

Original: There has been little workconsidering demand learning for service systems with reusable resources due to complex underlyingsystem dynamics.
Revised: There has been little work considering demand learning for service systems with reusable resources due to complex underlying system dynamics.

Original: The only studies that considered demand learning for reusable resources are Chenet al.
Revised: The only studies that considered demand learning for reusable resources are Chen et al.

Original: (2020), Jia et al.
Revised: (2020), Jia et al.

Original: (2020, 2022), and all considered only the single resource setting.
Revised: (2020, 2022), and all considered only the single resource setting.

Original: Chen et al.(2020) proposed an online learning framework for a dynamic pricing and capacity sizing problem ina GI/GI/1 queue and Jia et al.
Revised: Chen et al. (2020) proposed an online learning framework for a dynamic pricing and capacity sizing problem in a GI/GI/1 queue and Jia et al.

Original: (2020, 2022) developed MAB based algorithms for optimal dynamicpricing in an M/M/c queue for the single product single resource setting.
Revised: (2020, 2022) developed MAB-based algorithms for optimal dynamic pricing in an M/M/c queue for the single product single resource setting.

Original: To the best of our knowledge, we are the ﬁrst to thoroughly study all three elements.
Revised: To the best of our knowledge, we are the first to thoroughly study all three elements.

Original: Transient Analysis of Jackson Networks.
Revised: Transient Analysis of Jackson Networks.

Original: Jackson Network has received much attention amongthe queueing networks because its steady-state distribution is particularly simple to compute asa product-form solution (Jackson 1963, 1957).
Revised: Jackson Network has received much attention among the queueing networks because its steady-state distribution is particularly simple to compute as a product-form solution (Jackson 1963, 1957).

Original: The queue lengths across the nodes in Jackson
Revised: The queue lengths across the nodes in Jackson

Original: Networks, i.e., a network of M/M/c queues, during the transient state can be bounded via techniquesof stochastic dominance (see Massey 1986, 1984, 1987).
Revised: Networks, i.e., a network of M/M/c queues, during the transient state can be bounded via techniques of stochastic dominance (see Massey 1986, 1984, 1987).

Original: However, how the network reacts to systemchanges remains an open question (see, e.g., Mohamed et al. 2003, 2005).
Revised: However, how the network reacts to system changes remains an open question (see, e.g., Mohamed et al. 2003, 2005).

Original: Kelton and Law (1985)pointed out that a queueing system needs to run “long-enough” for removing the effects brought bythe initial starting point and then reach an ensured “steady-state” portion of the run.
Revised: Kelton and Law (1985) pointed out that a queueing system needs to run “long-enough” for removing the effects brought by the initial starting point and then reach an ensured “steady-state” portion of the run.

Original: The only workthat explicitly developed a ﬁnite-time bound on mixing times of a queueing system is a recent paperby Jia et al.
Revised: The only work that explicitly developed a finite-time bound on mixing times of a queueing system is a recent paper by Jia et al.

Original: In contrast, our paper gives the ﬁrst ﬁnite-time high probability bound on mixingtimes of Jackson Networks (consisting of a network  of M/M/c queues) upon action changes.
Revised: In contrast, our paper gives the first finite-time high probability bound on mixing times of Jackson Networks (a network consisting of M/M/c queues) upon action changes.

Original: Batched Bandit Approaches.
Revised: Batched Bandit Approaches.

Original: UCB2 (Auer et al. 2002) and Improved-UCB (Auer and Ortner 2010)were implicitly M -batch policies with M = Θ(log( T )) .
Revised: UCB2 (Auer et al. 2002) and Improved-UCB (Auer and Ortner 2010) were implicitly M -batch policies with M = Θ(log( T )) .

Original: Later, Perchet et al.
Revised: Later, Perchet et al.

Original: studied batched
Revised: studied batched

Original: MAB in the two-armed case under a static batch design and Gao et al.
Revised: MAB in the two-armed case under a static batch design and Gao et al.

Original: (2019) extended the algorithmsto the general multiple-armed case and adaptively determined batch size.
Revised: (2019) extended the algorithms to the general multiple-armed case and adaptively determined batch size.

Original: There is also rich literaturefocusing on batched MAB problems with switching cost.
Revised: There is also rich literature focusing on batched MAB problems with switching cost.

Original: For example, Cesa-Bianchi et al.
Revised: For example, Cesa-Bianchi et al.

Original: (2013)considered uniform switching cost in the objective and Simchi-Levi and Xu (2019) treated the ﬁniteswitching cost budget as a hard constraint.
Revised: (2013) considered uniform switching cost in the objective and Simchi-Levi and Xu (2019) treated the finite switching cost budget as a hard constraint.

Original: In contrast, we construct the batched action framework tocontrol the loss of the nonstationarity (resulted from the transient time after each action change).
Revised: In contrast, we construct the batched action framework to control the loss of the nonstationarity (resulted from the transient time after each action change).

Original:  Problem formulation
Revised:  Problem Formulation

Original: Consider a ﬁrm that manages I types of reusable resources to provide J types of products over aﬁnite time horizon T .
Revised: Consider a firm that manages I types of reusable resources to provide J types of products over a finite time horizon T .

Original: Denote the capacity of server type i ∈ I as c i .
Revised: Denote the capacity of resource type i ∈ I as c i .

Original: At the beginning of each period, the ﬁrm posts a price x = { x j : j ∈ J } for all products from a given price candidate set X ⊂ R |J| .
Revised: At the beginning of each period, the firm posts a price x = { x j : j ∈ J } for all products from a given price candidate set X ⊂ R |J| .

Original: Let I = |I| , J = |J | , and X = |X| .
Revised: Let I = |I| , J = |J | , and X = |X| .

Original: Each arriving customer requests one product j ∈ J , which isprocessed through an ordered sequence of resources.
Revised: Each arriving customer requests one product j ∈ J , which is processed through an ordered sequence of resources.

Original: Customers arrive following a price-dependentmultivariate Poisson process with a total rate α ( x ) , which is unknown to the decision maker a priori .
Revised: Customers arrive following a price-dependent multivariate Poisson process with a total rate α ( x ) , which is unknown to the decision maker a priori .

Original: The service time of resource i follows an exponential distribution regardless of which product typeit belongs to, which also depends on posted price x .
Revised: The service time of resource i follows an exponential distribution regardless of which product type it belongs to, which also depends on posted price x .

Original: If there is not enough capacity for the currentresource, the customer will join the queue of this resource until being served.
Revised: If there is not enough capacity for the current resource, the customer will join the queue of this resource until being served.

Original: The resources arereleased to process the next customers upon ﬁnishing serving the current ones.
Revised: The resources are released to process the next customers upon finishing serving the current ones.

Revised: We consider two popular payment models, the lump sum payment model (presented in the main manuscript) and the service duration-dependent payment model (presented in Appendix F).

Revised: • ( Lump sum payment model.)

Original: Customers pay a lumpsum x j to the ﬁrm for product j .
Revised: A customer of product j ∈ J pays a lump sum x j to the firm.

Revised: More elaborately, if a customer chooses product j , then the customer will pay x j to the firm regardless of how long the services over resources are.

Original: This is referred to as  the lump sum payment model.
Revised: In our illustrative example, each product can be quoted a fixed price (i.e., not service duration dependent), which could be seen as a preset “cloud solution package”. The customers may prefer or benefit from this fixed price (contract) from their own accounting and cash flow management. Under this model, even though the customers are not sensitive to service durations, the cloud platform is nevertheless sensitive to service durations since they affect how many products/services can be completed within a finite horizon. • ( Service duration dependent payment model.)

Original: We also presentan alternative service duration dependent payment model where the customer  pays the amount that isequal to theproduct’s per-unit-time price multiplied by its total service  duration  in Appendix F.
Revised: A customer of product j ∈ J pays the amount that is equal to the per-unit-time x j multiplied by the total service duration. This payment scheme is duration dependent, which is also ubiquitous, especially in cloud computing applications (e g., how much time is spent using EC2 in our illustrative example). In the interest of space, the model and analytical details are presented in Appendix F.

Original: Consider a resource-consumption matrix A ji ∈ R
Revised: Consider a resource-consumption matrix A ji ∈ R

Original: J × I , where element a ji = 1 indicates that servicetype j requires to be processed on server type i .
Revised: J × I , where element a ji = 1 indicates that service type j requires to be processed on server type i .

Original: We use an I -dimensional vector, y ∈ { 0 , 1 , . . .
Revised: We use an I -dimensional vector, y ∈ { 0 , 1 , . . .

Original: , ∞} Ito denote the number of customers across different types of resources.
Revised: , ∞} I to denote the number of customers across different types of resources.

Original: The ﬁrm aims at ﬁnding apricing policy as π : { ( y , t ) : y ∈ { 0 , 1 , . . .
Revised: The firm aims at finding a pricing policy as π : { ( y , t ) : y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T } → X , where the ﬁrm selects priceπ ( y , t ) ∈
Revised: , T } → X , where the firm selects price π

Original:  X for period t when there are y customers in the system at the beginning of period t , tomaximize the expected cumulative revenue Y π .
Revised: ( y , t ) ∈ X for period t when there are y customers in the system at the beginning of period t , to maximize the expected cumulative revenue Y π .

Original: Assumption 1.
Revised: Assumption 1.

Original: (“Turn-off” price.)
Revised: (“Turn-off” price.)

Original: For each product j ∈ J , there exists a “turn-off” price ¯ p j < ∞such that, if the posted price is larger than this turn-off price, then the demand of this product is zero.
Revised: For each product j ∈ J , there exists a “turn-off” price ¯ p j < ∞ such that, if the posted price is larger than this turn-off price, then the demand for this product is zero.

Original: Assumption 1 is widely assumed in NRM studies (see, e.g., Lei and Jasin 2020).
Revised: Assumption 1 is widely assumed in NRM studies (see, e.g., Lei and Jasin 2020).

Original: This turn-off pricecan be used to effectively turn off  demand of undesired products whenever needed.
Revised: This turn-off price can be used to effectively turn off the demand for undesired products whenever needed.

Original: Therefore, foreach candidate price x ∈ X , we can further decompose the products into turn-on product set J on ( x )and a turn-off product set J off ( x ) , where the latter can be empty.
Revised: Therefore, for each candidate price x ∈ X , we can further decompose the products into turn-on product set J on ( x ) and a turn-off product set J off ( x ) , where the latter can be empty.

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists an order of server types I such that the processing sequence of serversi 1 , i 2 , . .
Revised: There exists an order of server types I such that the processing sequence of servers i 1 , i 2 , . .

Original: , i n always has i 1 <
Revised: , i n always has i 1 <

Original: i 2 < . .
Revised: i 2 < . .

Original: < i n for any product j ∈ J .
Revised: < i n for any product j ∈ J .

Original: One can understand Assumption 2 via viewing a network.
Revised: One can understand Assumption 2 via viewing a network.

Original: Consider a graph with resources as nodesand draw a directed arc from node i to i (cid:48) if any product requires successive processing on resourcesiand i (cid:48) .
Revised: Consider a graph with resources as nodes and draw a directed arc from node i to i ′ if any product requires successive processing on resources i and i ′ .

Original: If the system satisﬁes Assumption 2, then this graph is acyclic, i.e., no cycles, and there existsat least one topological order such that all arcs are from  smaller index to  larger index.
Revised: If the system satisfies Assumption 2, then this graph is acyclic, i.e., no cycles, and there exists at least one topological order such that all arcs are from a smaller index to a larger index.

Original: Without lossof generality, we consider one such topological order when referring to resource i in the rest part ofthis paper.
Revised: Without loss of generality, we consider one such topological order when referring to resource i in the rest part of this paper.

Original: To better utilize this network representation, we also add a source node s to representcustomers entering the system and a sink node t to represent customers exiting the system.
Revised: To better utilize this network representation, we also add a source node s to represent customers entering the system and a sink node t to represent customers exiting the system.

Original: Thus, theconsumption matrix can be extended to A ji ∈ R J ×
Revised: Thus, the consumption matrix can be extended to A ji ∈ R J ×

Original: ( I +2) and a js = a jt = 1 for all products.
Revised: ( I +2) and a js = a jt = 1 for all products.

Original: Assumption 3.
Revised: Assumption 3.

Original: (Product-form demand.)
Revised: (Product-form demand.)

Original: Any given price vector x induces a (Markovian) routing matrix P ( x ) where its ( i, i (cid:48) )
Revised: Any given price vector x induces a (Markovian) routing matrix P ( x ) where its ( i, i ′ )

Original: th entry p i,i (cid:48) represents the product demand portion who will consume resource i (cid:48) next out of those who consume resource i .
Revised: th entry p i,i ′ represents the product demand portion who will consume resource i ′ next out of those who consume resource i .

Original: Then the demand rate for any product j ∈ J , represented by a sequence of resources s, i 1 , i 2 , . .
Revised: Then the demand rate for any product j ∈ J , represented by a sequence of resources s, i 1 , i 2 , . .

Original: , i n , t , is given by
Revised: , i n , t , is given by

Original: [Equation]
Revised: [Equation]

Original: which means that the demand rate for product j is equal to the total demand rate α ( x ) times asequence of routing probabilities of P ( x ) , under posted price vector x .
Revised: which means that the demand rate for product j is equal to the total demand rate α ( x ) times a sequence of routing probabilities of P ( x ) , under posted price vector x .

Original: We use Figure 2 in the Appendix to demonstrate a concrete example.
Revised: We use Figure 2 in the Appendix to demonstrate a concrete example.

Original: With Assumptions 2 and 3, theservice system under any posted price x ∈ X can be reduced to a Jackson Network consisting with Inumber of M/M/c queues (when the system only contains customers arriving under this price, i.e.,after ﬁnishing serving other customers arriving under the previously posted prices).
Revised: With Assumptions 2 and 3, the service system under any posted price x ∈ X can be reduced to a Jackson Network consisting with I number of M/M/c queues (when the system only contains customers arriving under this price, i.e., after finishing serving other customers arriving under the previously posted prices).

Original: The main idea isthat there exists a mapping x (cid:55)→ α ( x ) , P ( x ) that gives rise to product demand pattern.
Revised: The main idea is that there exists a mapping x (cid:55)→ α ( x ) , P ( x ) that gives rise to product demand pattern.

Original: Note that ourassumptions are mild in the sense that this mapping can be nonparametric.
Revised: Note that our assumptions are mild in the sense that this mapping can be nonparametric.

Original: The decision maker doesnot know this underlying mapping and needs to learn it while maximizing revenue on the ﬂy.
Revised: The decision maker does not know this underlying mapping and needs to learn it while maximizing revenue on the fly.

Original: Given any pricevector x , (i) on one hand, this price affects the totalfuture demand α ( x ) entering thesystem.
Revised: Given any price x , (i) on one hand, this price affects the total demand α ( x ) entering the system.

Original: (ii)
Revised: (ii)

Original: On the other hand, this price affects the internal competition betweenthe products, whichis reﬂected in the server routing matrix P ( x ) .
Revised: On the other hand, this price affects the internal competition between products, which is reflected in the server routing matrix P ( x ) .

Original: As a result, α ( x ) and P ( x ) deﬁne a Jackson Networkwell and the demand rate α j ( x ) of each product type is also determined (use vector (cid:126)α x ∈ R J todenote demand of products).
Revised: As a result, α ( x ) and P ( x ) define a Jackson Network well and the demand rate α j ( x ) of each product type is also determined (use vector ⃗α x ∈ R J to denote demand of products).

Original: We use λ i ( x )  to denote the demand rate of resource type i .
Revised: We use λ i ( x ) and µ i ( x ) to denote the demand and service rates of resource type i .

Original: Assumption 4.
Revised: Assumption 4.

Original: (Stability.)
Revised: (Stability.)

Original: The queueing system of each resource is stable under any price, i.e.,c i µ i ( x ) >
Revised: The queueing system of each resource is stable under any price, i.e., c i µ i ( x ) >

Original: λ i ( x ) .
Revised: λ i ( x ) .

Original: Therefore, the Jackson Network is stable as well.
Revised: Therefore, the Jackson Network is stable as well.

Original: Assumption 4 ensures that the underlying Jackson Network is stable under any candidate price (i.e.,the queue length will not grow indeﬁnitely).
Revised: Assumption 4 ensures that the underlying Jackson Network is stable under any candidate price (i.e., the queue length will not grow indefinitely).

Original: This assumption can be readily satisﬁed by puttingappropriate lower bounds on the price vector.
Revised: This assumption can be readily satisfied by putting appropriate lower bounds on the price vector.

Original: Assumption 5.
Revised: Assumption 5.

Original: (Change of rates.)
Revised: (Change of rates.)

Original: For any node, consider the arrival and service rates ( λ 1 , µ 1 ) and( λ 2 , µ 2 ) under x 1 , x 2 ∈ X where λ 1 ≥ λ 2 , also denote the utilization factor by ρ 1 = λ 1 /µ 1 [-andρ 
-]
Revised: For any node, consider the arrival and service rates ( λ 1 , µ 1 ) and ( λ 2 , µ 2 ) under x 1 , x 2 ∈ X where λ 1 ≥ λ 2 , also denote the utilization factor by ρ 1 = λ 1 /µ 1 [-andρ 
-]and ρ 2 = λ 2 /µ 2 , respectively, then 1) λ 1 − λ 2 ≤ [Equation] and 2) if µ

Original: = λ 2 /µ 2 , respectively, then 1) λ 1 − λ 2 ≤ ρ 1 − 3 e log ρ 1 and 2) if µ 1 > µ 2 , then ρ 1 ≥ ρ 22 .
Revised: [Equation] > µ 2 , then ρ [Equation] ≥ ρ 22 .

Original: This assumption is used to justify Lemmas 3 and 4 (used in the proof of Propositions 3 and 4) and theoriginal proofs are in Jia et al.
Revised: This assumption is used to justify Lemmas 3 and 4 (used in the proof of Propositions 3 and 4) and the original proofs are in Jia et al.

Original: These conditions are mild and easy to validate in practice in thefollowing sense.
Revised: These conditions are mild and easy to validate in practice in the following sense.

Original: The difference in arrival rates between two prices is bounded (by a large constant).
Revised: The difference in arrival rates between two prices is bounded (by a large constant).

Original: This can be easily satisﬁed by interpolating intermediate prices in the candidate price set X . 2)
Revised: This can be easily satisfied by interpolating intermediate prices in the candidate price set X . 2)

Original: Ifboth the service and arrival rate under a price is higher than or equal to another price, the utilizationfactor in the former case must be greater than the latter squared.
Revised: If both the service and arrival rate under a price is higher than or equal to another price, the utilization factor in the former case must be greater than the latter squared.

Original: A simple sufﬁcient condition toguarantee ρ 1 ≥ ρ 22 is ρ 1 ≥ ρ 2 , which reduces to λ 1 /λ 2 ≥ µ 1 /µ 2 , and thus the practitioners onlyneed to validate that the ratio of change in arrival rates exceeds the ratio of change in service rates.
Revised: A simple sufficient condition to guarantee ρ 1 ≥ ρ 22 is ρ 1 ≥ ρ 2 , which reduces to λ 1 /λ 2 ≥ µ 1 /µ 2 , and thus the practitioners only need to validate that the ratio of change in arrival rates exceeds the ratio of change in service rates.

Original: Assumption 6.
Revised: Assumption 6.

Original: We assume log T ≥ 4 (to ensure sufﬁcient times for mixing).
Revised: We assume log T ≥ 4 (to ensure sufficient times for mixing).

Original:  Regret, relaxed regret, and static price benchmark
Revised:  Regret, Relaxed Regret, and Static Price Benchmark

Original: Under the full information scenario, where the ﬁrm knows the underlying arrival rate α j ( x ) of allproducts j ∈ J under any candidate price x ∈ X .
Revised: Under the full information scenario, where the firm knows the underlying arrival rate α j ( x ) of all products j ∈ J under any candidate price x ∈ X .

Original: We denote a state-dependent optimal policy as π ∗ ,
Revised: We denote a state-dependent optimal policy as π ∗ ,

Original: where π ∗ = argmax π Y π , and let Y ∗ =
Revised: where π ∗ = argmax π Y π , and let Y ∗ =

Original: Y π ∗ .
Revised: Y π ∗ .

Original: Therefore, the regret of any pricing policy π can bedeﬁned as Regret ( π, T ) =
Revised: Therefore, the regret of any pricing policy π can be defined as Regret ( π, T ) =

Original: Y ∗ − Y π .
Revised: Y ∗ − Y π .

Original: In short, we aim at ﬁnding heuristic pricing policies that leadto a small regret.
Revised: In short, we aim at finding heuristic pricing policies that lead to a small regret.

Original: However, ﬁnding the state-dependent optimal pricing policy π ∗ is computationallyintractable even under the full information scenario.
Revised: However, finding the state-dependent optimal pricing policy π ∗ is computationally intractable even under the full information scenario.

Original: Firstly, it requires to solve a dynamic programwith an inﬁnite number of potential states, in the form of ( y , t ) , which suffers from the curse ofdimensionality.
Revised: Firstly, it requires solving a dynamic program with an infinite number of potential states, in the form of ( y , t ) , which suffers from the curse of dimensionality.

Original: Moreover, the transient dynamics of a queueing network is complex and hard toanalyze especially when the offered price changes over periods.
Revised: Moreover, the transient dynamics of a queueing network are complex and hard to analyze especially when the offered price changes over periods.

Original: Thus, it is impossible to obtain π ∗and Y ∗ .
Revised: Thus, it is impossible to obtain π ∗ and Y ∗ .

Original: To tackle this problem, we develop an upper bound on the expected revenue achievable bythe state-dependent optimal expected revenue (see a similar approach in Jia et al. 2020).
Revised: To tackle this problem, we develop an upper bound on the expected revenue achievable by the state-dependent optimal expected revenue (see a similar approach in Jia et al. 2020).

Original: An LP upper bound.
Revised: An LP upper bound.

Original: Consider continuous decision variables π xy t for y ∈ { 0 , 1 , . .
Revised: Consider continuous decision variables π xy t for y ∈ { 0 , 1 , . .

Original: , ∞} I , t =1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T, x ∈ X , a y t for y ∈ { 0 , 1 , . . .
Revised: , T, x ∈ X , a y t for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T , and Y y t for y ∈ { 0 , 1 , . . .
Revised: , T , and Y y t for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t =1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T .
Revised: , T .

Original: We can formulate a linear program (LP) as follows.
Revised: We can formulate a linear program (LP) as follows.

Original: [Equation]
Revised: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: The LP solution provides an upper bound on the optimal revenue, i.e., Y ∗ ≤ Y LP ,and Y LP = T · (cid:126)α T ˜ x
Revised: The LP solution provides an upper bound on the optimal revenue, i.e., Y ∗ ≤ Y LP , and Y LP = T · ⃗α T ˜ x

Original: ˜ x where ˜ x = arg max x ∈X (cid:126)α T x x .
Revised: ˜ x where ˜ x = arg max x ∈X ⃗α T x x .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: Consider an arbitrary policy ˆ π .
Revised: Consider an arbitrary policy ˆ π .

Original: Let variable ˆ a y t take the value of probability that thesystem has y customers at the beginning of period t for y ∈ { 0 , 1 , . . .
Revised: Let variable ˆ a y t take the value of probability that the system has y customers at the beginning of period t for y ∈ { 0 , 1 , . . .

Original: , ∞} I and t = 1 , . . .
Revised: , ∞} I and t = 1 , . . .

Original: , T .
Revised: , T .

Original: Letvariable ˆ π xy t = 1 if the policy ˆ π chooses price x at state ( y , t ) , and 0 otherwise.
Revised: Let variable ˆ π xy t = 1 if the policy ˆ π chooses price x at state ( y , t ) , and 0 otherwise.

Original: Let variable ˆ Y y ttake the value of expected revenue collected before the end of period T from the customers whoarrive during period t , when the system has y customers at the beginning of period t .
Revised: Let variable ˆ Y y t take the value of expected revenue collected before the end of period T from the customers who arrive during period t , when the system has y customers at the beginning of period t .

Original: Then we canshow that the decisions variables ( π, a, Y ) associated with any admissible policy must satisfy the constraints of the linear program (1), and the expected revenue of this admissible policy is exactly thecorresponding objective value in (1).
Revised: Then we can show that the decision variables ( π, a, Y ) associated with any admissible policy must satisfy the constraints of the linear program (1), and the expected revenue of this admissible policy is exactly the corresponding objective value in (1).

Original: We can readily check that T · (cid:126)α T ˜ x
Revised: We can readily check that T · ⃗α T ˜ x

Original: ˜ x is an upper bound of Y LP .
Revised: ˜ x is an upper bound of Y LP .

Original: We can construct a solution which can obtain this upper bound and thus Y LP = T · (cid:126)α T ˜ x
Revised: We can construct a solution which can obtain this upper bound and thus Y LP = T · ⃗α T ˜ x

Original: ˜ x .
Revised: ˜ x .

Original: Deﬁnition 1.
Revised: Definition 1.

Original: The relaxed regret is deﬁned as Regret ( π, T ) =
Revised: The relaxed regret is defined as Regret ( π, T ) =

Original: Y LP − Y π .
Revised: Y LP − Y π .

Original: Based on Proposition 1, we have for any policy π , the corresponding relaxed regret is a valid upperbound of its regret, i.e., Regret ( π, T ) ≥ Regret ( π, T ) , and it sufﬁces to bound the relaxed regret.
Revised: Based on Proposition 1, we have for any policy π , the corresponding relaxed regret is a valid upper bound of its regret, i.e., Regret ( π, T ) ≥ Regret ( π, T ) , and it suffices to bound the relaxed regret.

Original:  Batched bandit algorithms
Revised:  Batched Bandit Algorithms

Original: The basis for implementing an Upper Conﬁdence Bound (UCB) type of algorithm is identifyingan upper conﬁdence bound of the expected collected revenue of each candidate price.
Revised: The basis for implementing an Upper Confidence Bound (UCB) type of algorithm is identifying an upper confidence bound of the expected collected revenue of each candidate price.

Original: We startwith the steady-state revenue rate.
Revised: We start with the steady-state revenue rate.

Original: We can show that this steady-state revenue rate can be used tocompute a lower bound of the expected revenue during consecutive periods (see proof of Theorem1).
Revised: We can show that this steady-state revenue rate can be used to compute a lower bound of the expected revenue during consecutive periods (see proof of Theorem 1).

Original: Therefore, it is equivalent to deriving an upper conﬁdence bound of the steady-state revenuerate based on historical sales information.
Revised: Therefore, it is equivalent to deriving an upper confidence bound of the steady-state revenue rate based on historical sales information.

Original: Jia et al.
Revised: Jia et al.

Original: (2020) established a concentration bound of thesteady-state revenue rate under the single product setting.
Revised: (2020) established a concentration bound of the steady-state revenue rate under the single product setting.

Original: Here we develop a concentration bound forthe multi-product multi-resource setting.
Revised: Here we develop a concentration bound for the multi-product multi-resource setting.

Original: We use n j ( x ) to denote the number of arrived customers requesting product j under price x .
Revised: We use n j ( x ) to denote the number of arrived customers requesting product j under price x .

Original: Weuse ˆ d jk ( x ) to denote the observed time interval between the arrival time of customer k and that ofcustomer k − 1 who are requesting product j under the price x .
Revised: We use ˆ d jk ( x ) to denote the observed time interval between the arrival time of customer k and the arrival time of customer k − 1 who are requesting product j under the price x .

Original: We use ¯ d j ( x ) to denote the empiricalmean of the inter-arrival time and thus ¯ d j ( x ) = (cid:80) n j ( x ) k
Revised: We use ¯ d j ( x ) to denote the empirical mean of the inter-arrival time and thus ¯ d j ( x ) = (cid:80) n j ( x ) k

Original: =1 ˆ d jk ( x ) /n
Revised: =1 ˆ d jk ( x ) /n

Original: j ( x ) , for j ∈ J .
Revised: j ( x ) , for j ∈ J .

Original: Proposition 2.
Revised: Proposition 2.

Original: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( J 14 T ) for j ∈ J on ( x ) , then we have[Equation] 
Revised: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( J 14 T ) for j ∈ J on ( x ) , then we have

Original: [Equation]
Revised: [Equation]

Original: The proof of Proposition 2 is based on the concentration inequality for sub-exponential randomvariables (see, e.g., Boucheron et al.
Revised: The proof of Proposition 2 is based on the concentration inequality for sub-exponential random variables (see, e.g., Boucheron et al.

Original: 2013, Jia et al. 2020)
Revised: 2013, Jia et al. 2020)

Original: and is provided in Appendix C. Note that(cid:126)α T x x is the steady-state revenue rate and denote it by r ( (cid:126)α T x , x ) .
Revised: and is provided in Appendix C. Note that ⃗α T x x is the steady-state revenue rate and denote it by r ( ⃗α x , x ) .

Original: Proposition 2 says that with a certainamount of arrival time observations of turn-on products under a candidate price, we can derive a validupper bound of the steady-state revenue rate with a high probability guarantee.
Revised: Proposition 2 says that with a certain amount of arrival time observations of turn-on products under a candidate price, we can derive a valid upper bound of the steady-state revenue rate with a high probability guarantee.

Original: Hence, we deﬁnethe corresponding upper and lower conﬁdence bounds of the stationary revenue rate associated withprice x by the end of period t as follows.
Revised: Hence, we define the corresponding upper and lower confidence bounds of the stationary revenue rate associated with price x by the end of period t as follows.

Original: Deﬁnition 2.
Revised: Definition 2.

Original: Deﬁne Rad
Revised: Define

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ¯ d j ( x ) √ n j ( x ) as the conﬁdence radius by the end ofperiod t .
Revised: √ n j ( x ) as the confidence radius by the end of period t .

Original: The upper and lower conﬁdence bounds of the revenue rate associated with price x are:
Revised: The upper and lower confidence bounds of the revenue rate associated with price x are:

Original: [Equation]
Revised: [Equation]

Original: 4.1 RNRM-UCB
Revised: 4.1 RNRM-UCB

Original: With Deﬁnition 2, we can leverage the idea of UCB to solve RNRM.
Revised: With Definition 2, we can leverage the idea of UCB to solve RNRM.

Original: We separate T periods into twophases.
Revised: We separate T periods into two phases.

Original: The ﬁrst phase is the Warm-up Phase, where we offer each candidate price one by one until wecollect 8 log( J 1/ 4 T ) number of arrival time intervals for each turn-on product j ∈ J on ( x ) under eachprice.
Revised: The first phase is the Warm-up Phase, where we offer each candidate price one by one until we collect 8 log( J 1 4 T ) number of arrival time intervals for each turn-on product j ∈ J on ( x ) under each price.

Original: Therefore, the total length of this Warm-up Phase is O
Revised: Therefore, the total length of this Warm-up Phase is O

Original: ( X log( J 1 / 4 T )) .
Revised: ( X log( J 14 T )) .

Original: After thisWarm-up
Revised: After this

Original:  Phase, Proposition 2 is valid to use.
Revised: Warm-up Phase, Proposition 2 is valid to use.

Original: The second phase is the Learning Phase.
Revised: The second phase is the Learning Phase.

Original: The UCB algorithmcannot be directly used because in order to estimate the expected revenuecollected when offeringa price, we have to wait until the system reaches the steady-state.
Revised: The UCB algorithm cannot be directly used because in order to estimate the expected revenue when offering a price, we have to wait until the system reaches the steady-state.

Original: The key idea is to reduce thenumber of price changes and further reduce the intractable transient state performance.
Revised: The key idea is to reduce the number of price changes and further reduce the intractable transient state performance.

Original: We separatethe Learning Phase into M number of batches.
Revised: We separate the Learning Phase into M number of batches.

Original: Deﬁne I m = 2 m , ∀ m = 1 , . . .
Revised: Define I m = 2 m , ∀ m = 1 , . . .

Original: , M .
Revised: , M .

Original: Each batch mcontains I m Lτ periods, where τ = (log T ) 2 .
Revised: Each batch m contains I m Lτ periods, where τ = (log T ) 2 and L is the number of layers defined in Definition 3.

Original: Therefore, the batch size is exponentially increasing.
Revised: Therefore, the batch size is exponentially increasing.

Original: We can then add m in the subscript of notation ( n mj ( x ) , ˆ d jk ( x ) , ¯ d mj ( x ) , Rad m ( x ) , U m ( x ) , L m ( x ))to represent their values by the end of batch m .
Revised: We can then add m in the subscript of notation ( n mj ( x ) , ˆ d jk ( x ) , ¯ d mj ( x ) , Rad m ( x ) , U m ( x ) , L m ( x )) to represent their values by the end of batch m .

Original: At the beginning of each batch m + 1 , RNRM-UCBchooses price x with the highest U m ( x ) and offers this price during batch m
Revised: At the beginning of each batch m + 1 , RNRM-UCB chooses price x with the highest U m ( x ) and offers this price during batch m

Original: + 1 .
Revised: + 1 .

Original: We present thealgorithmic details in Algorithm 1.
Revised: We present the algorithmic details in Algorithm 1.

Original: 4.2 RNRM-TS
Revised: 4.2 RNRM-TS

Original: Considering the ﬁrm holds prior information about the rates, then we can add the Bayesian assumptionto this RNRM problem.
Revised: Considering the firm holds prior information about the rates, then we can add the Bayesian assumption to this RNRM problem.

Original: By the conjugate prior theory (see, e.g., Minton et al.
Revised: By the conjugate prior theory (see, e.g., Minton et al.

Original: 1962, Diaconis and
Revised: 1962, Diaconis and

Original: Ylvisaker 1979), we assume that RNRM-TS holds Gamma priors on α j ( x ) of product j ∈ J on ( x ) ,∀ x ∈ X and the posteriors are updated by the end of batch m, ∀ m = 1 , . . .
Revised: Ylvisaker 1979), we assume that RNRM-TS holds Gamma priors on α j ( x ) of product j ∈ J on ( x ) , ∀ x ∈ X and the posteriors are updated by the end of batch m, ∀ m = 1 , . . .

Original: , M .
Revised: , M .

Original: Speciﬁcally, weset the prior distribution of α j ( x ) as Gamma ( η α j ( x ) , β α j ( x ) ) .
Revised: Specifically, we set the prior distribution of α j ( x ) as Gamma ( η α j ( x ) , β α j ( x ) ) .

Original: After observing one arrival timeinterval ˆ d jk ( x ) of product j while offering price x , the posterior distribution of α j ( x ) is computedas Gamma ( η α j ( x ) + 1 , β α j ( x ) + ˆ d jk ( x )) .
Revised: After observing one arrival time interval ˆ d jk ( x ) of product j while offering price x , the posterior distribution of α j ( x ) is computed as Gamma ( η α j ( x ) + 1 , β α j ( x ) + ˆ d jk ( x )) .

Original: To implement RNRM-TS, at the beginning of batch m ,we sample parameters ˆ α j ( x ) for product j ∈ J on ( x ) , ∀ x ∈ X , following the updated posteriordistributions.
Revised: To implement RNRM-TS, at the beginning of batch m , we sample parameters ˆ α j ( x ) for product j ∈ J on ( x ) , ∀ x ∈ X , following the updated posterior distributions.

Original: Then we offer the price with the maximum sampled stationary network revenue rater (ˆ (cid:126)α x , x ) = (cid:80) j ∈J on ( x ) x
Revised: Then we offer the price with the maximum sampled stationary network revenue rate r (ˆ ⃗α x , x ) = (cid:80) j ∈J on ( x ) x

Original: j ˆ α j ( x ) .
Revised: j ˆ α j ( x ) .

Original: We present the algorithmic details of RNRM-TS in Appendix.
Revised: We present the algorithmic details of RNRM-TS in Algorithm 2.

Revised: time ˆ d jk ( x m ) for arriving customers of product j ∈ J on ( x m ) .

Revised: Compute and update η α j ( x m ) , β α j ( x m ) , ∀ j ∈ J on ( x m ) .

Revised: 18: end for

Original:  Performance analysis
Revised:  Performance Analysis 5.1 High Probability Bound for Mixing Time

Original: 5.1 High probability bound for mixing time

Original: When the ﬁrm starts service from an empty state, i.e., zero customers in the network service system,or switches to another price from the currently offered price, the service (queueing network) systementers a transient state and needs a certain amount of time to reach the steady-state again (referredto as the mixing time).
Revised: When the firm starts service from an empty state, i.e., zero customers in the network service system, or switches to another price from the currently offered price, the service (queueing network) system enters a transient state and needs a certain amount of time to reach the steady-state again (referred to as the mixing time).

Original: Recall that the service system can be reduced to a Jackson Network undera speciﬁc offered price vector, upon completion of existing customers under the previously postedprice.
Revised: Recall that the service system can be reduced to a Jackson Network under a specifically offered price vector, upon completion of existing customers under the previously posted price.

Original: Therefore, we evaluate how quickly a Jackson Network can reach the steady-state.
Revised: Therefore, we evaluate how quickly a Jackson Network can reach the steady-state.

Original: To the bestof our knowledge, this is the ﬁrst result giving a ﬁnite-time high probability bound on mixing timesof Jackson Network system (upon action changes).
Revised: To the best of our knowledge, this is the first result giving a finite-time high probability bound on mixing times of Jackson Network system (upon action changes).

Original: Technically, we develop a new coupling argumentbetween this target system and a virtual system starting from a state sampled from the steady-statedistribution and maintaining the steady-state thereafter.
Revised: Technically, we develop a new coupling argument between this target system and a virtual system starting from a state sampled from the steady-state distribution and maintaining the steady-state thereafter.

Original: Poisson arrival and departure under steady-state.
Revised: Poisson arrival and departure under steady-state.

Original: Under the steady-state of an M/M/c queue (if itexists, i.e., ρ = λ/cµ < 1 ), then the process following which the customers leaving the system is alsoa Poisson process with rate λ .
Revised: Under the steady-state of an M/M/c queue (if it exists, i.e., ρ = λ/cµ < 1 ), then the process following which the customers leave the system is also a Poisson process with rate λ .

Original: Therefore, one can easily derive that the arrival as well as the departureprocesses of each type of resource (each node in the network representation) are Poisson processeswith the same rate λ i under the steady-state (see the famous Burke’s Theorem Burke 1956).
Revised: Therefore, one can easily derive that the arrival as well as the departure processes of each type of resource (each node in the network representation) are Poisson processes with the same rate λ i under the steady-state (see the famous Burke’s Theorem Burke 1956).

Original: Jia et al.
Revised: Jia et al.

Original: (2020) is the only work for studying ﬁnite-time bounds on the mixing time for an M/M/cqueue.
Revised: (2020) is the only work for studying finite-time bounds on the mixing time for an M/M/c queue.

Original: However, their results cannot be directly applied here because (i) the arrival process of eachresource is not a well-deﬁned Poisson process and (ii) the starting point of the later queues is notzero state nor a steady-state of old system parameters.
Revised: However, their results cannot be directly applied here because (i) the arrival process of each resource is not a well-defined Poisson process and (ii) the starting point of the later queues is not zero state nor a steady-state of old system parameters.

Original: Even with these two challenges tackled, thisapproach will induce ahigh-probability mixing time which is linear on the number of resources.
Revised: Even with these two challenges tackled, this approach will induce a mixing time which is linear on the number of resources.

Original: Thisresult is not desired for systems with multiple (especially large)number of resources.
Revised: This result is not desired for systems with multiple (especially large) resources.

Original: We develop anew mixing time analysis for Jackson Network.
Revised: We develop a new mixing time analysis for Jackson Network.

Original: We divide resource of the service network into layersand derive a high-probability mixing time that is only linear on the number of layers.
Revised: We divide the resources of the service network into layers and derive a high-probability mixing time that is only linear on the number of layers.

Original: Deﬁnition 3.
Revised: Definition 3.

Original: We divide the nodes (types of servers) into layers indexed by the maximum number ofpreviously occupied nodes (including the virtual source node) along the service routes of all products.
Revised: We divide the nodes (types of servers) into layers indexed by the maximum number of previously occupied nodes (including the virtual source node) along the service routes of all products.

Original: For example, Nodes 1 and 2 of the illustrative example (see Figure 2) is Layer-1 nodes and Node 3 isa Layer-2 node.
Revised: For example, Nodes 1 and 2 of the illustrative example (see Figure 2) is Layer-1 nodes, and Node 3 is a Layer-2 node.

Original: We denote the number of layers by L and the set of Layer- l node as I l .
Revised: We denote the number of layers by L and the set of Layer- l nodes as I l .

Original: Proposition 3.
Revised: Proposition 3.

Original: (Coupling probability of a Jackson Network starting from empty state)Deﬁne
Revised: (Coupling probability of a Jackson Network starting from empty state)

Original:  A t tobe the event in which the Jackson Network reaches the steady-state by time t .
Revised: Define A t to be the event in which the Jackson Network reaches the steady-state by time t .

Original: For t ≥ Lτ , whereτ = (log T ) 2 , we have P ( A t ) ≥ 1 − 2 IT 2 .
Revised: For t ≥ Lτ , where τ = (log T ) 2 , we have P ( A t ) ≥ 1 − 2 IT 2 .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: We use the aforementioned Poisson arrival and departure property of the nodes in a Jackson Network to bound the mixing time of the nodes layer by layer.
Revised: We use the aforementioned Poisson arrival and departure property of the nodes in a Jackson Network to bound the mixing time of the nodes layer by layer.

Original: We deﬁne events A t (cid:48) ,l = { all layer-l nodes reach the steady-state by time t (cid:48) } and A i t (cid:48)
Revised: We define events A t ′ ,l = { all layer-l nodes reach the steady-state by time t ′ } and A it ′

Original: = { the node i reaches the steady-state by time t (cid:48) } .
Revised: = { the node i reaches the steady-state by time t ′ } .

Original: By treating the nodes in the same layer together,
Revised: By treating the nodes in the same layer together,

Original: [Equation]
Revised: [Equation]

Original: τ, 1 )

Original: L (cid:89) l =

Original: P ( A lτ,l |A τ, 1 . . .

Original: A ( l − 1) τ,l − 1 ) .

Original: We then prove that P ( A lτ,l |A τ, 1 . . .
Revised: We then prove that P ( A lτ,l |A τ, 1 . . .

Original: A ( l − 1) τ,l − 1 ) ≥ 1 − (2 I l ) /T 2 , where I l = |I
Revised: A ( l − 1) τ,l − 1 ) ≥ 1 − (2 I l ) /T 2 , where I l = |I

Original: l | , by establishing acoupling argument between the target queue (with the help of an artiﬁcially constructed alternativequeue) and a virtual queue starting from a state sampled from the steady-state distribution andmaintaining the steady-state thereafter.
Revised: l | , by establishing a coupling argument between the target queue (with the help of an artificially constructed alternative queue) and a virtual queue starting from a state sampled from the steady-state distribution and maintaining the steady-state thereafter.

Original: The detailed proof is provided in Appendix E.
Revised: The detailed proof is provided in Appendix E.

Original: In addition, the transient state of the Jackson Network, as well as each resource queue (or each node),can be induced not only by starting from empty state but also by starting from a steady-state of apreviously posted price.
Revised: In addition, the transient state of the Jackson Network, as well as each resource queue (or each node), can be induced not only by starting from empty state but also by changing the price from the previously posted price.

Original: It is possible that when a new price is posted, the customers  arrived under thepreviously posted price still remain in the system.
Revised: It is possible that when a new price is posted, the customers who arrived under the previously posted price still remain in the system.

Original: Therefore we also need to ensure that the couplingalso occurs with high probability when price changes.
Revised: Therefore we also need to ensure that the coupling also occurs with high probability when price changes.

Original: Proposition 4.
Revised: Proposition 4.

Original: (Coupling probability of Jackson Network starting from steady-state of previousprice.)
Revised: (Coupling probability of Jackson Network starting from steady-state of the previous price.)

Original: Deﬁne B t to be the event in which the Jackson Network reaches the steady-state by time t .
Revised: Define B t to be the event in which the Jackson Network reaches the steady-state by time t .

Original: Fort ≥ 2 Lτ , where τ = (log T ) 2 , we have P ( B t ) ≥ 1 − 4 IT 2 .
Revised: For t ≥ 2 Lτ , where τ = (log T ) 2 , we have P ( B t ) ≥ 1 − 4 IT 2 .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: This proposition is derived based on Proposition 3 and Assumption 5.
Revised: This proposition is derived based on Proposition 3 and Assumption 5.

Original: We deﬁne B t (cid:48) ,land B it (cid:48) similarly and bound P ( B 2 lτ,l |B 2 τ, 1 . . .
Revised: We define B t ′ ,l and B it ′ similarly and bound P ( B 2 lτ,l |B 2 τ, 1 . . .

Original: B 2( l − 1) τ,l − 1 ) ≥ 1 − 4 I l T 2 by decomposing the scenariosby how the arrival and service rates are changing when considering each resource queue.
Revised: B 2( l − 1) τ,l − 1 ) ≥ 1 − 4 I l T 2 by decomposing the scenarios by how the arrival and service rates are changing when considering each resource queue.

Original: 5.2 Regret bounds of RNRM-UCB and RNRM-TS
Revised: 5.2 Regret Bounds of RNRM-UCB and RNRM-TS

Original: The high probability bounds for mixing time of Jackson Network developed in §5.1 help bound theloss due to the nonstationarity, and thus we are ready to carry out the regret analysis of RNRM-UCB.
Revised: The high probability bounds for mixing time of Jackson Network developed in §5.1 help bound the loss due to the nonstationarity, and thus we are ready to carry out the regret analysis of RNRM-UCB.

Original: Theorem 1.
Revised: Theorem 1.

Original: The T -period cumulative regret of RNRM-UCB is bounded by ˜ O (cid:16) J √ XT (cid:17) .
Revised: The T -period cumulative regret of RNRM-UCB is bounded by ˜ O (cid:16) J √ XT (cid:17) .

Original: Note that the regret lower bound for nominal MAB (where J = 1 ) is Ω( √ XT ) (see, e.g., Theorem2.1 in Slivkins (2019)), which suggests that our regret bound is tight, if the system size J is ﬁxed.
Revised: Note that the regret lower bound for nominal MAB (where J = 1 ) is Ω( √ XT ) (see, e.g., Theorem 2.1 in Slivkins (2019)), which suggests that our regret bound is tight, if the system size J is fixed.

Original: Proof Sketch.
Revised: Proof Sketch.

Original: The regret of the Warm-up phase is at most linear on the length of the Warm-up
Revised: The regret of the Warm-up phase is at most linear on the length of the Warm-up

Original: Phase, i.e., O ( XJ log( J 1 / 4 T )) .
Revised: Phase, i.e., O ( X log( J 1 / 4 T )) .

Original: We decompose the relaxed regret in the Learning Phase into twoparts.
Revised: We decompose the relaxed regret in the Learning Phase into two parts.

Original: The ﬁrst part is called loss of nonstationarity and it represents the revenue loss induced bythe transient performance when changing the price.
Revised: The first part is called loss of nonstationarity and it represents the revenue loss induced by the transient performance when changing the price.

Original: For batch m , we can derive that the expectedrevenue collected under the selected price x m as r ( (cid:126)α x m , x m )
Revised: For batch m , we can derive that the expected revenue collected under the selected price x m as r ( ⃗α x m , x m )

Original: I m Lτ − O
Revised: I m Lτ − O

Original: ( Lτ ) with Proposition 4.
Revised: ( Lτ ) with Proposition 4.

Original: The second part represents the loss of suboptimality , which is induced by not offering the optimalstate-independent price in certain time periods.
Revised: The second part represents the loss of suboptimality , which is induced by not offering the optimal state-independent price in certain time periods.

Original: This part can be further rewritten as ∆( x m ) I m Lτ and ∆( x m ) =
Revised: This part can be further rewritten as ∆( x m ) I m Lτ and ∆( x m ) =

Original: ( r ( (cid:126)α ˜ x , ˜ x ) − U m (˜ x ))
Revised: ( r ( ⃗α ˜ x , ˜ x ) − U m (˜ x ))

Original: + ( U m ( x m ) − L m ( x m ))
Revised: + ( U m ( x m ) − L m ( x m ))

Original: + ( L m ( x m ) − r ( (cid:126)α x m , x m )) .
Revised: + ( L m ( x m ) − r ( ⃗α x m , x m )) .

Original: We boundthese three terms separately based on Proposition 2 and Lemma 2.
Revised: We bound these three terms separately based on Proposition 2 and Lemma 2.

Original: Combine all results together, wereach a ˜ O ( J √ XT ) regret bound.
Revised: Combine all results together, we reach a ˜ O ( J √ XT ) regret bound.

Original: We provided detailed proof in Appendix E.
Revised: We provided detailed proof in Appendix E.

Original: We use Bayesian regret as the performance measure of RNRM-TS, which is typically used for the
Revised: We use Bayesian regret as the performance measure of RNRM-TS, which is typically used for the

Original: Bayesian setting and is computed as the expected regret of the instance in expectation over all probleminstances.
Revised: Bayesian setting and is computed as the expected regret of the instance in expectation over all problem instances.

Original: Similarly, we deﬁne a relaxed Bayesian regret:.
Revised: Similarly, we define a relaxed Bayesian regret.

Original: Deﬁnition 4.
Revised: Definition 4.

Original: Denoting all the events before period t by F t , ∀ t ≥ 0 , we deﬁne the relaxed Bayesian regret of policy π by period T as
Revised: Denoting all the events before period t by F t , ∀ t ≥ 0 , we define the relaxed Bayesian regret of policy π by period T as

Original: [Equation]
Revised: [Equation]

Original: The relaxed Bayesian regret bound of RNRM-TS is in the same order as RNRM-UCB (Appendix E).
Revised: The relaxed Bayesian regret bound of RNRM-TS is in the same order as RNRM-UCB (Appendix E).

Original: Theorem 2.
Revised: Theorem 2.

Original: The T -period cumulative Bayesian regret of RNRM-TS is bounded by ˜ O
Revised: The T -period cumulative Bayesian regret of RNRM-TS is bounded by ˜ O

Original: ( J √ XT ) .
Revised: ( J √ XT ) .

Original:  Numerical results
Revised:  Numerical Results

Original: We consider T = 2000 , I = 2 with capacity c 1 =
Revised: We consider T = 2000 , I = 2 with capacity c 1 =

Original: c
Revised: c

Original: = 2 ,
Revised: = 2 ,

Original: J = 3 with product-resource orders{ 1 , 2 , 1 → 2 } , and X = { [1 , 2 , 3] , [2 , 1 , 3] , [2 , 2 , 3] } .
Revised: J = 3 with product-resource orders { 1 , 2 , 1 → 2 } , and X = { [1 , 2 , 3] , [2 , 1 , 3] , [2 , 2 , 3] } .

Original: We set the underlying arrival rates of candidateprices as { [4 , 2 , 4] , [3 , 4 , 3] , [2 . 5 , 3 , 2 . 5] } and usea uniform service rates [5 , 4 . 5] for all prices.
Revised: We set the underlying arrival rates of candidate prices as { [4 , 2 , 4] , [3 , 4 , 3] , [2 . 5 , 3 , 2 . 5] } and use uniform service rates [5 , 4 . 5] for all prices.

Original: Weadopt the blind network revenue management algorithm proposed in Besbes and Zeevi (2012) toour settings as a benchmark.
Revised: We adopt the blind network revenue management algorithm proposed in Besbes and Zeevi (2012) to our setting as a benchmark.

Original: This benchmark algorithm tries each price for θT/X periods and thenselects the price with the highest revenue in the rest of the periods.
Revised: This benchmark algorithm tries each price for θT/X periods and then selects the price with the highest revenue for the rest of the periods.

Original: We set θ as 0 .
Revised: We set θ as 0 .

Original: 1 , and 0 .
Revised: 1 , and 0 .

Original: in three Besbes and Zeevi (2012) benchmarks, respectively (the same as the settings in Table 1 inthe original paper Besbes and Zeevi 2012).
Revised: in three Besbes and Zeevi (2012) benchmarks, respectively (the same as the settings in Table 1 in the original paper Besbes and Zeevi 2012).

Original: To conclude, we implement ﬁve pricing algorithms: 
Revised: To conclude, we implement five pricing algorithms: RNRM-UCB,

Original: RNRM-UCB, RNRM-TS, and three benchmark policies with different values of θ .
Revised: RNRM-TS, and three benchmark policies with different values of θ .

Original: Figure 1a shows the offered price over periods of each algorithm and Figure 1b depicts the cumulativetime-average relaxed regret, i.e., (cid:80) tt (cid:48)
Revised: Figure 1a shows the offered price over periods of each algorithm and Figure 1b depicts the cumulative time-average relaxed regret, i.e., (cid:80) tt ′

Original: =1 ( Y LP t (cid:48) − Y πt (cid:48) ) /t .
Revised: =1 ( Y LP t ′ − Y πt ′ ) /t .

Original: For all algorithms (except “OPT”), the revenueis collected from the simulation where the system starts with no customers at time 0 .
Revised: For all algorithms (except “OPT”), the revenue is collected from the simulation where the system starts with no customers at time 0 .

Original: We can observethat both RNRM-UCB and RNRM-TS identify the best price (P1) through the learning process andthe time average regret shrinks to zero very quickly.
Revised: We can observe that both RNRM-UCB and RNRM-TS identify the best price (P1) through the learning process and the time average regret shrinks to zero very quickly.

Original: [Figure]
Revised: [Figure]

Original:  Conclusion and social impacts
Revised:  Conclusion and Broader Impacts

Original: We proposed two new batch bandit algorithms for RNRM problems with multiple products andmultiple reusable resources.
Revised: We proposed two new batch bandit algorithms for RNRM problems with multiple products and multiple reusable resources.

Original: We established optimal regret upper bounds and demonstrated superiorcomputational results.
Revised: We established optimal regret upper bounds and demonstrated superior computational results.

Original: This work solves an important practical problem and we believe that ourapproach will open many doors to learning more complex stochastic systems (especially in thenetwork settings).
Revised: This work solves an important practical problem and we believe that our approach will open many doors to learning more complex stochastic systems (especially in network settings).

Original: There are two immediate future research questions: (1) how we can accommodatecontinuous price space; and (2) how we can extend arrivals and services to general distributions.
Revised: There are two immediate future research questions: (1) how we can accommodate continuous price space; and (2) how we can extend arrivals and services to general distributions.

Original: Bothdirections would need signiﬁcant methodological innovations to be developed.
Revised: Both directions would need significant methodological innovations to be developed.

Original: Checklist

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes] See Section 1.2.

Original: (b) Did you describe the limitations of your work?

Original: [Yes] We consider a speciﬁc service

Original: system that can be reduced to a Jackson Network.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] Nopotential negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 2.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We provide proof

Original: sketch in the main body of the paper and provide detailed proof in the appendix.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We include

Original: detailed experiment setup in the appendix.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [N/A]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [No] We are not comparing the running

Original: time and thus these information is not relevant.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cite existing

Original: lemmas.

Original: For example, see Lemma 3 in Appendix E.

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]
Revised: Acknowledgments and Disclosure of Funding

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: The authors would like to thank anonymous reviewers for their detailed and helpful comments, which have helped improve the content and exposition of this paper.

Original: Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: This research is partially supported by an Amazon research award and Department of Energy award #DE-SC0018018.

Original: [Table]

Original: [Table]

Original: B Illustrative example
Revised: B Illustrative Example

Original: We refer to this simple illustrative example several times in the rest of the paper.
Revised: We refer to this simple illustrative example several times in the rest of the paper.

Original: Consider 4 types ofresources and 4 types of products:
Revised: Consider 4 types of resources and 4 types of products:

Original: • Product No. 1 : ( s, 1 , 3 , t ) ,

Original: i

Original: e., SQL Database → EC • Product No. 2 : ( s, 1 , 3 , 4 , t ) , i.e., SQL Database → EC2 → SAP • Product No. 3 : ( s, 2 , 3 , t ) , i
Revised: • Product No. 1 : ( s, 1 , 3 , t ) , i.e., SQL Database → EC • Product No. 2 : ( s, 1 , 3 , 4 , t ) , i.e., SQL Database → EC2 → SAP • Product No. 3 : ( s, 2 , 3 , t ) , i.e., NoSQL Database → EC • Product No. 4 : ( s, 2 , 3 , 4 , t ) , i.e., NoSQL Database → EC2 → SAP

Original: e., NoSQL Database → EC

Original: • Product No. 4 : ( s, 2 , 3 , 4 , t ) , i.e., NoSQL Database → EC2 → SAP

Original: Typically, customers will choose between an SQL based or a NoSQL based database service to storeand preprocess the data.
Revised: Typically, customers will choose between an SQL based or a NoSQL based database service to store and preprocess the data.

Original: After running algorithms on elastic cloud compute EC2, only a subset ofcustomers may choose to further integrate SAP applications.
Revised: After running algorithms on elastic cloud compute EC2, only a subset of customers may choose to further integrate SAP applications.

Original: The network representation is given by
Revised: The network representation is given by

Original: Figure 2, and it is straightforward to check that this acyclic graph satisﬁes Assumption 2.
Revised: Figure 2, and it is straightforward to check that this acyclic graph satisfies Assumption 2.

Original: We assign a price vector x = ( x 1 , x 2 , x 3 , x 4 ) to these four products.
Revised: We assign a price vector x = ( x 1 , x 2 , x 3 , x 4 ) to these four products.

Original: Note that we do not assign prices to resources.
Revised: Note that we do not assign prices to resources.

Original: We only sell individual product (as a bundle of resources).
Revised: We only sell individual products (as bundles of resources).

Original: Each posted price vector induces two quantities, namely, α ( x ) and P ( x ) .
Revised: Each posted price vector induces two quantities, namely, α ( x ) and P ( x ) .

Original: Here α ( x ) is the total arrival rate of all products given price vector x while P ( x ) is the so-called (Markovian) “routing matrix” that captures the demand ﬂows for products.
Revised: Here α ( x ) is the total arrival rate of all products given price vector x while P ( x ) is the so-called (Markovian) “routing matrix” that captures the demand flows for products.

Original: One can represent a product j ∈ J using an s - t path, i.e., ( s, i 1 , . .
Revised: One can represent a product j ∈ J using an s - t path, i.e., ( s, i 1 , . .

Original: , i n , t ) .
Revised: , i n , t ) .

Original: Assume that Assumption 3 holds.
Revised: Assume that Assumption 3 holds.

Original: Then we can write the demand rate for this particular product j as
Revised: Then we can write the demand rate for this particular product j as

Original: [Equation]
Revised: [Equation]

Original: where p i,i (cid:48) are the ( i, i (cid:48) )
Revised: where p i,i ′ are the ( i, i ′ )

Original: th entry of the routing probability matrix P ( x ) .
Revised: th entry of the routing probability matrix P ( x ) .

Original: Note that in our setting, the mapping x (cid:55)→ α ( x ) , P ( x ) is unknown to the decision maker.
Revised: Note that in our setting, the mapping x (cid:55)→ α ( x ) , P ( x ) is unknown to the decision maker.

Original: Thus, weneed to offer different price vector x to learn this mapping.
Revised: Thus, we need to offer different price vectors x to learn this mapping.

Original: And we also assume this mapping isnonparametric.
Revised: And we also assume this mapping is nonparametric.

Original: For the sake of illustration, we give the following linear mapping as an example.
Revised: For the sake of illustration, we give the following linear mapping as an example.

Original: [Equation]
Revised: [Equation]

Original: Suppose the price vector x = (1 , 2 , 2 , 5) .
Revised: Suppose the price vector x = (1 , 2 , 2 , 5) .

Original: In this case, the total arrival rate α ( x ) = 100 and the routing matrix P ( x ) is given by
Revised: In this case, the total arrival rate α ( x ) = 100 and the routing matrix P ( x ) is given by

Original: [Equation]
Revised: [Equation]

Original: giving rise to
Revised:    p [Equation] p [Equation]

Revised: p

Revised: [Equation]

Revised: p

Revised: [Equation]

Revised: x

Original: Product Demand:
Revised: [Equation] .p [Equation] p

Original: [Equation]
Revised: [Equation]

Original: Server Demand:
Revised: p

Original: [Equation]
Revised: [Equation]

Original: Note that in this example, Assumption 3 also means that out of all the customers who have chosen EC2, the proportion of choosing SAP is 0.3 and the proportion of not choosing SAP is 0.7 (regardlessof database options).
Revised: , giving rise to

Original: We also consider two payment models, the lump sum payment model (presented in the mainmanuscript) and the service duration dependent payment model (presented in Appendix F). • ( Lump sum payment model.) A customer of product j ∈ J pays a lump sum x j to the ﬁrm.
Revised: Product Demand: 

Original: We provide justiﬁcation for this model as follows:
Revised: [Equation]

Original: If a customers chooses product j , then the customer will pay x j to the ﬁrm regardless of how long the services over resources are. In our illustrative example, each product can be quoted a ﬁxed price (i
Revised: Server Demand: 

Original: e., not service durationdependent), which could be seen a preset “cloud solution package”. The customers mayprefer or beneﬁt from this ﬁxed price (contract) from their own project/accounting/cash-ﬂowmanagement. Under this model, even though the customers are not sensitive to service durations, the cloud platform is nevertheless sensitive to service durations since they affect
Revised: [Equation]

Original: how many products/services can be completed within a ﬁnite horizon. • ( Service duration dependent payment model.) A customer of product j ∈ J pays theamount that is equal to the per-unit-time x j multiplied by the total service duration. This
Revised: Note that in this example, Assumption 3 also means that out of all the customers who have chosen

Original: payment scheme is duration dependent (e g., how much time is spent using EC2 in ourillustrative example). The model and analytical details are presented in Appendix F.
Revised: EC2, the proportion of choosing SAP is 0.3 and the proportion of not choosing SAP is 0.7 (regardless of database options).

Original: C Technical results in Section
Revised: C Technical Results in Section

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: We show that the decisions variables ( π, a, Y ) associated with any admissible policy must satisfy theconstraints of the linear program (1), and the expected revenue of this admissible policy is exactly thecorresponding objective value in (1).
Revised: We show that the decisions variables ( π, a, Y ) associated with any admissible policy must satisfy the constraints of the linear program (1), and the expected revenue of this admissible policy is exactly the corresponding objective value in (1).

Original: Consider an arbitrary policy ˆ π .
Revised: Consider an arbitrary policy ˆ π .

Original: We obtain the values of decisionvariables by the following steps.
Revised: We obtain the values of decision variables by the following steps.

Original: Variable ˆ a y t takes the value of probability that the system has ycustomers at the beginning of period t under the policy
Revised: Variable ˆ a y t takes the value of probability that the system has y customers at the beginning of period t under the policy

Original: ˆ π for y ∈ { 0 , 1 , . . .
Revised: ˆ π for y ∈ { 0 , 1 , . . .

Original: , ∞} I and t = 1 , . . .
Revised: , ∞} I and t = 1 , . . .

Original: , T .
Revised: , T .

Original: Therefore, constraints (1b) are immediately satisﬁed.
Revised: Therefore, constraints (1b) are immediately satisfied.

Original: Variable ˆ π xy t = 1 if the policy ˆ π chooses pricex at state ( y , t ) , otherwise, ˆ π xy t = 0 , for y ∈ { 0 , 1 , . . .
Revised: Variable ˆ π xy t = 1 if the policy ˆ π chooses price x at state ( y , t ) , otherwise, ˆ π xy t = 0 , for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T , y ∈ X .
Revised: , T , y ∈ X .

Original: As a result,constraints (1c) hold because in the left-hand side we have one variable equal to 1 and others allzero.
Revised: As a result, constraints (1c) hold because in the left-hand side we have one variable equal to 1 and others all zero.

Original: Variable ˆ Y y t takes the value of expected revenue collected before the end of period T from thecustomers who arrive during period t , when the system has y customers at the beginning of periodt , for y ∈ { 0 , 1 , . . .
Revised: Variable ˆ Y y t takes the value of expected revenue collected before the end of period T from the customers who arrive during period t , when the system has y customers at the beginning of period t , for y ∈ { 0 , 1 , . . .

Original: , ∞} I .
Revised: , ∞} I .

Original: Consider arbitrary y and t , we denote the price chosen at state ( y , t ) bythe policy ˆ π as ˆ x .
Revised: Consider arbitrary y and t , we denote the price chosen at state ( y , t ) by the policy ˆ π as ˆ x .

Original: Then, by plugging in ˆ π ˆ xy t = 1 , in the right-hand side of constraints (1d), we have(cid:126)α T x x , where (cid:126)α T x denotes the average number of arrived customers for consuming products duringperiod t and ˆ x is the revenue vector of these products.
Revised: Then, by plugging in ˆ π ˆ xy t = 1 , in the right-hand side of constraints (1d), we have ⃗α T x x , where ⃗α T x denotes the average number of arrived customers for consuming products during period t and ˆ x is the revenue vector of these products.

Original: Therefore, (cid:126)α T x x denotes the expected revenuecollected from customers who arrive during period t (where assuming all arrived customers beingserved successfully).
Revised: Therefore, ⃗α T x x denotes the expected revenue collected from customers who arrive during period t (assuming all arrived customers are served successfully).

Original: Consequently, we can conclude that the right-hand side of (1d) is as large as theleft-hand side, because the right-hand side may also compute the revenue collected after the end ofperiod T .
Revised: Consequently, we can conclude that the right-hand side of (1d) is as large as the left-hand side, because the right-hand side may also compute the revenue collected after the end of period T .

Original: Hence, constraints (1d) are also satisﬁed.
Revised: Hence, constraints (1d) are also satisfied.

Original: By the value assignment steps of ˆ a y t and ˆ Y y t ,for y ∈ { 0 , 1 , . . .
Revised: By the value assignment steps of ˆ a y t and ˆ Y y t , for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T , one has that the corresponding LP objective value at solution(ˆ π, ˆ a, ˆ J ) is Y ˆ π .
Revised: , T , one has that the corresponding LP objective value at solution (ˆ π, ˆ a, ˆ J ) is Y ˆ π .

Original: Therefore, we have Y LP ≥ Y π , for any admissible policy π , and thus also for theoptimal state-dependent policy π ∗ .
Revised: Therefore, we have Y LP ≥ Y π , for any admissible policy π , and thus also for the optimal state-dependent policy π ∗ .

Original: By deﬁning ˜ x = argmax x ∈X (cid:126)α T x x , we ﬁnd an optimal solution (˜ π, ˜ a, ˜ Y ) to the LP in (1) as follows.
Revised: By defining ˜ x = argmax x ∈X ⃗α T x x , we find an optimal solution (˜ π, ˜ a, ˜ Y ) to the LP in (1) as follows.

Original: [Equation]
Revised: [Equation]

Original: where y 

Revised: where y

Original: =
Revised: =

Original: [0] J .
Revised: [0] J .

Original: Then, we can compute Y LP = T · (cid:126)α T ˜ x
Revised: Then, we can compute Y LP = T · ⃗α T ˜ x

Original: ˜ x .
Revised: ˜ x .

Original: Note that (cid:126)α T x x is the stationary revenuerate of price x , denoted as r ( (cid:126)α x , x ) .
Revised: Note that ⃗α T x x is the stationary revenue rate of price x , denoted as r ( ⃗α x , x ) .

Original: Q.E.D.
Revised: Q.E.D.

Original: D Technical results in Section 4. 
Revised: D Technical Results in Section 4. Proof of Proposition

Original: Proof of Proposition 2.

Original: The proof is based on the concentration bound for sub-exponential random variables.
Revised: The proof is based on the concentration bound for sub-exponential random variables.

Original: For productj ∈ J off , their demand is off, i.e., α j ( x ) = 0 , j ∈ J off ( x ) .
Revised: For product j ∈ J off , their demand is off, i.e., α j ( x ) = 0 , j ∈ J off ( x ) .

Original: In this proof, we only consider one priceand thus we use α j to represent α j ( x ) for notation simplicity.
Revised: In this proof, we only consider one price and thus we use α j to represent α j ( x ) for notation simplicity.

Original: We can rewrite the right-hand-side as:
Revised: We can rewrite the right-hand-side as:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ( x
Revised: [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original:   . We can derive that for one product j ∈ J on ( x ) :P 
Revised: We can derive that for one product j ∈ J on ( x ) :

Original: [Equation]
Revised: [Equation]

Original: The exponential arrival time intervals are also sub-exponential random variables with parameters( 4 α 2 j , 2 α j ) , denoted by SE ( 4 α 2 j , 2 α j ) , see Lemma 1 in the appendix.
Revised: The exponential arrival time intervals are also sub-exponential random variables with parameters ( 4 α 2 j , 2 α j ) , denoted by SE ( 4 α 2 j , 2 α j ) , see Lemma 1 in the appendix.

Original: Consider the concentration inequality for i.i.d.
Revised: Consider the concentration inequality for i.i.d.

Original: exponential variables ξ i , i = 1 , . . .
Revised: exponential variables ξ i , i = 1 , . . .

Original: n with mean 1 /λ :
Revised: n with mean 1 /λ :

Original: [Equation]
Revised: [Equation]

Original: By the assumption n j ( x ) ≥ 8 log( J 14 T ) , we can compute
Revised: By the assumption n j ( x ) ≥ 8 log( J 14 T ) , we can compute

Original: [Equation]
Revised: [Equation]

Original: and thus we can apply the ﬁrst inequality in (6) to (5).
Revised: and thus we can apply the first inequality in (6) to (5).

Original: Therefore, we can derive
Revised: Therefore, we can derive

Revised: P 

Original: [Equation]
Revised: [Equation]

Original: Then, we can compute a lower bound of (4) as
Revised: Then, we can compute a lower bound of (4) as

Original: [Equation]
Revised: [Equation]

Revised: |J on ( x ) | ≥ (cid:18) 1 −

Revised: J · T(cid:19)

Original:  ≥ 1 − 2 J
Revised: J ≥ 1 − 2 J

Original: J · T 4 = 1 −
Revised: J · T 4 = 1 −

Original: T 4 .
Revised: T 4 .

Original: Q.E.D.
Revised: Q.E.D.

Original: Lemma 1.
Revised: Lemma 1.

Original: (Sub-exponential property, Lemma 3 in Jia et al.
Revised: (Sub-exponential property, Lemma 3 in Jia et al.

Original: If a random variable X follows anexponential distribution with mean 1 /λ , then the random variable X − 1 λ is ( 4 λ 2 , 2 λ ) -sub-exponential.
Revised: If a random variable X follows an exponential distribution with mean 1 /λ , then the random variable X − 1 λ is ( 4 λ 2 , 2 λ ) -sub-exponential.

Original: E Technical results in Section5
Revised: E Technical Results in Section

Original: Proof of Proposition 3.
Revised: Proof of Proposition 3.

Original: In this proof, we analyze the coupling time of nodes layer by layer.
Revised: In this proof, we analyze the coupling time of nodes layer by layer.

Original: We deﬁne events as follows (all starting from empty state).[Equation] the steady [Equation]
Revised: We define events as follows (all starting from empty state).

Original: state

Original: [Equation]
Revised: [Equation]

Original: time

Original: [Equation] steady [Equation] First, we discuss the coupling analysis of an M/M/c queue, which is the basis for completing thisproof.
Revised: First, we discuss the coupling analysis of an M/M/c queue, which is the basis for completing this proof.

Original: With the assumption log > 4 , the probability that an M/M/c queue reaches the steady-statefrom empty state after (log T ) 2 periods of time is bounded by 1 − 2 /T 2 , see Lemma 3 (Proposition4 in Jia et al.
Revised: With the assumption log > 4 , the probability that an M/M/c queue reaches the steady-state from empty state after (log T ) 2 periods of time is bounded by 1 − 2 /T 2 , see Lemma 3 (Proposition 4 in Jia et al.

Original: This results are derived following the law of total probability, the ﬁrst-orderstochastic dominance between busy times of an M/M/c queue, and the concentration inequality forindependent samples.
Revised: These results are derived following the law of total probability, the first-order stochastic dominance between busy times of an M/M/c queue, and the concentration inequality for independent samples.

Original: We denote node i is a prerequisite node of i (cid:48) , i < i (cid:48) if there exists a productrequires to use nodes i and i (cid:48) .
Revised: We denote node i is a prerequisite node of i ′ , i < i ′ if there exists a product requires to use nodes i and i ′ .

Original: Thus, it is easy to show that the prerequisite nodes of a Layer- l nodemust have smaller layer indices.
Revised: Thus, it is easy to show that the prerequisite nodes of a Layer- l node must have smaller layer indices.

Original: In our Jackson Network, node i (cid:48) reaches its steady-state indicates that the arrival process of nodei (cid:48) is a stable Poisson process, and therefore it requires that all the prerequisite nodes also reach thesteady-state.
Revised: In our Jackson Network, node i ′ reaches its steady-state indicates that the arrival process of node i ′ is a stable Poisson process, and therefore it requires that all the prerequisite nodes also reach the steady-state.

Original: In this proof, we analyze the nodes layer by layer.
Revised: In this proof, we analyze the nodes layer by layer.

Original: By treating the nodes in the samelayer together, we can derive that
Revised: By treating the nodes in the same layer together, we can derive that

Original: [Equation]
Revised: [Equation]

Original: In short, we derive a lower bound of this mixing probability by requiring the Layer- l nodes reachthe steady-state within the ﬁrst L × τ periods, conditional on that Layer- l (cid:48) nodes also reach thesteady-state within the ﬁrst l (cid:48) × τ periods for l (cid:48) < l , or equivalently, all prerequisite nodes of Layer- lnodes reach the steady-state within ( l − 1) × τ periods.
Revised: In short, we derive a lower bound of this mixing probability by requiring the Layer- l nodes to reach the steady-state within the first L × τ periods, conditional on that Layer- l ′ nodes also reach the steady-state within the first l ′ × τ periods for l ′ < l , or equivalently, all prerequisite nodes of Layer- l nodes reach the steady-state within ( l − 1) × τ periods.

Original: Layer-1 Nodes.
Revised: Layer-1 Nodes.

Original: In the ﬁrst τ periods, by applying the probability bound for an M/M/c queue, wecan easily conclude that all Layer-1 nodes (the correspondent M/M/c queues) reach the steady-statewith probability 1 − 2 T 3 .
Revised: In the first τ periods, by applying the probability bound for an M/M/c queue, we can easily conclude that all Layer-1 nodes (the correspondent M/M/c queues) reach the steady-state with probability 1 − 2 T 3 .

Original: Recall I 1 is the collection of all Layer-1 nodes and |I 1 | is the cardinality.
Revised: Recall I 1 is the collection of all Layer-1 nodes and |I 1 | is the cardinality.

Original: Therefore we can compute that the probability that all Layer-1 nodes reach the steady-state by ﬁrst τperiods is
Revised: Therefore we can compute that the probability that all Layer-1 nodes reach the steady-state by first τ periods is

Original: [Equation]
Revised: [Equation]

Original: because the queueing evolution of all Layer-1 nodes are independent with each other and thus we cansimplify multiply the probability together, in the ﬁrst equality in the above expression.
Revised: because the queueing evolution of all Layer-1 nodes are independent of each other and thus we can simply multiply the probability together, in the first equality in the above expression.

Original: Layer-2 Nodes.
Revised: Layer-2 Nodes.

Original: We then analyze the Layer-2 nodes.
Revised: We then analyze the Layer-2 nodes.

Original: Consider one arbitrary layer-2 node, denoted byi without loss of generality.
Revised: Consider one arbitrary layer-2 node, denoted by i without loss of generality.

Original: We then focus on the corresponding queue of this node i conditional onthat all Layer-1 nodes have reached steady-state.
Revised: We then focus on the corresponding queue of this node i conditional on that all Layer-1 nodes have reached steady-state.

Original: We ﬁrstly show a virtual queue can reach steadystate within the next τ periods with probability at least 1 − 2 T 2 .
Revised: We firstly show a virtual queue can reach steadystate within the next τ periods with probability at least 1 − 2 T 2 .

Original: Then we show that an alternativequeue of the actual queue takes a shorter time to reach steady-state and thus the actual queue can alsoreach steady-state within the next τ periods with probability at least 1 − 2 T 2 .•
Revised: Then we show that an alternative queue of the actual queue takes a shorter time to reach steady-state and thus the actual queue can also reach steady-state within the next τ periods with probability at least 1 − 2 T 2 .

Original:  Virtual queue.
Revised: • Virtual queue.

Original: We consider a virtual queue which rejects all customers  arrived before τperiods.
Revised: We consider a virtual queue that rejects all customers who arrived before τ periods.

Original: Then by Lemma 3, this virtual queue reaches thesteady-state with
Revised: Then by Lemma 3, this virtual queue reaches the

Original: probability

Original: 1 − 2 T 2 after the next τ periods, i.e., after the ﬁrst 2 τ periods.

Original: • Alternative queue.
Revised: • Alternative queue.

Original: Denote the probability that the number of customers in Node i at the end of period τ is less than N by P i ( N ) , including the customers both using the serveror in the waiting queue of node i .
Revised: Denote the probability that the number of customers in Node i at the end of period τ is less than N by P i ( N ) , including the customers both using the server or in the waiting queue of node i .

Original: The actual queue accepts customers during the ﬁrst τperiods with a changing rate from an initial rate to the stable arrival rate λ i (conditionalon all Layer-1 nodes reach steady-state and thus provides stable Poisson arrivals for nodeswith high layer indices).
Revised: The actual queue accepts customers during the first τ periods with a changing rate from an initial rate to the stable arrival rate λ i (conditional on that all Layer-1 nodes reach steady-state and thus provides stable Poisson arrivals for nodes with high layer indices).

Original: Denote the demand of all products to the Jackson Network by α .
Revised: Denote the demand for all products to the Jackson Network by α .

Original: The stable arrival rate λ i = αp si + α
Revised: The stable arrival rate λ i = αp si + α

Original: (cid:80) i (cid:48) :(
Revised: (cid:80) i ′ :(

Original: i (cid:48) ,i ∈ E ) p si (cid:48) p i (cid:48) i , where αp si is the initial rate, i.e., the demand rate of products that request node i as the ﬁrst service, and αp si (cid:48)
Revised: i ′ ,i ∈ E ) p si ′ p i ′ i , where αp si is the initial rate, i.e., the demand rate of products that request node i as the first service, and αp si ′ p

Original: p i (cid:48)
Revised: i ′

Original: i is thestable demand ﬂowing from prerequisite node i (cid:48) to node i .
Revised: i is the stable demand flowing from prerequisite node i ′ to node i .

Original: Consider an alternative queuewhich accepts customers at the stable rate λ i .
Revised: Consider an alternative queue that accepts customers at the stable rate λ i .

Original: If the queue starts from time 0 , then theprobability that this virtual queue has customers with a size less than N at the end of periodτ is smaller than P i ( N ) .
Revised: If the queue starts from time 0 , then the probability that this virtual queue has customers with a size less than N at the end of period τ is smaller than P i ( N ) .

Original: This is because this alternative queue accepts more customers while the service times are all following a same distribution.
Revised: This is because this alternative queue accepts more customers while the service times are all following the same distribution.

Original: Then we postpone the startingpoint of this alternative queue to a time stamp between 0 and τ such that the probabilitythat this alternative queue has customers with a size less than N at the end of period τis the same as P i ( N ) .
Revised: Then we postpone the starting point of this alternative queue to a time stamp between 0 and τ such that the probability that this alternative queue has customers with a size less than N at the end of period τ is the same as P i ( N ) .

Original: Note that this time stamps is exist, which can be easily proved by the memoryless property of the exponential service time assumptions.
Revised: Note that this timestamps exists, which can be easily proved by the memoryless property of the exponential service time assumptions.

Original: Then we have the following two results.
Revised: Then we have the following two results.

Original: First, this alternative queue has a same evolution path as the actualqueue.
Revised: First, this alternative queue has the same evolution path as the actual queue.

Original: Second, because this alternative queue starts accepting customers earlier than time τ ,and thus it will reach steady -state earlier than the virtual queue.
Revised: Second, because this alternative queue starts accepting customers earlier than time τ , and thus it will reach steady-state earlier than the virtual queue.

Original: We provide an illustration of the actual, alternative, and virtual queues in Figure 3.
Revised: We provide an illustration of the actual, alternative, and virtual queues in Figure 3.

Revised: alternative and the actual queues and thus they are equivalent systems after time τ .

Original: To conclude,we have the actual queue can reach the steady-state with a higher probability after theﬁrst 2 τ periods.
Revised: To conclude, the actual queue can reach the steady-state with a higher probability after the first 2 τ periods.

Original: Similarly, the queueing evolution of all Layer-2 nodes are independent with eachother and thus we can simplify multiply the probability together.
Revised: Similarly, the queueing evolution of all Layer-2 nodes are independent of each other and thus we can simply multiply the probability together.

Original: Therefore, we have that for all Layer-2 nodes
Revised: Therefore, we have that for all Layer-2 nodes

Original: [Equation]
Revised: [Equation]

Original: Layers with Higher Indices.
Revised: Layers with Higher Indices.

Original: Then we apply the same analysis steps of the Layer-2 nodes to nodesin layers with higher indices.
Revised: Then we apply the same analysis steps of the Layer-2 nodes to nodes in layers with higher indices.

Original: Conditional on the events that all nodes in layers with lower indiceshave reached the steady-state by the end of ( l − 1) τ periods, we can compute thatthe probability the all nodes of layer l can reach steady-state in the next τ period with a probability:
Revised: Conditional on the events that all nodes in layers with lower indices have reached the steady-state by the end of ( l − 1) τ periods, we can compute that all nodes of layer l can reach steady-state in the next τ period with a probability:

Original: [Equation]
Revised: [Equation]

Original: With the results in (9), (10), and (11), we are ready to bound the probability in (8):
Revised: With the results in (9), (10), and (11), we are ready to bound the probability in (8):

Original: [Equation]
Revised: [Equation]

Original: We refer to Figure 4 for a direct representation of the above proof sketch, which shows the mixingtime analysis of an illustrative example.
Revised: We refer to Figure 4 for a direct representation of the above proof sketch, which shows the mixing time analysis of an illustrative example.

Original: Q.E.D.
Revised: [Figure]

Original: Proof of Proposition 4.
Revised: Proof of Proposition 4.

Original: Similar to the proof of Proposition 4, we analyze the mixing time of nodes layer by layer.
Revised: Similar to the proof of Proposition 4, we analyze the mixing time of nodes layer by layer.

Original: Before conducting the proof, we introduce an existing work for an M/M/c queue mixing time.
Revised: Before conducting the proof, we introduce an existing work for an M/M/c queue mixing time.

Original: We similarly deﬁne the following events when the Jackson Network starts to operate under a new price starting from the steady-state of the old price:
Revised: We similarly define the following events when the Jackson Network starts to operate under a new price starting from the steady-state of the old price:

Original: [Equation]
Revised: [Equation]

Original: Lemma 4 says that for one M/M/c queue where Assumption 5 holds, then the probability that thequeue reaches a new steady-state from a old steady-state within 2 τ time periods after the price changeis bounded by 1 − 4 /T 2 .
Revised: Lemma 4 says that for one M/M/c queue where Assumption 5 holds, then the probability that the queue reaches a new steady-state from an old steady-state within 2 τ time periods after the price change is bounded by 1 − 4 /T 2 .

Original: To derive this result, the authors decompose possible price changes intothe following two cases.
Revised: To derive this result, the authors decompose possible price changes into the following two cases.

Original: The ﬁrst is when the arrival rate changes from a higher value to a lowervalue and the second is vice versa.
Revised: The first is when the arrival rate changes from a higher value to a lower value and the second is vice versa.

Original: For each case, the authors further consider two sub-cases: the unitservice rate increase or decrease.
Revised: For each case, the authors further consider two sub-cases: the unit service rate increase or decrease.

Original: Consider the Jackson Network system in our problem.
Revised: Consider the Jackson Network system in our problem.

Original: We can bound the probability by:
Revised: We can bound the probability by:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The above inequality is derived following the same steps in (8) and we neglect the details here.
Revised: The above inequality is derived following the same steps in (8) and we neglect the details here.

Original: Assumption 5 allows us to apply Lemma 4 when we analyze one speciﬁc node, i.e., one speciﬁc
Revised: Assumption 5 allows us to apply Lemma 4 when we analyze one specific node, i.e., one specific

Original: M/M/c queue.
Revised: M/M/c queue.

Original: The structure of the following analysis is similar to the structure of the proof of
Revised: The structure of the following analysis is similar to the structure of the proof of

Original: Proposition 3, we ﬁrst analyze Layer-1 node and then the nodes with higher layer indices.
Revised: Proposition 3, we first analyze Layer-1 nodes and then the nodes with higher layer indices.

Original: For nodeswith higher layer indices, we apply the aforementioned proposition to a virtual queue, and then showthis virtual queue takes longer than an alternative system with the actual queue.
Revised: For nodes with higher layer indices, we apply the aforementioned proposition to a virtual queue, and then show this virtual queue takes longer than an alternative system with the actual queue.

Original: Layer-1 Nodes.
Revised: Layer-1 Nodes.

Original: In the ﬁrst 2 τ periods, by applying the probability bound for an M/M/c queue withprice change (Lemma 4), we can easily conclude that all Layer-1 nodes (the correspondent M/M/cqueues) reach the steady-state with probability 1 − 4 T 2 .
Revised: In the first 2 τ periods, by applying the probability bound for an M/M/c queue with price change (Lemma 4), we can easily conclude that all Layer-1 nodes (the correspondent M/M/c queues) reach the steady-state with probability 1 − 4 T 2 .

Original: Consider the probabilistic independency of thecoupling process among Layer-1 nodes, the probability that all Layer-1 nodes reach the steady-stateby ﬁrst 2 τ periods is
Revised: Consider the probabilistic independency of the coupling process among Layer-1 nodes, the probability that all Layer-1 nodes reach the steady-state by the first 2 τ periods is

Original: [Equation]
Revised: [Equation]

Original: Layer-2 Nodes and Nodes with Higher Layer Indices.
Revised: Layer-2 Nodes and Nodes with Higher Layer Indices.

Original: We then analyze nodes with a layer index 2or higher.
Revised: We then analyze nodes with a layer indexor higher.

Original: Consider one arbitrary Layer-l node, denoted by i without loss of generality.
Revised: Consider one arbitrary Layer-l node, denoted by i without loss of generality.

Original: We then focuson the corresponding queue of this node i conditional on that all prerequisites nodes have reachedsteady-state.
Revised: We then focus on the corresponding queue of this node i conditional on that all prerequisites nodes have reached steady-state.

Original: We ﬁrstly show a virtual queue can reach steady-state within the next 2 τ periods withprobability at least 1 − 4 T 2 .
Revised: We firstly show a virtual queue can reach steady-state within the next 2 τ periods with probability at least 1 − 4 T 2 .

Original: Then we show that an alternative queue of the actual queue takes a shortertime to reach steady-state and thus the actual queue can also reach steady-state within the next 2 τperiods with probability at least 1 − 4 T 2 .
Revised: Then we show that an alternative queue of the actual queue takes a shorter time to reach steady-state and thus the actual queue can also reach steady-state within the next 2 τ periods with probability at least 1 − 4 T 2 .

Original: • Virtual queue.
Revised: • Virtual queue.

Original: We consider a virtual queue which accepts customers under the old queueing parameters in the ﬁrst 2( l − 1) τ periods and then accepts customers under the new queueing parameter in the next 2 τ periods.With
Revised: We consider a virtual queue that accepts customers under the old queueing parameters in the first 2( l − 1) τ periods and then accepts customers under the new queueing parameter in the next 2 τ periods.

Original:  Assumption  we can directly apply the results for one M/M/c queue to reach steady-state starting from a steady-state and conclude that this virtual queue reaches the steady-state with probability 1 − 4 T 2 after the next 2 τ periods, i
Revised: With Assumption 5, we can directly apply the results for one M/M/c queue to reach steady-state starting from a steady-state and conclude that this virtual queue reaches the steady-state with probability 1 − 4 T 2 after the next 2 τ periods, i.e., after the first 2 lτ periods.

Original: e.,

Original: after the ﬁrst 2 lτ periods

Original: • Alternative queue.
Revised: • Alternative queue.

Original: Denote the probability that in Node i at the end of 2( l − 1) τ periods, the number of old customers is less than N o and the number of new customers is less than N n by P i ( N o , N n ) , including the customers both using the server or in the waiting queue of node i .
Revised: Denote the probability that in Node i at the end of 2( l − 1) τ periods, the number of old customers is less than N o and the number of new customers is less than N n by P i ( N o , N n ) , including the customers both using the server or in the waiting queue of node i .

Original: Consider an alternative queue which starts accepting new customers under the new queueingsystem parameter from time 0 and denote the corresponding probability for having lessthan N o number of old customers and less than N n new customers at the end of 2( l − 1) τperiods by P(cid:48)
Revised: Consider an alternative queue that starts accepting new customers under the new queueing system parameter from time 0 and denote the corresponding probability for having less than N o number of old customers and less than N n new customers at the end of 2( l − 1) τ periods by P

Original: i ( N o , N n ) .
Revised: i ( N o , N n ) .

Original: By the analysis in Proposition 3, we know there is a mismatchbetween P i ( N o , N n ) .
Revised: By the analysis in Proposition 3, we know there is a mismatch between P i ( N o , N n ) .

Original: Then we postpone the starting point of this alternative queue to atime stamp between 0 and 2( L − 1) τ such that P (cid:48) i ( N o , N n ) 
Revised: Then we postpone the starting point of this alternative queue to a time stamp between 0 and 2( L − 1) τ such that P ′ i ( N o , N n ) =

Original: = P i ( N o , N n ) .
Revised: P i ( N o , N n ) .

Original: Note that this time stamps is always exist, which can be easily proved by the memoryless property of theexponential service time assumptions because we can always ﬁnd this stamp backwardsfrom any customer state at time 2( l − 1) τ .
Revised: Note that these timestamps always exist, which can be easily proved by the memoryless property of the exponential service time assumptions because we can always find this stamp backward from any customer state at time 2( l − 1) τ .

Original: Then we have the following two results.
Revised: Then we have the following two results.

Original: First,this alternative queue has a same evolution path as the actual queue.
Revised: First, this alternative queue has the same evolution path as the actual queue.

Original: Second, because thisalternative queue starts accepting new customers earlier than time 2( l − 1) τ , and thus it willreach steady-state earlier than the virtual queue.
Revised: Second, because this alternative queue starts accepting new customers earlier than time 2( l − 1) τ , and thus it will reach steady-state earlier than the virtual queue.

Original: To conclude,we have the actual queue can reach the steady-state with a higher probability after theﬁrst 2 τ periods.
Revised: To conclude, the actual queue can reach the steady-state with a higher probability after the first 2 τ periods.

Original: Therefore , conditional on the events that all nodes in layers with lower indices havereached the steady-state by the end of 2( l − 1) τ periods, we can compute that the probability the allnodes of layer l can reach steady-state in the next 2 τ period with a probability: 
Revised: Therefore, conditional on the events that all nodes in layers with lower indices have reached the steady-state by the end of 2( l − 1) τ periods, we can compute that all nodes of layer l can reach steady-state in the next 2 τ period with a probability: [Equation]

Original: [Equation]
Revised: [Equation]

Original: With the results in (13) and (14), we are ready to bound the probability in (12):
Revised: With the results in (13) and (14), we are ready to bound the probability in (12):

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: We bound the relaxed regret during the Warm-up and Learning Phases separately.
Revised: We bound the relaxed regret during the Warm-up and Learning Phases separately.

Original: We furtherdecompose the relaxed regret during the Learning Phase into two parts.
Revised: We further decompose the relaxed regret during the Learning Phase into two parts.

Original: The ﬁrst part is the regret ofsuboptimality, which is induced by not offering the state-independent optimal price in certain timeperiods.
Revised: The first part is the regret of suboptimality, which is induced by not offering the state-independent optimal price in certain time periods.

Original: We bound the loss due to suboptimality with the help of Proposition 2.
Revised: We bound the loss due to suboptimality with the help of Proposition 2.

Original: The second part isthe regret of nonstationarity, which represents the revenue loss induced by changing offered prices.
Revised: The second part is the regret of nonstationarity, which represents the revenue loss induced by changing offered prices.

Original: We bound the loss due to nonstationarity with the help of Propositions 3 and 4.
Revised: We bound the loss due to nonstationarity with the help of Propositions 3 and 4.

Original: The structure of thisproof is similar to that of Theorem 1 in Jia et al.
Revised: The structure of this proof is similar to that of Theorem 1 in Jia et al.

Original: (2020), where the authors consider a single reusableproduct setting.
Revised: (2020), where the authors consider a single reusable product setting.

Original: Regret of the Warm-up Phase.
Revised: Regret of the Warm-up Phase.

Original: The difference between Y LPWarm-up and Y RNRM-UCBWarm-up is at most linearon the length of the Warm-up Phase, which is O ( X log( J 1 / 4 T )) .
Revised: The difference between Y LPWarm-up and Y RNRM-UCBWarm-up is at most linear on the length of the Warm-up Phase, which is O ( X log( J 1 / 4 T )) .

Original: Therefore we can directly have
Revised: Therefore we can directly have

Original: Regret
Revised: Regret

Original: Warm-
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Regret of nonstationarity during the Learning Phase.
Revised: Regret of nonstationarity during the Learning Phase.

Original: We denote the offered price of batch m in
Revised: We denote the offered price of batch m in

Original: RNRM-UCB as x m .
Revised: RNRM-UCB as x m .

Original: Therefore, the expected revenue collected in batch m is
Revised: Therefore, the expected revenue collected in batch m is

Original: [Equation]
Revised: [Equation]

Original: where Y RNRM-UCB m, 0 ∼ 2 Lτ ( x m ) denotes the expected revenue collected in the ﬁrst 2 Lτ periods and
Revised: where Y RNRM-UCB m, 0 ∼ 2 Lτ ( x m ) denotes the expected revenue collected in the first 2 Lτ periods and

Original: Y RNRM-UCB m, 2 Lτ ∼
Revised: Y RNRM-UCB m, 2 Lτ ∼

Original: I m Lτ ( x m ) denotes the expected revenue collected from (2 Lτ + 1) th period to the endof batch m .
Revised: I m Lτ ( x m ) denotes the expected revenue collected from (2 Lτ + 1) th period to the end of batch m .

Original: We deﬁne event C 2 Lτ as that the network service system reaches the steady-state underthe new price within the ﬁrst 2 Lτ periods and deﬁne ¯ C 2 Lτ as the complementary event of C 2 Lτ .
Revised: We define event C 2 Lτ as that the network service system reaches the steady-state under the new price within the first 2 Lτ periods and define ¯ C 2 Lτ as the complementary event of C 2 Lτ .

Original: By
Revised: By

Original: Propositions 3 and 4, we have P [ C 2 Lτ ] ≥ 1 − 4 I/T 2 .
Revised: Propositions 3 and 4, we have P [ C 2 Lτ ] ≥ 1 − 4 I/T 2 .

Original: Consequently, we deﬁne Y RNRM-UCB m, C 2 Lτ ( x m ) tobe the expected value of Y RNRM-UCB m, 2 Lτ ∼
Revised: Consequently, we define Y RNRM-UCB m, C 2 Lτ ( x m ) to be the expected value of Y RNRM-UCB m, 2 Lτ ∼

Original: I m Lτ ( x m ) conditional on C 2 Lτ and Y RNRM-UCB m, ¯ C 2 Lτ ( x m ) similarly.
Revised: I m Lτ ( x m ) conditional on C 2 Lτ and Y RNRM-UCB m, ¯ C 2 Lτ ( x m ) similarly.

Original: Asa result, we can express the expected revenue during batch m as:
Revised: As a result, we can express the expected revenue during batch m as:

Original: [Equation]
Revised: [Equation]

Original: Regret of suboptimality during the Learning Phase.
Revised: Regret of suboptimality during the Learning Phase.

Original: Recall the upper and lower conﬁdence bounds of the revenue rate deﬁned in Deﬁnition 2.
Revised: Recall the upper and lower confidence bounds of the revenue rate defined in Definition 2.

Original: It is clear that
Revised: It is clear that

Original: [Equation]
Revised: [Equation]

Original: For batch m , we can compute the difference in the revenue rates between the optimal price andselected price x m as 
Revised: For batch m [Equation] we can compute the difference in the revenue rates between the optimal price and selected price x m as [Equation]

Revised: 17a

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds because U m ( x m ) ≥ U m (˜ x ) by the algorithm construction.
Revised: where the last inequality holds because U m ( x m ) ≥ U m (˜ x ) by the algorithm construction.

Original: Then the cumulative regret during the Learning Phase is
Revised: Then the cumulative regret during the Learning Phase is

Original: [Equation]
Revised: [Equation]

Original: By (17), we have
Revised: By (17) [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst analyze the middle term of (19).
Revised: We first analyze the middle term of (19).

Original: Let M x denote the batches choosing price x and T x thenumber of periods choosing price x , i.e., T x = (cid:80) m
Revised: Let M x denote the batches choosing price x and T x the number of periods choosing price x , i.e., T x = (cid:80) m

Original: ∈M x
Revised: ∈M x

Original: I m Lτ .
Revised: I m Lτ .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where (20c) by Lemma 2, (20e) is derived by the number of the batches, and (20f) follows Jensen’sinequality.
Revised: where (20c) by Lemma 2, (20e) is derived by the number of the batches, and (20f) follows Jensen’s inequality.

Original: The number of batches is log2 (cid:16) 1 + T
Revised: The number of batches is log

Original: − 8 X log ( J 1 / 4 T ) 2 L log( T )(cid:17)
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: O (cid:0) log( T ) (cid:1) . Below we analyze the ﬁrst term of (19), and the same logic applies to the third term, of which weomit the details here.
Revised: Below we analyze the first term of (19), and the same logic applies to the third term, of which we omit the details here.

Original: [Equation]
Revised: [Equation]

Original: 21f) where (21e) is because U m (˜ x ) > 0 by Deﬁnition 2, and (21f) follows Proposition 2.
Revised: where (21e) is because U m (˜ x ) > 0 by Definition 2, and (21f) follows Proposition 2.

Original: Given that O (cid:0) n Mj ( x ) (cid:1) =
Revised: Given that O (cid:0) n Mj ( x ) (cid:1) =

Original: O ( T x ) , for any product j ∈ J on ( x ) , we have (cid:80) Mm =
Revised: O ( T x ) , for any product j ∈ J on ( x ) , we have (cid:80) Mm =

Original: 1 O ( Lτ ) =
Revised: 1 O ( Lτ ) =

Original: O (cid:0) L (log( T )) 3 (cid:1) .
Revised: O (cid:0) L (log( T )) 3 (cid:1) .

Original: Combining the three terms together,
Revised: Combining the three terms together,

Revised: Regret

Original: [Equation]
Revised: [Equation]

Original: Total Regret of RNRM-UCB algorithm.
Revised: Total Regret of RNRM-UCB algorithm.

Original: By adding up the relaxed regret during the Warm-up and722 Learning Phases, we have the regret of RNRM-UCB algorithm as:
Revised: By adding up the relaxed regret during the Warm-up and Learning Phases, we have the regret of RNRM-UCB algorithm as (typically we always have

Original: [Equation]
Revised: [Equation]

Original: Q.
Revised: Q.E

Original: [Equation]
Revised: [Equation]

Original: .D
Revised: D.

Original: [Equation] Proof of Theorem 
Revised: Proof of Theorem 2.

Original: The analysis here is similar to that in the proof of Theorem 1.
Revised: The analysis here is similar to that in the proof of Theorem 1.

Original: We ﬁrst bound the Bayesian regret ofthe Warm-up Phase and then for the Learning Phase.
Revised: We first bound the Bayesian regret of the Warm-up Phase and then for the Learning Phase.

Original: Bayesian regret in the Warm-up Phase.
Revised: Bayesian regret in the Warm-up Phase.

Original: Similarly, we can bound the Bayesian regret in the
Revised: Similarly, we can bound the Bayesian regret in the

Original: Warm-up Phase as
Revised: Warm-up Phase as

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Bayesian regret in the Learning Phase.
Revised: Bayesian regret in the Learning Phase.

Original: In batch m , the chosen price x m and the state-independentoptimal price ˜ x are identically distributed conditional on F m , which indicates that E (cid:2)
Revised: In batch m , the chosen price x m and the state-independent optimal price ˜ x are identically distributed conditional on F m , which indicates that E (cid:2)

Original: U m ( x m ) −
Revised: U m ( x m ) −

Original: U m (˜ x ) |F m (cid:3) = 
 
Revised: U m (˜ x ) |F m (cid:3) = 
 (e [Equation]

Original: (e.g., see the proof of Lemma 3.7 in Slivkins (2019)).
Revised: g., see the proof of Lemma 3.7 in Slivkins (2019)).

Original: Thus, the Bayesian regret in batch m is:
Revised: Thus, the Bayesian regret in batch m is:

Revised: BR m

Original: [Equation]
Revised: [Equation]

Original:  where (25g) is derived following the analysis of nonstationarity in (16).
Revised: 25a [Equation] where (25g) is derived following the analysis of nonstationarity in (16).

Original: Derived from Proposition 2, for some γ > 0 , two properties hold (see, e
Revised: Derived from Proposition 2, for some γ > 0 , two properties hold (see, e

Original: [Equation]
Revised: [Equation]

Original: g., Slivkins 2019):
Revised: g., Slivkins 2019):

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Following these two properties of the stationary revenue rate in (26)
Revised: Following these two properties of the stationary revenue rate in (26)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Revised: E

Original: [Equation]
Revised: [Equation]

Original: and similarly,
Revised: and similarly,

Original: [Equation]
Revised: [Equation]

Original: Summing over all batches m, ∀ m = 1 , . .
Revised: Summing over all batches m, ∀ m = 1 , . .

Original: , M , we have the cumulative relaxed Bayesian regret bythe end of the Learning Phase as:
Revised: , M , we have the cumulative relaxed Bayesian regret by the end of the Learning Phase as:

Original: [Equation]
Revised: [Equation]

Original: where (29e) is by Lemma 2 and (29f) is by Jensen’s inequality.
Revised: where (29f) is by Lemma 2 and (29g) is by Jensen’s inequality.

Original: Regret of RNRM-TS algorithm.
Revised: Regret of RNRM-TS algorithm.

Original: By adding up the relaxed regret during the Warm-up and Learning
Revised: By adding up the relaxed regret during the Warm-up and Learning

Original: Phases, we have the regret of RNRM-TS algorithm as:
Revised: Phases, we have the regret of RNRM-TS algorithm as:

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: For any price x ∈ X ,(cid:88)
Revised: For any price x ∈ X ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: Plugging the deﬁnition of Rad m ( x ) , ∀ m = 1 , . .
Revised: Plugging the definition of Rad m ( x ) , ∀ m = 1 , . .

Original: , M , one has:
Revised: , M , one has:

Original: [Equation]
Revised: [Equation]

Original: 32g) where and (32b) is by n mj ( x ) =
Revised: where and (32b) is by n mj ( x ) =

Original: O ( I m Lτ ) , ∀ j ∈ J on ( x ) and (32e) is by Jensen’s inequality.
Revised: O ( I m Lτ ) , ∀ j ∈ J on ( x ) and (32e) is by Jensen’s inequality.

Original: Lemma 3.
Revised: Lemma 3.

Original: (Coupling probability of M/M/c queue starting from empty state, Proposition 4 in
Revised: (Coupling probability of M/M/c queue starting from empty state, Proposition 4 in

Original: Jia et al. 2020)
Revised: Jia et al. 2020)

Original: For τ = (log T ) 2 , if an M/M/c queue starts from empty state at time 0 , then theprobability that the system reaches the steady-state within τ is bounded by 1 − 2 T 2 .
Revised: For τ = (log T ) 2 , if an M/M/c queue starts from empty state at time 0 , then the probability that the system reaches the steady-state within τ is bounded by 1 − 2 T 2 .

Original: Lemma 4.
Revised: Lemma 4.

Original: (Coupling probability of M/M/c queue when price changes, Proposition 5 in Jia et al.2020)
Revised: (Coupling probability of M/M/c queue when price changes, Proposition 5 in Jia et al.

Original: For τ = (log T ) 2 , if an M/M/c queue starts from steady-state and changes system parameter attime 0 , then the probability that the system reaches the steady-state within 2 τ is bounded by 1 − 4 T 2 .
Revised: For τ = (log T ) 2 , if an M/M/c queue starts from steady-state and changes system parameter at time 0 , then the probability that the system reaches the steady-state within 2 τ is bounded by 1 − 4 T 2 .

Original: F Service Duration Dependent Payment Model
Revised: F Service Duration Dependent Payment Model

Original: We consider a new variant payment model where the customer pays the product’s per-unit-time pricemultiplied by its total service duration.
Revised: We consider a new variant payment model where the customer pays the product’s per-unit-time price multiplied by its total service duration.

Original: The ﬁrm then needs to decide the per-unit-time price for eachproduct j ∈ J .
Revised: The firm then needs to decide the per-unit-time price for each product j ∈ J .

Original: Under this payment setting, we modify the deﬁnition of the steady-state revenue rate,derive a new concentration bound of historical observations and actual system parameters, and thencorrespondingly develop  new UCB of the revenue rate of each price.
Revised: Under this payment setting, we modify the definition of the steady-state revenue rate, derive a new concentration bound of historical observations and actual system parameters, and then correspondingly develop a new UCB for the revenue rate of each price.

Original: With those modiﬁed deﬁnitions,we present a new RNRM-UCB variant and show that it achieves a same regret bound.
Revised: With those modified definitions, we present a new RNRM-UCB variant and show that it achieves the same regret bound.

Original: Data collection.
Revised: Data collection.

Original: Similar to n j , ˆ d jk ( x ) , and ¯ d j ( x ) , we also collect service time observations overresources.
Revised: Similar to n j , ˆ d jk ( x ) , and ¯ d j ( x ) , we also collect service time observations over resources.

Original: We use n ri ( x ) to denote the number of customers who successfully consume resource iunder price x .
Revised: We use n ri ( x ) to denote the number of customers who successfully consume resource i under price x .

Original: We use ˆ g ik ( x ) to denote the service time of customer k over resource i under the pricex .
Revised: We use ˆ g ik ( x ) to denote the service time of customer k over resource i under the price x .

Original: We use ¯ g i ( x ) to denote the empirical mean of the service time and ¯ g i ( x ) =
Revised: We use ¯ g i ( x ) to denote the empirical mean of the service time and ¯ g i ( x ) =

Original: (cid:80) n ri ( x ) k
Revised: (cid:80) n ri ( x ) k

Original: =

Revised: =

Original: ˆ g ik ( x ) /n ri ( x ) ,for
Revised: ˆ g ik ( x ) /n ri ( x ) , for

Original: i ∈
Revised: i ∈

Original: I .
Revised: I .

Original: We further assume that there is a known upper bound r max = max x,i,j α j /µ i , n min j ( x )
Revised: We further assume that there is a known upper bound r max = max x,i,j α j /µ i , n min j ( x )

Original: =min { n j ( x ) , min i : a ji =1 n si ( x ) } .
Revised: = min { n j ( x ) , min i : a ji =1 n si ( x ) } .

Original: Revenue rate.
Revised: Revenue rate.

Original: Under this duration-dependent payment model, we offer unit price x ∈ R J to products
Revised: Under this duration-dependent payment model, we offer unit price x ∈ R J to products

Original: J .
Revised: J .

Original: Customer k who requests to use product j pays the per-unit-time price multiplied by the totalservice duration, i.e., x j (cid:80) i : a ji
Revised: Customer k who requests to use product j pays the per-unit-time price multiplied by the total service duration, i.e., x j (cid:80) i : a ji

Original: =1 ˆ g ik ( x ) , to the ﬁrm.
Revised: =1 ˆ g ik ( x ) , to the firm.

Original: Different than the lump sum payment model,the steady-state revenue rate for this setting is r v ( x ) =
Revised: Different than the lump sum payment model, the steady-state revenue rate for this setting is r v ( x ) =

Original: (cid:80) j ∈J on ( x ) α j x j (cid:80) i : a ji =1 

Revised: (cid:80) j ∈J on ( x ) α j x j (cid:80) i : a ji =1

Original: /µ i ( x ) .
Revised: /µ i ( x ) .

Original: Proposition 5.
Revised: Proposition 5.

Original: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( L 14 J 14 T ) for j ∈ J on ( x ) and the number of service time observations for each resource n si ( x ) ≥ 8 log( L 1 4 J 1 4 T ) ∀ i ∈ { i ∈ I : (cid:80) j ∈J on ( x ) a ji ≥ 1 } , then we have
Revised: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( L 14 J 14 T ) for j ∈ J on ( x ) and the number of service time observations for each resource n si ( x ) ≥ 8 log( L 1 4 J 1 4 T ) ∀ i ∈ { i ∈ I : (cid:80) j ∈J on ( x ) a ji ≥ 1 } , then we have

Original: P
Revised: P

Original: [Equation]

Original: (cid:113)

Original: [Equation]
Revised: [Equation]

Original: where r maxis a known upper bound with r max = max x,i,j α j  /µ i  ,  n min j ( x ) =min { n j ( x ) , min i : a ji =1 n si ( x ) } , and L is the number of layers.
Revised: where r max = max x ,i,j α j ( x ) /µ i ( x ) , and n min j ( x ) = min { n j ( x ) , min i : a ji =1 n si ( x ) } , and L is the number of layers.

Original: Proof of Proposition5.
Revised: Proof of Proposition

Revised: [Table]

Original: The above probability can be bounded by the product of the following probabilities from below.
Revised: The above probability can be bounded by the product of the following probabilities from below.

Original: Firstly we can bound that for i, j with a ji = 1 :
Revised: Firstly we can bound that for i

Original: [Table]
Revised: [Table]

Original: 
Revised: j with a ji = 1 :

Original: [Table]
Revised: [Table]

Original: The ﬁrst and second probability in the above equation can be bounded with the same technique as that we used in (7) with Lemma 1 and thus we have
Revised: The first and second probabilities in the above equation can be bounded with the same technique as that we used in (7) with Lemma 1 and thus we have

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: ≤ [Equation] [Equation] [Equation]
Revised: [Table]

Original: x

Original: [Equation]
Revised: [Equation]

Original: x
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: LJT
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: P [Equation] 
Revised: P 

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have the above probability can be bounded by 
Revised: Therefore, we have the above probability can be bounded by (cid:18)

Original: [Equation]
Revised: [Equation]

Original: Then, for product j ∈ J on ( x ) , we can further boundP 
Revised: Then, for product j ∈ J on ( x ) , we can further bound

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: i : 
Revised: i : a

Original: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: P 
Revised: P 

Original: [Equation]
Revised: [Equation]

Original: µ

Original: [Equation] ≤ ( [Equation] because (cid:80) i ∈I a ji ≤ L ∀ j ∈ J .
Revised: because (cid:80) i ∈I a ji ≤ L ∀ j ∈ J .

Original: Now we are ready to derive the concentration bound.
Revised: Now we are ready to derive the concentration bound.

Original: [Table]
Revised: [Table]

Original: According to the above proposition
Revised: According to the above proposition

Original: [Table]
Revised: [Table]

Original: we can deﬁne the conﬁdence radius as follows.
Revised: we can define the confidence radius as follows.

Original: (We use the superscript v to denote this duration-dependent payment model.)
Revised: (We use the superscript v to denote this duration-dependent payment model.)

Original: [Equation]
Revised: [Equation]

Original: Therefore, the upper and lower conﬁdence bound of the revenue rate associated with each price is
Revised: Therefore, the upper and lower confidence bound of the revenue rate associated with each price is

Original: U vt ( x

Original: [Equation]
Revised: [Equation]

Original: =
Revised: vt

Original: [Equation]
Revised: [Equation]

Original: Algorithm design.
Revised: Algorithm design.

Original: The algorithmic framework remains unchanged except for two modiﬁcations.(i)
Revised: The algorithmic framework remains unchanged except for two modifications. (i)

Original: In Step 6, the algorithm also requires the number of service time observations to be at least8 log( L 14 J 14 T ) .
Revised: In Step 6, the algorithm also requires the number of service time observations to be at least 8 log( L 14 J 14 T ) .

Original: (ii)
Revised: (ii)

Original: In Steps 13 and 16, the algorithm uses the new UCB deﬁnition.
Revised: In Steps 13 and 16, the algorithm uses the new UCB definition.

Original: Algorithm 3 RNRM-UCB (Duration-Dependent Payment Model.)

Original: 1: Input

Original: T

Original: X

Original: service

Original: turn

Original: off price.

Original: Choose argmax x ∈ U vm − 1 ( x ) .

Original: Offer x m in batch m , i.e., for I m Lτ periods.
Revised: Offer x m in batch m , i.e., for I m Lτ periods.

Original: Record inter-arrival time ˆ d jk ( x m ) for arriving customers of product j ∈ J on ( x m ) .
Revised: Record inter-arrival time ˆ d jk ( x m ) for arriving customers of product j ∈ J on ( x m ) .

Original: Compute and update U vm ( x m ) .
Revised: Compute and update U vm ( x m ) .

Original: 17: end for
Revised: 17: end for

Original: Regret analysis.
Revised: Regret analysis.

Original: We change the constraints (1d) in the LP benchmark to
Revised: We change the constraints (1d) in the LP benchmark to

Original: [Equation]
Revised: [Equation]

Original: Under this duration-dependent payment setting, we can follow the same analysis technique to showthat this LP model also gives a valid upper bound of the optimal revenue.
Revised: Under this duration-dependent payment setting, we can follow the same analysis technique to show that this LP model also gives a valid upper bound of the optimal revenue.

Original: The results of Propositions1, 3, and 4 and Theorem 1 remain the same for this duration-dependent payment model.
Revised: The results of Propositions 1, 3, and 4 and Theorem 1 remain the same for this duration-dependent payment model.