Original: Online Learning and Pricing for Network Revenue Management with Reusable Resources
Revised: Online Learning and Pricing for Network Revenue Management with Reusable Resources

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: We consider a price-based network revenue management problem with multipleproducts and multiple reusable resources.
Revised: We consider a price-based network revenue management problem with multipleproducts and multiple reusable resources.

Original: Each randomly arriving customer requests a product (service) that needs to occupy a sequence of reusable resources(servers).
Revised: Each randomly arriving customer requests a product (service) that needs to occupy a sequence of reusable resources(servers).

Original: We adopt an incomplete information setting where the ﬁrm does notknow the price-demand function for each product and the goal is to dynamically setprices of all products to maximize the total expected revenue of serving customers.
Revised: We adopt an incomplete information setting where the ﬁrm does notknow the price-demand function for each product and the goal is to dynamically setprices of all products to maximize the total expected revenue of serving customers.

Original: We propose novel batched bandit learning algorithms for ﬁnding near-optimalpricing policies, and show that they admit a near-optimal cumulative regret boundof
Revised: We propose novel batched bandit learning algorithms for ﬁnding near-optimalpricing policies, and show that they admit a near-optimal cumulative regret boundof

Original: ˜ O
Revised: ˜ O

Original: ( J √ XT ) , where J , X , and T are the numbers of products, candidate prices,and service periods, respectively.
Revised: ( J √ XT ) , where J , X , and T are the numbers of products, candidate prices,and service periods, respectively.

Original: As part of our regret analysis, we develop theﬁrst ﬁnite-time mixing time analysis of an open network queueing system (i.e., the
Revised: As part of our regret analysis, we develop theﬁrst ﬁnite-time mixing time analysis of an open network queueing system (i.e., the

Original: Jackson Network), which could be of independent interest.
Revised: Jackson Network), which could be of independent interest.

Original: Our numerical studiesshow very promising results of the proposed approaches.
Revised: Our numerical studiesshow very promising results of the proposed approaches.

Original:  Introduction
Revised:  Introduction

Original: Network revenue management (NRM) (Gallego and Van Ryzin 1997) has attained remarkable resultsin a wide range of applications including airline management, electricity pricing, retailing, healthcare,and leisure (see, e.g., Williamson 1992, Talluri and Van Ryzin 2006, Özer et al. 2012).
Revised: Network revenue management (NRM) (Gallego and Van Ryzin 1997) has attained remarkable resultsin a wide range of applications including airline management, electricity pricing, retailing, healthcare,and leisure (see, e.g., Williamson 1992, Talluri and Van Ryzin 2006, Özer et al. 2012).

Original: The bulkof existing literature focuses primarily on canonical settings with perishable resources in whichthe product once sold is permanently removed from inventory (Besbes and Zeevi 2012, Ferreiraet al.
Revised: The bulkof existing literature focuses primarily on canonical settings with perishable resources in whichthe product once sold is permanently removed from inventory (Besbes and Zeevi 2012, Ferreiraet al.

Original: 2018, Chen and Shi 2019).
Revised: 2018, Chen and Shi 2019).

Original: Little progress has been made for NRM with reusable resources,mainly due to the complexity of analyzing the underlying stochastic systems (typically manifestedas stochastic queueing networks).
Revised: Little progress has been made for NRM with reusable resources,mainly due to the complexity of analyzing the underlying stochastic systems (typically manifestedas stochastic queueing networks).

Original: In this paper, we develop easy-to-implement online learning andpricing policies for NRM with reusable resources shared by multiple products and analyze theirtheoretical and empirical performances.
Revised: In this paper, we develop easy-to-implement online learning andpricing policies for NRM with reusable resources shared by multiple products and analyze theirtheoretical and empirical performances.

Original: 1.1 Problem description
Revised: 1.1 Problem description

Original: A service-providing ﬁrm is endowed with multiple types of reusable resources, each having a ﬁxedcapacity.
Revised: A service-providing ﬁrm is endowed with multiple types of reusable resources, each having a ﬁxedcapacity.

Original: The ﬁrm sells multiple types of products (services), each requiring the use of a sequenceof different types of reusable resources (servers).
Revised: The ﬁrm sells multiple types of products (services), each requiring the use of a sequenceof different types of reusable resources (servers).

Original: (See, e.g., Figure 2.)
Revised: (See, e.g., Figure 2.)

Original: Customers randomly arriveaccording to a price-dependent multivariate Poisson process to request products.
Revised: Customers randomly arriveaccording to a price-dependent multivariate Poisson process to request products.

Original: Upon requesting oneparticular product, a customer enters the system by going through a sequence of product-speciﬁedreusable resources one by one.
Revised: Upon requesting oneparticular product, a customer enters the system by going through a sequence of product-speciﬁedreusable resources one by one.

Original: During the service process, if there is no available next type ofresource upon ﬁnishing using the previous type in the sequence, she will join the queue in front ofthis next type of resource and wait until the next available unit.
Revised: During the service process, if there is no available next type ofresource upon ﬁnishing using the previous type in the sequence, she will join the queue in front ofthis next type of resource and wait until the next available unit.

Original: Both the mean demand rate of eachproduct and the average usage time of each resource depend on the posted price vector.
Revised: Both the mean demand rate of eachproduct and the average usage time of each resource depend on the posted price vector.

Original: However,such dependence is unknown to the ﬁrm a priori .
Revised: However,such dependence is unknown to the ﬁrm a priori .

Original: We note that under a ﬁxed price vector, the servicesystem can be reduced to the celebrated Jackson Network (Jackson 1963, 1957).
Revised: We note that under a ﬁxed price vector, the servicesystem can be reduced to the celebrated Jackson Network (Jackson 1963, 1957).

Original: The main objectiveof this paper is to design a “learning-while-doing” algorithm to dynamically post prices of differentproducts, in order to maximize the expected total revenue collected over a ﬁnite time horizon
Revised: The main objectiveof this paper is to design a “learning-while-doing” algorithm to dynamically post prices of differentproducts, in order to maximize the expected total revenue collected over a ﬁnite time horizon

Original: It is evident that NRM with reusable resources ﬁnds many important real-world applications, includingcloud computing, hospitality industry, project management, and leisure (Lei and Jasin 2020).
Revised: It is evident that NRM with reusable resources ﬁnds many important real-world applications, includingcloud computing, hospitality industry, project management, and leisure (Lei and Jasin 2020).

Original: Forexample, ﬁrms such as Amazon and Microsoft offer diverse cloud computing services, each requiringa different set of computational resources (virtual machines) for a certain duration of time to ﬁnishtheir computational jobs (see, e.g., Kaewpuang et al.
Revised: Forexample, ﬁrms such as Amazon and Microsoft offer diverse cloud computing services, each requiringa different set of computational resources (virtual machines) for a certain duration of time to ﬁnishtheir computational jobs (see, e.g., Kaewpuang et al.

Original: 2013, Püschel et al. 2015).
Revised: 2013, Püschel et al. 2015).

Original: 1.2 Main results and contributions
Revised: 1.2 Main results and contributions

Original: We propose new batched bandit algorithms for NRM with reusable resources (RNRM).
Revised: We propose new batched bandit algorithms for NRM with reusable resources (RNRM).

Original: The performance measure is regret, which is the revenue difference between our learning algorithms anda clairvoyant optimal pricing policy under full information.
Revised: The performance measure is regret, which is the revenue difference between our learning algorithms and aclairvoyant optimal pricing policy under full information.

Original: We use J, X, T to denote the numbersof products, candidate prices, and service periods, respectively.
Revised: We use J, X, T to denote the numbers ofproducts, candidate prices, and service periods, respectively.

Original: We prove in Theorems 1 and 2 thatthe cumulative regret is ˜ O
Revised: We prove in Theorems 1 and 2 that thecumulative regret is ˜ O

Original: ( J √ XT ) , which matches the lower bound along the dimensions of Xand T up to a logarithmic factor.
Revised: ( J √ XT ) , which matches the lower bound along the dimensions of X and Tup to a logarithmic factor.

Original: It remains open whether one could improve the dependence on Jsince all prior studies (e.g., Miao and Wang 2021) show polynomial dependence on J at best (albeitfor a slightly different setting).
Revised: It remains open whether one could improve the dependence on J sinceall prior studies (e.g., Miao and Wang 2021) show polynomial dependence on J at best (albeit for anon-reusable setting).

Original: We test diverse instances to show numerical performance of ourapproaches.
Revised: We test diverse instances to show numerical performance of our approaches.

Original: The main contribution of our work is three-fold:( Modeling
Revised: The main contribution of our work is three-fold:(Modeling)

Original: ) We are the ﬁrst to consider the joint learning and optimization problem in NRM withreusable resources.
Revised: We are the ﬁrst to consider the joint learning and optimization problem in NRM withreusable resources.

Original: Prior literature on reusable resources only focused on single-product settings(see Jia et al.
Revised: Prior literature on reusable resources only focused on single-product settings(see Jia et al.

Original: (2020, 2022) and references therein).
Revised: (2020, 2022) and references therein).

Original: It is considerably more challenging to model andanalyze the network setting where multiple products share multiple resources (with ﬁnite capacities),mainly due to the sequential use of resources (to ﬁnish up a service) and the evolving nature ofavailable capacities in real-time.
Revised: It is considerably more challenging to model andanalyze the network setting where multiple products share multiple resources (with ﬁnite capacities),mainly due to the sequential use of resources (to ﬁnish up a service) and the evolving nature ofavailable capacities in real-time.

Original: We establish a connection between our model and the celebrated
Revised: We establish a connection between our model and the celebrated

Original: Jackson Network (Jackson 1963, 1957).(Algorithm)
Revised: Jackson Network (Jackson 1963, 1957).(Algorithm)

Original: We ﬁrst develop a linear programming (LP) upper bound on the optimal objective valuegiven full information.
Revised: We ﬁrst develop a linear programming (LP) upper bound on the optimal objective valuegiven full information.

Original: The LP model provides a static pricing control policy.
Revised: The LP model provides a static pricing control policy.

Original: Therefore, we cancast our RNRM as a multi-armed bandits (MAB) problem where we select among candidate prices(analogs of arms) to minimize the cumulative regret.
Revised: Therefore, we cancast our RNRM as a multi-armed bandits (MAB) problem where we select among candidate prices(analogs of arms) to minimize the cumulative regret.

Original: There are two key challenges.
Revised: There are two key challenges.

Original: The ﬁrst liesin the fact that whenever the system posts a new price, the arrival processes change immediatelywhile the system still contains “old” customers from the previously posted price.
Revised: The ﬁrst liesin the fact that whenever the system posts a new price, the arrival processes change immediatelywhile the system still contains “old” customers from the previously posted price.

Original: Therefore, itis difﬁcult to analyze the transient system performance and keep track of the cumulative revenuein a dynamic pricing scheme.
Revised: Therefore, itis difﬁcult to analyze the transient system performance and keep track of the cumulative revenuein a dynamic pricing scheme.

Original: We resolve this challenge by proposing batched algorithms.
Revised: We resolve this challenge by proposing batched algorithms.

Original: Ouralgorithms separate the time horizon into successive batches and select a price for each batch.
Revised: Ouralgorithms separate the time horizon into successive batches and select a price for each batch.

Original: Wefurther divide each batch into two intervals.
Revised: Wefurther divide each batch into two intervals.

Original: In the ﬁrst interval, the system completes serving theprevious customers and reaches the steady-state under the new price.
Revised: In the ﬁrst interval, the system completes serving theprevious customers and reaches the steady-state under the new price.

Original: In the second interval, thesystem maintains the steady-state (exploiting the new price).
Revised: In the second interval, thesystem maintains the steady-state (exploiting the new price).

Original: This batched framework not only allowsfor system stabilization (under new prices) but also allows for infrequent O (log T ) price changes,which is more implementable in practical settings (see Cheung et al. 2017).
Revised: This batched framework not only allowsfor system stabilization (under new prices) but also allows for infrequent O (log T ) price changes,which is more implementable in practical settings (see Cheung et al. 2017).

Original: The second challenge ismore mathematical as we need to ensure that the regret does not scale exponentially in the number ofproducts or resources (given the combinatorial nature of the underlying graph).(Analysis) To cope with the ﬁrst challenge mentioned above, we bound the loss of nonstationarity bybounding the length of the ﬁrst interval of each batch.
Revised: The second challenge ismore mathematical as we need to ensure that the regret does not scale exponentially in the number ofproducts or resources (given the combinatorial nature of the underlying graph).(Analysis) To cope with the ﬁrst challenge mentioned above, we bound the loss of nonstationarity bybounding the length of the ﬁrst interval of each batch.

Original: We prove that the system can reach the newsteady-state reasonably fast (see mixing times of Jackson Network in Propositions 3 and 4).
Revised: We prove that the system can reach the newsteady-state reasonably fast (see mixing times of Jackson Network in Propositions 3 and 4).

Original: To thebest of our knowledge, this is the ﬁrst result giving a ﬁnite-time bound on mixing times of Jackson
Revised: To thebest of our knowledge, this is the ﬁrst result giving a ﬁnite-time bound on mixing times of Jackson

Original: Networks.
Revised: Networks.

Original: To cope with the second challenge, we decompose the resulting Jackson Network intotopological layers and present a novel nested coupling argument (based on the layers).
Revised: To cope with the second challenge, we decompose the resulting Jackson Network intotopological layers and present a novel nested coupling argument (based on the layers).

Original: The couplingargument of Jia et al.
Revised: The couplingargument of Jia et al.

Original: (2020) in the single-resource single-product setting cannot be directly applied toour network setting.
Revised: (2020) in the single-resource single-product setting cannot be directly applied toour network setting.

Original: First, in the network of M/M/c queues, the arrival and waiting processes dependon previously visited nodes, which is signiﬁcantly more complex to analyze upon price changes.
Revised: First, in the network of M/M/c queues, the arrival and waiting processes dependon previously visited nodes, which is signiﬁcantly more complex to analyze upon price changes.

Original: To resolve this challenge, we have to carefully construct virtual and alternative queues to explicitlyaccount for such dependency.
Revised: To resolve this challenge, we have to carefully construct virtual and alternative queues to explicitlyaccount for such dependency.

Original: On a high level, the difﬁculty lies in that the original queues have atime-varying rate from an initial rate to the stable rate upon any price change.
Revised: On a high level, the difﬁculty lies in that the original queues have atime-varying rate from an initial rate to the stable rate upon any price change.

Original: One could constructvirtual queues that could clear all previously arrived customers at a speciﬁc time τ , but it is far toostrong for the coupling analysis.
Revised: One could constructvirtual queues that could clear all previously arrived customers at a speciﬁc time τ , but it is far toostrong for the coupling analysis.

Original: Instead, we construct alternative queues that start from zero initialrate and ramp up directly to the stable rate at a time stamp before τ so that the customer distributionmatches the original queues in period τ .
Revised: Instead, we construct alternative queues that start from zero initialrate and ramp up directly to the stable rate at a time stamp before τ so that the customer distributionmatches the original queues in period τ .

Original: Second, even ignoring the subtleties of time-varying rates(upon price changes), if one applied the coupling argument of Jia et al.
Revised: Second, even ignoring the subtleties of time-varying rates(upon price changes), if one applied the coupling argument of Jia et al.

Original: (2020) in a brute-force way to every node in the network, the coupling probability would result in a undesirable exponential scalingin the number of products.
Revised: (2020) in a brute-force way to every node in the network, the coupling probability would result in a undesirable exponential scalingin the number of products.

Original: We decompose the network into topological layers so that the resultingcoupling probability only scales linearly in the number of products.
Revised: We decompose the network into topological layers so that the resultingcoupling probability only scales linearly in the number of products.

Original: 1.3 Literature review
Revised: 1.3 Literature review

Original: NRM with Reusable Resources and Demand Learning.
Revised: NRM with Reusable Resources and Demand Learning.

Original: Network, demand learning, and reusableresources are three key elements of this study.
Revised: Network, demand learning, and reusableresources are three key elements of this study.

Original: We only review related literature that shares at leasttwo of the three elements.
Revised: We only review related literature that shares at leasttwo of the three elements.

Original: NRM with demand learning.
Revised: NRM with demand learning.

Original: Due to easy accessibility of real-time sales data, there have been recentstudies on developing learning and pricing algorithms with probably near-optimal regret bound under
Revised: Due to easy accessibility of real-time sales data, there have been recentstudies on developing learning and pricing algorithms with probably near-optimal regret bound under

Original: NRM settings (see, e.g., Besbes and Zeevi 2012, Chen et al.
Revised: NRM settings (see, e.g., Besbes and Zeevi 2012, Chen et al.

Original: 2019, Chen and Shi 2019, Badanidiyuruet al.
Revised: 2019, Chen and Shi 2019, Badanidiyuruet al.

Original: 2013, Agrawal and Devanur 2016, Ferreira et al. 2018).
Revised: 2013, Agrawal and Devanur 2016, Ferreira et al. 2018).

Original: All the aforementioned study considersthe canonical settings (ﬁrst deﬁned by Gallego and Van Ryzin 1997), i.e., perishable (non-reusable)resources.
Revised: All the aforementioned study considersthe canonical settings (ﬁrst deﬁned by Gallego and Van Ryzin 1997), i.e., perishable (non-reusable)resources.

Original: NRM with reusable resources.
Revised: NRM with reusable resources.

Original: The only existing studies on RNRM are Owen and Simchi-Levi (2018),
Revised: The only existing studies on RNRM are Owen and Simchi-Levi (2018),

Original: Rusmevichientong et al.
Revised: Rusmevichientong et al.

Original: and Lei and Jasin (2020).
Revised: and Lei and Jasin (2020).

Original: Lei and Jasin (2020) considered a deterministic service time assumption, and developed an asymptotically near-optimal real-time pricing controlalgorithm.
Revised: Lei and Jasin (2020) considered a deterministic service time assumption, and developed an asymptotically near-optimal real-time pricing controlalgorithm.

Original: Owen and Simchi-Levi (2018) considered a joint assortment and pricing problem under amulti-resource setting and provided constant factor performance guarantees.
Revised: Owen and Simchi-Levi (2018) considered a joint assortment and pricing problem under amulti-resource setting and provided constant factor performance guarantees.

Original: Rusmevichientong et al.(2020) considered a dynamic assortment and pricing problem with reusable products and proposed aprovably near-optimal policy based on approximate dynamic programming.
Revised: Rusmevichientong et al.(2020) considered a dynamic assortment and pricing problem with reusable products and proposed aprovably near-optimal policy based on approximate dynamic programming.

Original: Demand learning with reusable resources.
Revised: Demand learning with reusable resources.

Original: The aforementioned literature on reusable resourcesall assumed known distributional information of the underlying model.
Revised: The aforementioned literature on reusable resourcesall assumed known distributional information of the underlying model.

Original: There has been little workconsidering demand learning for service systems with reusable resources due to complex underlyingsystem dynamics.
Revised: There has been little workconsidering demand learning for service systems with reusable resources due to complex underlyingsystem dynamics.

Original: The only studies that considered demand learning for reusable resources are Chenet al.
Revised: The only studies that considered demand learning for reusable resources are Chenet al.

Original: (2020), Jia et al.
Revised: (2020), Jia et al.

Original: (2020, 2022), and all considered only the single resource setting.
Revised: (2020, 2022), and all considered only the single resource setting.

Original: Chen et al.(2020) proposed an online learning framework for a dynamic pricing and capacity sizing problem ina GI/GI/1 queue and Jia et al.
Revised: Chen et al.(2020) proposed an online learning framework for a dynamic pricing and capacity sizing problem ina GI/GI/1 queue and Jia et al.

Original: (2020, 2022) developed MAB based algorithms for optimal dynamicpricing in an M/M/c queue for the single product single resource setting.
Revised: (2020, 2022) developed MAB based algorithms for optimal dynamicpricing in an M/M/c queue for the single product single resource setting.

Original: To the best of our knowledge, we are the ﬁrst to thoroughly study all three elements.
Revised: To the best of our knowledge, we are the ﬁrst to thoroughly study all three elements.

Original: Transient Analysis of Jackson Networks.
Revised: Transient Analysis of Jackson Networks.

Original: Jackson Network has received much attention amongthe queueing networks because its steady-state distribution is particularly simple to compute asa product-form solution (Jackson 1963, 1957).
Revised: Jackson Network has received much attention amongthe queueing networks because its steady-state distribution is particularly simple to compute asa product-form solution (Jackson 1963, 1957).

Original: The queue lengths across the nodes in Jackson
Revised: The queue lengths across the nodes in Jackson

Original: Networks, i.e., a network of M/M/c queues, during the transient state can be bounded via techniquesof stochastic dominance (see Massey 1986, 1984, 1987).
Revised: Networks, i.e., a network of M/M/c queues, during the transient state can be bounded via techniquesof stochastic dominance (see Massey 1986, 1984, 1987).

Original: However, how the network reacts to systemchanges remains an open question (see, e.g., Mohamed et al. 2003, 2005).
Revised: However, how the network reacts to systemchanges remains an open question (see, e.g., Mohamed et al. 2003, 2005).

Original: Kelton and Law (1985)pointed out that a queueing system needs to run “long-enough” for removing the effects brought bythe initial starting point and then reach an ensured “steady-state” portion of the run.
Revised: Kelton and Law (1985)pointed out that a queueing system needs to run “long-enough” for removing the effects brought bythe initial starting point and then reach an ensured “steady-state” portion of the run.

Original: The only workthat explicitly developed a ﬁnite-time bound on mixing times of a queueing system is a recent paperby Jia et al.
Revised: The only workthat explicitly developed a ﬁnite-time bound on mixing times of a queueing system is a recent paperby Jia et al.

Original: In contrast, our paper gives the ﬁrst ﬁnite-time high probability bound on mixingtimes of Jackson Networks (consisting of a network of M/M/c queues) upon action changes.
Revised: In contrast, our paper gives the ﬁrst ﬁnite-time high probability bound on mixingtimes of Jackson Networks (consisting of a network of M/M/c queues) upon action changes.

Original: Batched Bandit Approaches.
Revised: Batched Bandit Approaches.

Original: UCB2 (Auer et al. 2002) and Improved-UCB (Auer and Ortner 2010)were implicitly M -batch policies with M = Θ(log( T )) .
Revised: UCB2 (Auer et al. 2002) and Improved-UCB (Auer and Ortner 2010)were implicitly M -batch policies with M = Θ(log( T )) .

Original: Later, Perchet et al.
Revised: Later, Perchet et al.

Original: studied batched
Revised: studied batched

Original: MAB in the two-armed case under a static batch design and Gao et al.
Revised: MAB in the two-armed case under a static batch design and Gao et al.

Original: (2019) extended the algorithmsto the general multiple-armed case and adaptively determined batch size.
Revised: (2019) extended the algorithmsto the general multiple-armed case and adaptively determined batch size.

Original: There is also rich literaturefocusing on batched MAB problems with switching cost.
Revised: There is also rich literaturefocusing on batched MAB problems with switching cost.

Original: For example, Cesa-Bianchi et al.
Revised: For example, Cesa-Bianchi et al.

Original: (2013)considered uniform switching cost in the objective and Simchi-Levi and Xu (2019) treated the ﬁniteswitching cost budget as a hard constraint.
Revised: (2013)considered uniform switching cost in the objective and Simchi-Levi and Xu (2019) treated the ﬁniteswitching cost budget as a hard constraint.

Original: In contrast, we construct the batched action framework tocontrol the loss of the nonstationarity (resulted from the transient time after each action change).
Revised: In contrast, we construct the batched action framework tocontrol the loss of the nonstationarity (resulted from the transient time after each action change).

Original:  Problem formulation
Revised:  Problem formulation

Original: Consider a ﬁrm that manages I types of reusable resources to provide J types of products over aﬁnite time horizon T .
Revised: Consider a ﬁrm that manages I types of reusable resources to provide J types of products over aﬁnite time horizon T .

Original: Denote the capacity of server type i ∈ I as c i .
Revised: Denote the capacity of server type i ∈ I as c i .

Original: At the beginning of each period, the ﬁrm posts a price x = { x j : j ∈ J } for all products from a given price candidate set X ⊂ R |J| .
Revised: At the beginning of each period, the ﬁrm posts a price x = { x j : j ∈ J } for all products from a given price candidate set X ⊂ R |J| .

Original: Let I = |I| , J = |J | , and X = |X| .
Revised: Let I = |I| , J = |J | , and X = |X| .

Original: Each arriving customer requests one product j ∈ J , which isprocessed through an ordered sequence of resources.
Revised: Each arriving customer requests one product j ∈ J , which isprocessed through an ordered sequence of resources.

Original: Customers arrive following a price-dependentmultivariate Poisson process with a total rate α ( x ) , which is unknown to the decision maker a priori .
Revised: Customers arrive following a price-dependentmultivariate Poisson process with a total rate α ( x ) , which is unknown to the decision maker a priori .

Original: The service time of resource i follows an exponential distribution regardless of which product typeit belongs to, which also depends on posted price x .
Revised: The service time of resource i follows an exponential distribution regardless of which product typeit belongs to, which also depends on posted price x .

Original: If there is not enough capacity for the currentresource, the customer will join the queue of this resource until being served.
Revised: If there is not enough capacity for the currentresource, the customer will join the queue of this resource until being served.

Original: The resources arereleased to process the next customers upon ﬁnishing serving the current ones.
Revised: The resources arereleased to process the next customers upon ﬁnishing serving the current ones.

Original: Customers pay (alump sum) x j to the ﬁrm for product j .
Revised: Customers pay a lumpsum x j to the ﬁrm for product j .

Revised: This is referred to as the lump sum payment model.

Revised: We also presentan alternative service duration dependent payment model where the customer pays the amount that isequal to the product’s per-unit-time price multiplied by its total service duration in Appendix F.

Original: Consider a resource-consumption matrix A ji ∈ R
Revised: Consider a resource-consumption matrix A ji ∈ R

Original: J × I , where element a ji = 1 indicates that servicetype j requires to be processed on server type i .
Revised: J × I , where element a ji = 1 indicates that servicetype j requires to be processed on server type i .

Original: We use an I -dimensional vector, y ∈ { 0 , 1 , . . .
Revised: We use an I -dimensional vector, y ∈ { 0 , 1 , . . .

Original: , ∞} Ito denote the number of customers across different types of resources.
Revised: , ∞} Ito denote the number of customers across different types of resources.

Original: The ﬁrm aims at ﬁnding apricing policy as π : { ( y , t ) : y ∈ { 0 , 1 , . . .
Revised: The ﬁrm aims at ﬁnding apricing policy as π : { ( y , t ) : y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T } → X , where the ﬁrm selects priceπ ( y , t ) ∈
Revised: , T } → X , where the ﬁrm selects priceπ ( y , t ) ∈

Original: X for period t when there are y customers in the system at the beginning of period t , tomaximize the expected cumulative revenue Y π .
Revised: X for period t when there are y customers in the system at the beginning of period t , tomaximize the expected cumulative revenue Y π .

Original: Assumption 1.
Revised: Assumption 1.

Original: (“Turn-off” price.)
Revised: (“Turn-off” price.)

Original: For each product j ∈ J , there exists a “turn-off” price ¯ p j < ∞such that, if the posted price is larger than this turn-off price, then the demand of this product is zero.
Revised: For each product j ∈ J , there exists a “turn-off” price ¯ p j < ∞such that, if the posted price is larger than this turn-off price, then the demand of this product is zero.

Original: Assumption 1 is widely assumed in NRM studies (see, e.g., Lei and Jasin 2020).
Revised: Assumption 1 is widely assumed in NRM studies (see, e.g., Lei and Jasin 2020).

Original: This turn-off pricecan be used to effectively turn off demand of undesired products whenever needed.
Revised: This turn-off pricecan be used to effectively turn off demand of undesired products whenever needed.

Original: Therefore, foreach candidate price x ∈ X , we can further decompose the products into turn-on product set J on ( x )and a turn-off product set J off ( x ) , where the latter can be empty.
Revised: Therefore, foreach candidate price x ∈ X , we can further decompose the products into turn-on product set J on ( x )and a turn-off product set J off ( x ) , where the latter can be empty.

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists an order of server types I such that the processing sequence of serversi 1 , i 2 , . .
Revised: There exists an order of server types I such that the processing sequence of serversi 1 , i 2 , . .

Original: , i n always has i 1 <
Revised: , i n always has i 1 <

Original: i 2 < . .
Revised: i 2 < . .

Original: < i n for any product j ∈ J .
Revised: < i n for any product j ∈ J .

Original: One can understand Assumption 2 via viewing a network.
Revised: One can understand Assumption 2 via viewing a network.

Original: Consider a graph with resources as nodesand draw a directed arc from node i to i (cid:48) if any product requires successive processing on resources iand i (cid:48) .
Revised: Consider a graph with resources as nodesand draw a directed arc from node i to i (cid:48) if any product requires successive processing on resources iand i (cid:48) .

Original: If the system satisﬁes Assumption 2, then this graph is acyclic, i.e., no cycles, and there existsat least one topological order such that all arcs are from smaller index to larger index.
Revised: If the system satisﬁes Assumption 2, then this graph is acyclic, i.e., no cycles, and there existsat least one topological order such that all arcs are from smaller index to larger index.

Original: Without lossof generality, we consider one such topological order when referring to resource i in the rest part ofthis paper.
Revised: Without lossof generality, we consider one such topological order when referring to resource i in the rest part ofthis paper.

Original: To better utilize this network representation, we also add a source node s to representcustomers entering the system and a sink node t to represent customers exiting the system.
Revised: To better utilize this network representation, we also add a source node s to representcustomers entering the system and a sink node t to represent customers exiting the system.

Original: Thus, theconsumption matrix can be extended to A ji ∈ R J ×
Revised: Thus, theconsumption matrix can be extended to A ji ∈ R J ×

Original: ( I +2) and a js = a jt = 1 for all products.
Revised: ( I +2) and a js = a jt = 1 for all products.

Original: Assumption 3.
Revised: Assumption 3.

Original: (Product-form demand.)
Revised: (Product-form demand.)

Original: Any given price vector x induces a (Markovian) routing matrix P ( x ) where its ( i, i (cid:48) )
Revised: Any given price vector x induces a (Markovian) routing matrix P ( x ) where its ( i, i (cid:48) )

Original: th entry p
Revised: th entry p

Original: i,i (cid:48) represents the product demand portion who will consume resource i (cid:48)
Revised: i,i (cid:48) represents the product demand portion who will consume resource i (cid:48)

Original: next out of those who consume resource i .
Revised: next out of those who consume resource i .

Original: Then the demand rate for any product j ∈ J , represented by a sequence of resources s, i 1 , i 2 , . .
Revised: Then the demand rate for any product j ∈ J , represented by a sequence of resources s, i 1 , i 2 , . .

Original: , i n , t , is given by
Revised: , i n , t , is given by

Original: [Equation]
Revised: [Equation]

Original: , which means that the demand rate for product j is equal to the total demand rate α ( x ) times asequence of routing probabilities of P ( x ) , under posted price vector x .
Revised: which means that the demand rate for product j is equal to the total demand rate α ( x ) times asequence of routing probabilities of P ( x ) , under posted price vector x .

Original: We use Figure 2 in the Appendix to demonstrate a concrete example.
Revised: We use Figure 2 in the Appendix to demonstrate a concrete example.

Original: With Assumptions 2 and 3, theservice system under any posted price x ∈ X can be reduced to a Jackson Network consisting with Inumber of M/M/c queues (when the system only contains customers arriving under this price, i.e.,after ﬁnishing serving other customers arriving under the previously posted prices).
Revised: With Assumptions 2 and 3, theservice system under any posted price x ∈ X can be reduced to a Jackson Network consisting with Inumber of M/M/c queues (when the system only contains customers arriving under this price, i.e.,after ﬁnishing serving other customers arriving under the previously posted prices).

Original: The main idea isthat there exists a mapping x (cid:55)→ α ( x ) , P ( x ) that gives rise to product demand pattern.
Revised: The main idea isthat there exists a mapping x (cid:55)→ α ( x ) , P ( x ) that gives rise to product demand pattern.

Original: Note that ourassumptions are mild in the sense that this mapping can be nonparametric.
Revised: Note that ourassumptions are mild in the sense that this mapping can be nonparametric.

Original: The decision maker doesnot know this underlying mapping and needs to learn it while maximizing revenue on the ﬂy.
Revised: The decision maker doesnot know this underlying mapping and needs to learn it while maximizing revenue on the ﬂy.

Original: Given any price vector x , (i) on one hand, this price affects the total future demand α ( x ) entering thesystem.
Revised: Given any price vector x , (i) on one hand, this price affects the total future demand α ( x ) entering thesystem.

Original: (ii)
Revised: (ii)

Original: On the other hand, this price affects the internal competition between the products, whichis reﬂected in the server routing matrix P ( x ) .
Revised: On the other hand, this price affects the internal competition between the products, whichis reﬂected in the server routing matrix P ( x ) .

Original: As a result, α ( x ) and P ( x ) deﬁne a Jackson Networkwell and the demand rate α j ( x ) of each product type is also determined (use vector (cid:126)α x ∈ R J todenote demand of products).
Revised: As a result, α ( x ) and P ( x ) deﬁne a Jackson Networkwell and the demand rate α j ( x ) of each product type is also determined (use vector (cid:126)α x ∈ R J todenote demand of products).

Original: We use λ i ( x ) to denote the demand rate of resource type i .
Revised: We use λ i ( x ) to denote the demand rate of resource type i .

Original: Assumption 4.
Revised: Assumption 4.

Original: (Stability.)
Revised: (Stability.)

Original: The queueing system of each resource is stable under any price, i.e.,c i µ i ( x ) >
Revised: The queueing system of each resource is stable under any price, i.e.,c i µ i ( x ) >

Original: λ i ( x ) .
Revised: λ i ( x ) .

Original: Therefore, the Jackson Network is stable as well.
Revised: Therefore, the Jackson Network is stable as well.

Original: Assumption 4 ensures that the underlying Jackson Network is stable under any candidate price (i.e.,the queue length will not grow indeﬁnitely).
Revised: Assumption 4 ensures that the underlying Jackson Network is stable under any candidate price (i.e.,the queue length will not grow indeﬁnitely).

Original: This assumption can be readily satisﬁed by puttingappropriate lower bounds on the price vector.
Revised: This assumption can be readily satisﬁed by puttingappropriate lower bounds on the price vector.

Original: Assumption 5.
Revised: Assumption 5.

Original: (Change of rates.)
Revised: (Change of rates.)

Original: For any node, consider the arrival and service rates ( λ 1 , µ 1 ) and( λ 2 , µ 2 ) under x 1 , x 2 ∈ X where λ 1 ≥ λ 2 , also denote the utilization factor by ρ 1 = λ 1 /µ 1 andρ 

Revised: For any node, consider the arrival and service rates ( λ 1 , µ 1 ) and( λ 2 , µ 2 ) under x 1 , x 2 ∈ X where λ 1 ≥ λ 2 , also denote the utilization factor by ρ 1 = λ 1 /µ 1 andρ

Original: = λ 2 /µ 2 , respectively, then 1) λ 1 − λ 2 ≤ ρ 1 − 3 e log ρ 1 and 2) if µ 1 > µ 2 , then ρ 1 ≥ ρ 22 .
Revised: = λ 2 /µ 2 , respectively, then 1) λ 1 − λ 2 ≤ ρ 1 − 3 e log ρ 1 and 2) if µ 1 > µ 2 , then ρ 1 ≥ ρ 22 .

Revised: This assumption is used to justify Lemmas 3 and 4 (used in the proof of Propositions 3 and 4) and theoriginal proofs are in Jia et al.

Revised: These conditions are mild and easy to validate in practice in thefollowing sense.

Original: Assumption 5 indicates the following: 1) the difference in arrival rates between two prices is bounded(by a large constant); and 2) if both the service and arrival rate under a price is higher than or equal toanother price, the utilization factor in the former case must be greater than the latter squared.
Revised: The difference in arrival rates between two prices is bounded (by a large constant). This can be easily satisﬁed by interpolating intermediate prices in the candidate price set X . 2) Ifboth the service and arrival rate under a price is higher than or equal to another price, the utilizationfactor in the former case must be greater than the latter squared.

Original: Thiscondition (which is used in the proof of Proposition 4) is rather mild in the following sense: a simplesufﬁcient condition to guarantee ρ l ≥ ρ 2 h is ρ l ≥ ρ h , which reduces to λ 1 /λ 2 ≥ µ 1 /µ 2 , meaning thatthe ratio of change in arrival rates(with respect to price) exceeds the ratio of change in service rates.
Revised: A simple sufﬁcient condition toguarantee ρ 1 ≥ ρ 22 is ρ 1 ≥ ρ 2 , which reduces to λ 1 /λ 2 ≥ µ 1 /µ 2 , and thus the practitioners onlyneed to validate that the ratio of change in arrival rates exceeds the ratio of change in service rates.

Original: Assumption 6.
Revised: Assumption 6.

Original: We assume log T ≥ 4 (to ensure sufﬁcient times for mixing).
Revised: We assume log T ≥ 4 (to ensure sufﬁcient times for mixing).

Original:  Regret, relaxed regret, and static price benchmark
Revised:  Regret, relaxed regret, and static price benchmark

Original: Under the full information scenario, where the ﬁrm knows the underlying arrival rate α j ( x ) of allproducts j ∈ J under any candidate price x ∈ X .
Revised: Under the full information scenario, where the ﬁrm knows the underlying arrival rate α j ( x ) of allproducts j ∈ J under any candidate price x ∈ X .

Original: We denote a state-dependent optimal policy as π ∗ ,
Revised: We denote a state-dependent optimal policy as π ∗ ,

Original: where π ∗ = argmax π Y π , and let Y ∗ =
Revised: where π ∗ = argmax π Y π , and let Y ∗ =

Original: Y π ∗ .
Revised: Y π ∗ .

Original: Therefore, the regret of any pricing policy π can bedeﬁned as Regret ( π, T ) =
Revised: Therefore, the regret of any pricing policy π can bedeﬁned as Regret ( π, T ) =

Original: Y ∗ − Y π .
Revised: Y ∗ − Y π .

Original: In short, we aim at ﬁnding heuristic pricing policies that leadto a small regret.
Revised: In short, we aim at ﬁnding heuristic pricing policies that leadto a small regret.

Original: However, ﬁnding the state-dependent optimal pricing policy π ∗ is computationallyintractable even under the full information scenario.
Revised: However, ﬁnding the state-dependent optimal pricing policy π ∗ is computationallyintractable even under the full information scenario.

Original: Firstly, it requires to solve a dynamic programwith an inﬁnite number of potential states, in the form of ( y , t ) , which suffers from the curse ofdimensionality.
Revised: Firstly, it requires to solve a dynamic programwith an inﬁnite number of potential states, in the form of ( y , t ) , which suffers from the curse ofdimensionality.

Original: Moreover, the transient dynamics of a queueing network is complex and hard toanalyze
Revised: Moreover, the transient dynamics of a queueing network is complex and hard toanalyze

Original: especially when the offered price changes over periods.
Revised: especially when the offered price changes over periods.

Original: Thus, it is impossible to obtain π ∗and Y ∗ .
Revised: Thus, it is impossible to obtain π ∗and Y ∗ .

Original: To tackle this problem, we develop an upper bound on the expected revenue achievable bythe state-dependent optimal expected revenue (see a similar approach in Jia et al. 2020).
Revised: To tackle this problem, we develop an upper bound on the expected revenue achievable bythe state-dependent optimal expected revenue (see a similar approach in Jia et al. 2020).

Original: An LP upper bound.
Revised: An LP upper bound.

Original: Consider continuous decision variables π xy t for y ∈ { 0 , 1 , . .
Revised: Consider continuous decision variables π xy t for y ∈ { 0 , 1 , . .

Original: , ∞} I , t =1 , . . .
Revised: , ∞} I , t =1 , . . .

Original: , T, x ∈ X , a y t for y ∈ { 0 , 1 , . . .
Revised: , T, x ∈ X , a y t for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T , and Y y t for y ∈ { 0 , 1 , . . .
Revised: , T , and Y y t for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t =1 , . . .
Revised: , ∞} I , t =1 , . . .

Original: , T .
Revised: , T .

Original: We can formulate a linear program (LP) as follows.
Revised: We can formulate a linear program (LP) as follows.

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proposition 1.
Revised: Proposition 1.

Original: The LP solution provides an upper bound on the optimal revenue, i.e., Y ∗ ≤ Y LP ,and Y LP = T · (cid:126)α T ˜ x
Revised: The LP solution provides an upper bound on the optimal revenue, i.e., Y ∗ ≤ Y LP ,and Y LP = T · (cid:126)α T ˜ x

Original: ˜ x where ˜ x = arg max x ∈X (cid:126)α T x x .
Revised: ˜ x where ˜ x = arg max x ∈X (cid:126)α T x x .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: Consider an arbitrary policy ˆ π .
Revised: Consider an arbitrary policy ˆ π .

Original: Let variable ˆ a y t take the value of probability that thesystem has y customers at the beginning of period t for y ∈ { 0 , 1 , . . .
Revised: Let variable ˆ a y t take the value of probability that thesystem has y customers at the beginning of period t for y ∈ { 0 , 1 , . . .

Original: , ∞} I and t = 1 , . . .
Revised: , ∞} I and t = 1 , . . .

Original: , T .
Revised: , T .

Original: Letvariable ˆ π xy t
Revised: Letvariable ˆ π xy t

Original: = 1 if the policy ˆ π chooses price x at state ( y , t ) , and 0 otherwise.
Revised: = 1 if the policy ˆ π chooses price x at state ( y , t ) , and 0 otherwise.

Original: Let variable ˆ Y y ttake the value of expected revenue collected before the end of period T from the customers whoarrive during period t , when the system has y customers at the beginning of period t .
Revised: Let variable ˆ Y y ttake the value of expected revenue collected before the end of period T from the customers whoarrive during period t , when the system has y customers at the beginning of period t .

Original: Then we canshow that the decisions variables ( π, a, Y ) associated with any admissible policy must satisfy theconstraints of the linear program (1), and the expected revenue of this admissible policy is exactly thecorresponding objective value in (1).
Revised: Then we canshow that the decisions variables ( π, a, Y ) associated with any admissible policy must satisfy the constraints of the linear program (1), and the expected revenue of this admissible policy is exactly thecorresponding objective value in (1).

Original: We can readily check that T · (cid:126)α T ˜ x
Revised: We can readily check that T · (cid:126)α T ˜ x

Original: ˜ x is an upper bound of Y LP .
Revised: ˜ x is an upper bound of Y LP .

Original: We can construct a solution which can obtain this upper bound and thus Y LP = T · (cid:126)α T ˜ x
Revised: We can construct a solution which can obtain this upper bound and thus Y LP = T · (cid:126)α T ˜ x

Original: ˜ x .
Revised: ˜ x .

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: The relaxed regret is deﬁned as Regret ( π, T ) =
Revised: The relaxed regret is deﬁned as Regret ( π, T ) =

Original: Y LP − Y π .
Revised: Y LP − Y π .

Original: Based on Proposition 1, we have for any policy π , the corresponding relaxed regret is a valid upperbound of its regret, i.e., Regret ( π, T ) ≥ Regret ( π, T ) , and it sufﬁces to bound the relaxed regret.
Revised: Based on Proposition 1, we have for any policy π , the corresponding relaxed regret is a valid upperbound of its regret, i.e., Regret ( π, T ) ≥ Regret ( π, T ) , and it sufﬁces to bound the relaxed regret.

Original:  Batched bandit algorithms
Revised:  Batched bandit algorithms

Original: The basis for implementing an Upper Conﬁdence Bound (UCB) type of algorithm is identifyingan upper conﬁdence bound of the expected
Revised: The basis for implementing an Upper Conﬁdence Bound (UCB) type of algorithm is identifyingan upper conﬁdence bound of the expected

Original: collected revenue of each candidate price.
Revised: collected revenue of each candidate price.

Original: We startwith the steady-state revenue rate.
Revised: We startwith the steady-state revenue rate.

Original: We can show that this steady-state revenue rate can be used tocompute a lower bound of the expected revenue during consecutive periods (see proof of Theorem1).
Revised: We can show that this steady-state revenue rate can be used tocompute a lower bound of the expected revenue during consecutive periods (see proof of Theorem1).

Original: Therefore, it is equivalent to deriving an upper conﬁdence bound of the steady-state revenuerate based on historical sales information.
Revised: Therefore, it is equivalent to deriving an upper conﬁdence bound of the steady-state revenuerate based on historical sales information.

Original: Jia et al.
Revised: Jia et al.

Original: (2020) established a concentration bound of thesteady-state revenue rate under the single product setting.
Revised: (2020) established a concentration bound of thesteady-state revenue rate under the single product setting.

Original: Here we develop a concentration bound forthe multi-product multi-resource setting.
Revised: Here we develop a concentration bound forthe multi-product multi-resource setting.

Original: We use n j ( x ) to denote the number of arrived customers requesting product j under price x .
Revised: We use n j ( x ) to denote the number of arrived customers requesting product j under price x .

Original: Weuse ˆ d jk ( x ) to denote the observed time interval between the arrival time of customer k and that ofcustomer k − 1 who are requesting product j under the price x .
Revised: Weuse ˆ d jk ( x ) to denote the observed time interval between the arrival time of customer k and that ofcustomer k − 1 who are requesting product j under the price x .

Original: We use ¯ d j ( x ) to denote the empiricalmean of the inter-arrival time and thus ¯ d j ( x ) = (cid:80) n j ( x ) k
Revised: We use ¯ d j ( x ) to denote the empiricalmean of the inter-arrival time and thus ¯ d j ( x ) = (cid:80) n j ( x ) k

Original: =1 ˆ d jk ( x ) /n
Revised: =1 ˆ d jk ( x ) /n

Original: j ( x ) , for j ∈ J .
Revised: j ( x ) , for j ∈ J .

Original: Proposition 2.
Revised: Proposition 2.

Original: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( J 14 T ) for j ∈ J on ( x ) , then we have 
Revised: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( J 14 T ) for j ∈ J on ( x ) , then we have [Equation] 

Original: [Equation]
Revised: [Equation]

Original: The proof of Proposition 2 is based on the concentration inequality for sub-exponential randomvariables (see, e.g., Boucheron et al.
Revised: The proof of Proposition 2 is based on the concentration inequality for sub-exponential randomvariables (see, e.g., Boucheron et al.

Original: 2013, Jia et al. 2020)
Revised: 2013, Jia et al. 2020)

Original: and is provided in Appendix C. Note that(cid:126)α T x x is the steady-state revenue rate and denote it by r ( (cid:126)α T x , x ) .
Revised: and is provided in Appendix C. Note that(cid:126)α T x x is the steady-state revenue rate and denote it by r ( (cid:126)α T x , x ) .

Original: Proposition 2 says that with a certainamount of arrival time observations of turn-on products under a candidate price, we can derive a validupper bound of the steady-state revenue rate with a high probability guarantee.
Revised: Proposition 2 says that with a certainamount of arrival time observations of turn-on products under a candidate price, we can derive a validupper bound of the steady-state revenue rate with a high probability guarantee.

Original: Hence, we deﬁnethe corresponding upper and lower conﬁdence bounds of the stationary revenue rate associated withprice x by the end of period t as follows.
Revised: Hence, we deﬁnethe corresponding upper and lower conﬁdence bounds of the stationary revenue rate associated withprice x by the end of period t as follows.

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: Deﬁne Rad
Revised: Deﬁne Rad

Original: [Equation]
Revised: [Equation]

Original: x ) =

Original: [Equation]
Revised: [Equation]

Original: ¯ d j ( x ) √ n j ( x ) as the conﬁdence radius by the end ofperiod t .
Revised: ¯ d j ( x ) √ n j ( x ) as the conﬁdence radius by the end ofperiod t .

Original: The upper and lower conﬁdence bounds of the revenue rate associated with price x are:
Revised: The upper and lower conﬁdence bounds of the revenue rate associated with price x are:

Original: [Equation]
Revised: [Equation]

Original: 4.1 RNRM-UCB
Revised: 4.1 RNRM-UCB

Original: With Deﬁnition 2, we can leverage the idea of UCB to solve RNRM.
Revised: With Deﬁnition 2, we can leverage the idea of UCB to solve RNRM.

Original: We separate T periods into twophases.
Revised: We separate T periods into twophases.

Original: The ﬁrst phase is the Warm-up Phase, where we offer each candidate price one by one until wecollect 8 log( J 1 / 4 T ) number of arrival time intervals for each turn-on product j ∈ J on ( x ) under eachprice.
Revised: The ﬁrst phase is the Warm-up Phase, where we offer each candidate price one by one until wecollect 8 log( J 1 / 4 T ) number of arrival time intervals for each turn-on product j ∈ J on ( x ) under eachprice.

Original: Therefore, the total length of this Warm-up Phase is O
Revised: Therefore, the total length of this Warm-up Phase is O

Original: ( X log( J 1 / 4 T )) .
Revised: ( X log( J 1 / 4 T )) .

Original: After this Warm-up
Revised: After this Warm-up

Original: Phase, Proposition 2 is valid to use.
Revised: Phase, Proposition 2 is valid to use.

Original: The second phase is the Learning Phase.
Revised: The second phase is the Learning Phase.

Original: The UCB algorithmcannot be directly used because in order to estimate the expected revenue collected when offeringa price, we have to wait until the system reaches the steady-state.
Revised: The UCB algorithmcannot be directly used because in order to estimate the expected revenue collected when offeringa price, we have to wait until the system reaches the steady-state.

Original: The key idea is to reduce thenumber of price changes and further reduce the intractable transient state performance.
Revised: The key idea is to reduce thenumber of price changes and further reduce the intractable transient state performance.

Original: We separatethe Learning Phase into M number of batches.
Revised: We separatethe Learning Phase into M number of batches.

Original: Deﬁne I m = 2 m , ∀ m = 1 , . . .
Revised: Deﬁne I m = 2 m , ∀ m = 1 , . . .

Original: , M .
Revised: , M .

Original: Each batch mcontains I m Lτ periods, where τ = (log T ) 2 .
Revised: Each batch mcontains I m Lτ periods, where τ = (log T ) 2 .

Original: Therefore, the batch size is exponentially increasing.
Revised: Therefore, the batch size is exponentially increasing.

Original: We can then add m in the subscript of notation ( n mj ( x ) , ˆ d jk ( x ) , ¯ d mj ( x ) , Rad m ( x ) , U m ( x ) , L m ( x ))to represent their values by the end of batch m .
Revised: We can then add m in the subscript of notation ( n mj ( x ) , ˆ d jk ( x ) , ¯ d mj ( x ) , Rad m ( x ) , U m ( x ) , L m ( x ))to represent their values by the end of batch m .

Original: At the beginning of each batch m + 1 , RNRM-UCBchooses price x with the highest U m ( x ) and offers this price during batch m
Revised: At the beginning of each batch m + 1 , RNRM-UCBchooses price x with the highest U m ( x ) and offers this price during batch m

Original: + 1 .
Revised: + 1 .

Original: We present thealgorithmic details in Algorithm 1.
Revised: We present thealgorithmic details in Algorithm 1.

Original: 4.2 RNRM-TS
Revised: 4.2 RNRM-TS

Original: Considering the ﬁrm holds prior information about the rates, then we can add the Bayesian assumptionto this RNRM problem.
Revised: Considering the ﬁrm holds prior information about the rates, then we can add the Bayesian assumptionto this RNRM problem.

Original: By the conjugate prior theory (see, e.g., Minton et al.
Revised: By the conjugate prior theory (see, e.g., Minton et al.

Original: 1962, Diaconis and
Revised: 1962, Diaconis and

Original: Ylvisaker 1979), we assume that RNRM-TS holds Gamma priors on α j ( x ) of product j ∈ J on ( x ) ,∀ x ∈ X and the posteriors are updated by the end of batch m, ∀ m = 1 , . . .
Revised: Ylvisaker 1979), we assume that RNRM-TS holds Gamma priors on α j ( x ) of product j ∈ J on ( x ) ,∀ x ∈ X and the posteriors are updated by the end of batch m, ∀ m = 1 , . . .

Original: , M .
Revised: , M .

Original: Speciﬁcally, weset the prior distribution of α j ( x ) as Gamma ( η α j ( x ) , β α j ( x ) ) .
Revised: Speciﬁcally, weset the prior distribution of α j ( x ) as Gamma ( η α j ( x ) , β α j ( x ) ) .

Original: After observing one arrival timeinterval ˆ d jk ( x ) of product j while offering price x , the posterior distribution of α j ( x ) is computedas Gamma ( η α j ( x ) + 1 , β α j ( x ) + ˆ d jk ( x )) .
Revised: After observing one arrival timeinterval ˆ d jk ( x ) of product j while offering price x , the posterior distribution of α j ( x ) is computedas Gamma ( η α j ( x ) + 1 , β α j ( x ) + ˆ d jk ( x )) .

Original: To implement RNRM-TS, at the beginning of batch m ,we sample parameters ˆ α j ( x ) for product j ∈ J on ( x ) , ∀ x ∈ X , following the updated posteriordistributions.
Revised: To implement RNRM-TS, at the beginning of batch m ,we sample parameters ˆ α j ( x ) for product j ∈ J on ( x ) , ∀ x ∈ X , following the updated posteriordistributions.

Original: Then we offer the price with the maximum sampled stationary network revenue rater (ˆ (cid:126)α x , x ) = (cid:80) j ∈J on ( x ) x
Revised: Then we offer the price with the maximum sampled stationary network revenue rater (ˆ (cid:126)α x , x ) = (cid:80) j ∈J on ( x ) x

Original: j ˆ α j ( x ) .
Revised: j ˆ α j ( x ) .

Original: We present the algorithmic details of RNRM-TS in Appendix.
Revised: We present the algorithmic details of RNRM-TS in Appendix.

Original:  Performance analysis
Revised:  Performance analysis

Original: 5.1 High probability bound for mixing time
Revised: 5.1 High probability bound for mixing time

Original: When the ﬁrm starts service from an empty state, i.e., zero customers in the network service system,or switches to another price from the currently offered price, the service (queueing network) systementers a transient state and needs a certain amount of time to reach the steady-state again (referredto as the mixing time).
Revised: When the ﬁrm starts service from an empty state, i.e., zero customers in the network service system,or switches to another price from the currently offered price, the service (queueing network) systementers a transient state and needs a certain amount of time to reach the steady-state again (referredto as the mixing time).

Original: Recall that the service system can be reduced to a Jackson Network undera speciﬁc offered price vector, upon completion of existing customers under the previously postedprice.
Revised: Recall that the service system can be reduced to a Jackson Network undera speciﬁc offered price vector, upon completion of existing customers under the previously postedprice.

Original: Therefore, we evaluate how quickly a Jackson Network can reach the steady-state.
Revised: Therefore, we evaluate how quickly a Jackson Network can reach the steady-state.

Original: To the bestof our knowledge, this is the ﬁrst result giving a ﬁnite-time high probability bound on mixing timesof Jackson Network system (upon action changes).
Revised: To the bestof our knowledge, this is the ﬁrst result giving a ﬁnite-time high probability bound on mixing timesof Jackson Network system (upon action changes).

Original: Technically, we develop a new coupling argumentbetween this target system and a virtual system starting from a state sampled from the steady-statedistribution and maintaining the steady-state thereafter.
Revised: Technically, we develop a new coupling argumentbetween this target system and a virtual system starting from a state sampled from the steady-statedistribution and maintaining the steady-state thereafter.

Original: Poisson arrival and departure under steady-state.
Revised: Poisson arrival and departure under steady-state.

Original: Under the steady-state of an M/M/c queue (if itexists, i.e., ρ = λ/cµ < 1 ), then the process following which the customers leaving the system is alsoa Poisson process with rate λ .
Revised: Under the steady-state of an M/M/c queue (if itexists, i.e., ρ = λ/cµ < 1 ), then the process following which the customers leaving the system is alsoa Poisson process with rate λ .

Original: Therefore, one can easily derive that the arrival as well as the departureprocesses of each type of resource (each node in the network representation) are Poisson processeswith the same rate
Revised: Therefore, one can easily derive that the arrival as well as the departureprocesses of each type of resource (each node in the network representation) are Poisson processeswith the same rate

Original: λ i under the steady-state (see the famous Burke’s Theorem Burke 1956).
Revised: λ i under the steady-state (see the famous Burke’s Theorem Burke 1956).

Original: Jia et al.
Revised: Jia et al.

Original: (2020) is the only work for studying ﬁnite-time bounds on the mixing time for an M/M/cqueue.
Revised: (2020) is the only work for studying ﬁnite-time bounds on the mixing time for an M/M/cqueue.

Original: However, their results cannot be directly applied here because (i) the arrival process of eachresource is not a well-deﬁned Poisson process and (ii) the starting point of the later queues is notzero state nor a steady-state of old system parameters.
Revised: However, their results cannot be directly applied here because (i) the arrival process of eachresource is not a well-deﬁned Poisson process and (ii) the starting point of the later queues is notzero state nor a steady-state of old system parameters.

Original: Even with these two challenges tackled, this approach will induce a high-probability mixing time which is linear on the number of resources.
Revised: Even with these two challenges tackled, thisapproach will induce a high-probability mixing time which is linear on the number of resources.

Original: Thisresult is not desired for systems with multiple (especially large) number of resources.
Revised: Thisresult is not desired for systems with multiple (especially large) number of resources.

Original: We develop anew mixing time analysis for Jackson Network.
Revised: We develop anew mixing time analysis for Jackson Network.

Original: We divide resource of the service network into layersand derive a high-probability mixing time that is only linear on the number of layers.
Revised: We divide resource of the service network into layersand derive a high-probability mixing time that is only linear on the number of layers.

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: We divide the nodes (types of servers) into layers indexed by the maximum number ofpreviously occupied nodes (including the virtual source node) along the service routes of all products.
Revised: We divide the nodes (types of servers) into layers indexed by the maximum number ofpreviously occupied nodes (including the virtual source node) along the service routes of all products.

Original: For example, Nodes 1 and 2 of the illustrative example (see Figure 2) is Layer-1 nodes and Node 3 isa Layer-2 node.
Revised: For example, Nodes 1 and 2 of the illustrative example (see Figure 2) is Layer-1 nodes and Node 3 isa Layer-2 node.

Original: We denote the number of layers by L and the set of Layer- l node as I l .
Revised: We denote the number of layers by L and the set of Layer- l node as I l .

Original: Proposition 3.
Revised: Proposition 3.

Original: (Coupling probability of a Jackson Network starting from empty state) Deﬁne
Revised: (Coupling probability of a Jackson Network starting from empty state) Deﬁne

Original: A t tobe the event in which the Jackson Network reaches the steady-state by time t .
Revised: A t tobe the event in which the Jackson Network reaches the steady-state by time t .

Original: For t ≥ Lτ , whereτ = (log T ) 2 , we have P ( A t ) ≥ 1 − 2 IT 2 .
Revised: For t ≥ Lτ , whereτ = (log T ) 2 , we have P ( A t ) ≥ 1 − 2 IT 2 .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: We use the aforementioned Poisson arrival and departure property of the nodes in a Jackson Network to bound the mixing time of the nodes layer by layer.
Revised: We use the aforementioned Poisson arrival and departure property of the nodes in a Jackson Network to bound the mixing time of the nodes layer by layer.

Original: We deﬁne events A t (cid:48) ,l = { all layer-l nodes reach the steady-state by time t (cid:48) } and A it (cid:48)
Revised: We deﬁne events A t (cid:48) ,l = { all layer-l nodes reach the steady-state by time t (cid:48) } and A i t (cid:48)

Original: = { the node i reaches the steady-state by time t (cid:48) } .
Revised: = { the node i reaches the steady-state by time t (cid:48) } .

Original: By treating the nodes in the same layer together,
Revised: By treating the nodes in the same layer together,

Original: P

Original: [Equation]
Revised: [Equation]

Revised: τ, 1 )

Revised: L (cid:89) l =

Revised: P ( A lτ,l |A τ, 1 . . .

Revised: A ( l − 1) τ,l − 1 ) .

Original: We then prove that P ( A lτ,l |A τ, 1 . . .
Revised: We then prove that P ( A lτ,l |A τ, 1 . . .

Original: A ( l − 1) τ,l − 1 ) ≥ 1 − (2 I l ) /T 2 , where I l = |I
Revised: A ( l − 1) τ,l − 1 ) ≥ 1 − (2 I l ) /T 2 , where I l = |I

Original: l | , by establishing acoupling argument between the target queue (with the help of an artiﬁcially constructed alternativequeue) and a virtual queue starting from a state sampled from the steady-state distribution andmaintaining the steady-state thereafter.
Revised: l | , by establishing acoupling argument between the target queue (with the help of an artiﬁcially constructed alternativequeue) and a virtual queue starting from a state sampled from the steady-state distribution andmaintaining the steady-state thereafter.

Original: The detailed proof is provided in Appendix E.
Revised: The detailed proof is provided in Appendix E.

Original: In addition, the transient state of the Jackson Network, as well as each resource queue (or each node),can be induced not only by starting from empty state but also by starting from a steady-state of apreviously posted price.
Revised: In addition, the transient state of the Jackson Network, as well as each resource queue (or each node),can be induced not only by starting from empty state but also by starting from a steady-state of apreviously posted price.

Original: It is possible that when a new price is posted, the customers arrived under thepreviously posted price still remain in the system.
Revised: It is possible that when a new price is posted, the customers arrived under thepreviously posted price still remain in the system.

Original: Therefore we also need to ensure that the couplingalso occurs with high probability when price changes.
Revised: Therefore we also need to ensure that the couplingalso occurs with high probability when price changes.

Original: Proposition 4.
Revised: Proposition 4.

Original: (Coupling probability of Jackson Network starting from steady-state of previousprice.)
Revised: (Coupling probability of Jackson Network starting from steady-state of previousprice.)

Original: Deﬁne B t to be the event in which the Jackson Network reaches the steady-state by time t .
Revised: Deﬁne B t to be the event in which the Jackson Network reaches the steady-state by time t .

Original: Fort ≥ 2 Lτ , where τ = (log T ) 2 , we have P ( B t ) ≥ 1 − 4 IT 2 .
Revised: Fort ≥ 2 Lτ , where τ = (log T ) 2 , we have P ( B t ) ≥ 1 − 4 IT 2 .

Original: Proof Sketch.
Revised: Proof Sketch.

Original: This proposition is derived based on Proposition 3 and Assumption 5.
Revised: This proposition is derived based on Proposition 3 and Assumption 5.

Original: We deﬁne B t (cid:48) ,land B it (cid:48) similarly and bound P ( B 2 lτ,l |B 2 τ, 1 . . .
Revised: We deﬁne B t (cid:48) ,land B it (cid:48) similarly and bound P ( B 2 lτ,l |B 2 τ, 1 . . .

Original: B 2( l − 1) τ,l − 1 ) ≥ 1 − 4 I l T 2 by decomposing the scenariosby how the arrival and service rates are changing when considering each resource queue.
Revised: B 2( l − 1) τ,l − 1 ) ≥ 1 − 4 I l T 2 by decomposing the scenariosby how the arrival and service rates are changing when considering each resource queue.

Original: 5.2 Regret bounds of RNRM-UCB and RNRM-TS
Revised: 5.2 Regret bounds of RNRM-UCB and RNRM-TS

Original: The high probability bounds for mixing time of Jackson Network developed in §5.1 help bound theloss due to the nonstationarity, and thus we are ready to carry out the regret analysis of RNRM-UCB.
Revised: The high probability bounds for mixing time of Jackson Network developed in §5.1 help bound theloss due to the nonstationarity, and thus we are ready to carry out the regret analysis of RNRM-UCB.

Original: Theorem 1.
Revised: Theorem 1.

Original: The T -period cumulative regret of RNRM-UCB is bounded by ˜ O (cid:16) J √ XT (cid:17) .
Revised: The T -period cumulative regret of RNRM-UCB is bounded by ˜ O (cid:16) J √ XT (cid:17) .

Original: Note that the regret lower bound for nominal MAB (where J = 1 ) is Ω( √ XT ) (see, e.g., Theorem2.1 in Slivkins (2019)), which suggests that our regret bound is tight, if the system size J is ﬁxed.
Revised: Note that the regret lower bound for nominal MAB (where J = 1 ) is Ω( √ XT ) (see, e.g., Theorem2.1 in Slivkins (2019)), which suggests that our regret bound is tight, if the system size J is ﬁxed.

Original: Proof Sketch.
Revised: Proof Sketch.

Original: The regret of the Warm-up phase is at most linear on the length of the Warm-up
Revised: The regret of the Warm-up phase is at most linear on the length of the Warm-up

Original: Phase, i.e., O ( XJ log( J 1 / 4 T )) .
Revised: Phase, i.e., O ( XJ log( J 1 / 4 T )) .

Original: We decompose the relaxed regret in the Learning Phase into twoparts.
Revised: We decompose the relaxed regret in the Learning Phase into twoparts.

Original: The ﬁrst part is called loss of nonstationarity and it represents the revenue loss induced bythe transient performance when changing the price.
Revised: The ﬁrst part is called loss of nonstationarity and it represents the revenue loss induced bythe transient performance when changing the price.

Original: For batch m , we can derive that the expectedrevenue collected under the selected price x m as r ( (cid:126)α x m , x m )
Revised: For batch m , we can derive that the expectedrevenue collected under the selected price x m as r ( (cid:126)α x m , x m )

Original: I m Lτ − O
Revised: I m Lτ − O

Original: ( Lτ ) with Proposition 4.
Revised: ( Lτ ) with Proposition 4.

Original: The second part represents the loss of suboptimality , which is induced by not offering the optimalstate-independent price in certain time periods.
Revised: The second part represents the loss of suboptimality , which is induced by not offering the optimalstate-independent price in certain time periods.

Original: This part can be further rewritten as ∆( x m ) I m Lτ and∆( x m ) =
Revised: This part can be further rewritten as ∆( x m ) I m Lτ and ∆( x m ) =

Original: ( r ( (cid:126)α ˜ x , ˜ x ) − U m (˜ x ))
Revised: ( r ( (cid:126)α ˜ x , ˜ x ) − U m (˜ x ))

Original: + ( U m ( x m ) − L m ( x m ))
Revised: + ( U m ( x m ) − L m ( x m ))

Original: + ( L m ( x m ) − r ( (cid:126)α x m , x m )) .
Revised: + ( L m ( x m ) − r ( (cid:126)α x m , x m )) .

Original: We boundthese three terms separately based on Proposition 2 and Lemma 2.
Revised: We boundthese three terms separately based on Proposition 2 and Lemma 2.

Original: Combine all results together, wereach a ˜ O ( J √ XT ) regret bound.
Revised: Combine all results together, wereach a ˜ O ( J √ XT ) regret bound.

Original: We provided detailed proof in Appendix E.
Revised: We provided detailed proof in Appendix E.

Original: We use Bayesian regret as the performance measure of RNRM-TS, which is typically used for the
Revised: We use Bayesian regret as the performance measure of RNRM-TS, which is typically used for the

Original: Bayesian setting and is computed as the expected regret of the instance in expectation over all probleminstances.
Revised: Bayesian setting and is computed as the expected regret of the instance in expectation over all probleminstances.

Original: Similarly, we deﬁne a relaxed Bayesian regret:.
Revised: Similarly, we deﬁne a relaxed Bayesian regret:.

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: Denoting all the events before period t by F t , ∀ t ≥ 0 , we deﬁne the relaxed Bayesian regret of policy π by period T as
Revised: Denoting all the events before period t by F t , ∀ t ≥ 0 , we deﬁne the relaxed Bayesian regret of policy π by period T as

Original: [Equation]
Revised: [Equation]

Original: The relaxed Bayesian regret bound of RNRM-TS is in the same order as RNRM-UCB (Appendix E).
Revised: The relaxed Bayesian regret bound of RNRM-TS is in the same order as RNRM-UCB (Appendix E).

Original: Theorem 2.
Revised: Theorem 2.

Original: The T -period cumulative Bayesian regret of RNRM-TS is bounded by ˜ O
Revised: The T -period cumulative Bayesian regret of RNRM-TS is bounded by ˜ O

Original: ( J √ XT ) .
Revised: ( J √ XT ) .

Original:  Numerical results
Revised:  Numerical results

Original: We consider T = 2000 , I = 2 with capacity c 1 =
Revised: We consider T = 2000 , I = 2 with capacity c 1 =

Original: c
Revised: c

Original: = 2 ,
Revised: = 2 ,

Original: J = 3 with product-resource orders{ 1 , 2 , 1 → 2 } , and X = { [1 , 2 , 3] , [2 , 1 , 3] , [2 , 2 , 3] } .
Revised: J = 3 with product-resource orders{ 1 , 2 , 1 → 2 } , and X = { [1 , 2 , 3] , [2 , 1 , 3] , [2 , 2 , 3] } .

Original: We set the underlying arrival rates of candidateprices as { [4 , 2 , 4] , [3 , 4 , 3] , [2 . 5 , 3 , 2 . 5] } and use a uniform service rates [5 , 4 . 5] for all prices.
Revised: We set the underlying arrival rates of candidateprices as { [4 , 2 , 4] , [3 , 4 , 3] , [2 . 5 , 3 , 2 . 5] } and use a uniform service rates [5 , 4 . 5] for all prices.

Original: Weadopt the blind network revenue management algorithm proposed in Besbes and Zeevi (2012) toour settings as a benchmark.
Revised: Weadopt the blind network revenue management algorithm proposed in Besbes and Zeevi (2012) toour settings as a benchmark.

Original: This benchmark algorithm tries each price for θT/X periods and thenselects the price with the highest revenue in the rest of the periods.
Revised: This benchmark algorithm tries each price for θT/X periods and thenselects the price with the highest revenue in the rest of the periods.

Original: We set θ as 0 .
Revised: We set θ as 0 .

Original: 1 , and 0 .
Revised: 1 , and 0 .

Original: in three Besbes and Zeevi (2012) benchmarks, respectively (the same as the settings in Table 1 inthe original paper Besbes and Zeevi 2012).
Revised: in three Besbes and Zeevi (2012) benchmarks, respectively (the same as the settings in Table 1 inthe original paper Besbes and Zeevi 2012).

Original: To conclude, we implement ﬁve pricing algorithms:
Revised: To conclude, we implement ﬁve pricing algorithms:

Original: RNRM-UCB, RNRM-TS, and three benchmark policies with different values of θ .
Revised: RNRM-UCB, RNRM-TS, and three benchmark policies with different values of θ .

Original: Figure 1a shows the offered price over periods of each algorithm and Figure 1b depicts the cumulativetime-average relaxed regret, i.e., (cid:80) tt (cid:48)
Revised: Figure 1a shows the offered price over periods of each algorithm and Figure 1b depicts the cumulativetime-average relaxed regret, i.e., (cid:80) tt (cid:48)

Original: =1 ( Y LP t (cid:48) − Y πt (cid:48) ) /t .
Revised: =1 ( Y LP t (cid:48) − Y πt (cid:48) ) /t .

Original: For all algorithms (except “OPT”), the revenueis collected from the simulation where the system starts with no customers at time 0 .
Revised: For all algorithms (except “OPT”), the revenueis collected from the simulation where the system starts with no customers at time 0 .

Original: We can observethat both RNRM-UCB and RNRM-TS identify the best price (P1) through the learning process andthe time average regret shrinks to zero very quickly.
Revised: We can observethat both RNRM-UCB and RNRM-TS identify the best price (P1) through the learning process andthe time average regret shrinks to zero very quickly.

Original: [Figure]
Revised: [Figure]

Original:  Conclusion and social impacts
Revised:  Conclusion and social impacts

Original: We proposed two new batch bandit algorithms for RNRM problems with multiple products andmultiple reusable resources.
Revised: We proposed two new batch bandit algorithms for RNRM problems with multiple products andmultiple reusable resources.

Original: We established optimal regret upper bounds and demonstrated superiorcomputational results.
Revised: We established optimal regret upper bounds and demonstrated superiorcomputational results.

Original: This work solves an important practical problem and we believe that ourapproach will open many doors to learning more complex stochastic systems (especially in thenetwork settings).
Revised: This work solves an important practical problem and we believe that ourapproach will open many doors to learning more complex stochastic systems (especially in thenetwork settings).

Original: There are two immediate future research questions: (1) how we can accommodatecontinuous price space; and (2) how we can extend arrivals and services to general distributions.
Revised: There are two immediate future research questions: (1) how we can accommodatecontinuous price space; and (2) how we can extend arrivals and services to general distributions.

Original: Bothdirections would need signiﬁcant methodological innovations to be developed.
Revised: Bothdirections would need signiﬁcant methodological innovations to be developed.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] See Section 1.2.
Revised: [Yes] See Section 1.2.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We consider a speciﬁc service
Revised: [Yes] We consider a speciﬁc service

Original: system that can be reduced to a Jackson Network.
Revised: system that can be reduced to a Jackson Network.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] Nopotential negative societal impacts.
Revised: [N/A] Nopotential negative societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 2.
Revised: [Yes] See Section 2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We provide proof
Revised: [Yes] We provide proof

Original: sketch in the main body of the paper and provide detailed proof in the appendix.
Revised: sketch in the main body of the paper and provide detailed proof in the appendix.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We include
Revised: [Yes] We include

Original: detailed experiment setup in the appendix.
Revised: detailed experiment setup in the appendix.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [No] We are not comparing the running
Revised: [No] We are not comparing the running

Original: time and thus these information is not relevant.
Revised: time and thus these information is not relevant.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cite existing
Revised: [Yes] We cite existing

Original: lemmas.
Revised: lemmas.

Original: For example, see Lemma 3 in Appendix E.
Revised: For example, see Lemma 3 in Appendix E.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]
Revised: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: B Illustrative example
Revised: B Illustrative example

Original: We refer to this simple illustrative example several times in the rest of the paper.
Revised: We refer to this simple illustrative example several times in the rest of the paper.

Original: Consider 4 types ofresources and 4 types of products:
Revised: Consider 4 types ofresources and 4 types of products:

Original: • Product No. 1 : ( s, 1 , 3 , t ) , i.e., SQL Database → EC
Revised: • Product No. 1 : ( s, 1 , 3 , t ) , i e., SQL Database → EC

Original: • Product No. 2 : ( s, 1 , 3 , 4 , t ) , i.e., SQL Database → EC2 → SAP
Revised: • Product No. 2 : ( s, 1 , 3 , 4 , t ) , i.e., SQL Database → EC2 → SAP

Original: • Product No. 3 : ( s, 2 , 3 , t ) ,
Revised: • Product No. 3 : ( s, 2 , 3 , t ) ,

Original: i
Revised: i

Original: e., NoSQL Database → EC
Revised: e., NoSQL Database → EC

Original: • Product No. 4 : ( s, 2 , 3 , 4 , t ) , i.e., NoSQL Database → EC2 → SAP
Revised: • Product No. 4 : ( s, 2 , 3 , 4 , t ) , i.e., NoSQL Database → EC2 → SAP

Original: Typically, customers will choose between an SQL based or a NoSQL based database service to storeand preprocess the data.
Revised: Typically, customers will choose between an SQL based or a NoSQL based database service to storeand preprocess the data.

Original: After running algorithms on elastic cloud compute EC2, only a subset ofcustomers may choose to further integrate SAP applications.
Revised: After running algorithms on elastic cloud compute EC2, only a subset ofcustomers may choose to further integrate SAP applications.

Original: The network representation is given by
Revised: The network representation is given by

Original: Figure 2, and it is straightforward to check that this acyclic graph satisﬁes Assumption 2.
Revised: Figure 2, and it is straightforward to check that this acyclic graph satisﬁes Assumption 2.

Original: We assign a price vector x = ( x 1 , x 2 , x 3 , x 4 ) to these four products.
Revised: We assign a price vector x = ( x 1 , x 2 , x 3 , x 4 ) to these four products.

Original: Note that we do not assign prices to resources.
Revised: Note that we do not assign prices to resources.

Original: We only sell individual product (as a bundle of resources).
Revised: We only sell individual product (as a bundle of resources).

Original: Each posted price vector induces two quantities, namely, α ( x ) and P ( x ) .
Revised: Each posted price vector induces two quantities, namely, α ( x ) and P ( x ) .

Original: Here α ( x ) is the total arrival rate of all products given price vector x while P ( x ) is the so-called (Markovian) “routing matrix” that captures the demand ﬂows for products.
Revised: Here α ( x ) is the total arrival rate of all products given price vector x while P ( x ) is the so-called (Markovian) “routing matrix” that captures the demand ﬂows for products.

Original: One can represent a product j ∈ J using an s - t path, i.e., ( s, i 1 , . .
Revised: One can represent a product j ∈ J using an s - t path, i.e., ( s, i 1 , . .

Original: , i n , t ) .
Revised: , i n , t ) .

Original: Assume that Assumption 3 holds.
Revised: Assume that Assumption 3 holds.

Original: Then we can write the demand rate for this particular product j as
Revised: Then we can write the demand rate for this particular product j as

Original: [Equation]
Revised: [Equation]

Original: where p i,i (cid:48) are the ( i, i (cid:48) )
Revised: where p i,i (cid:48) are the ( i, i (cid:48) )

Original: th entry of the routing probability matrix P ( x ) .
Revised: th entry of the routing probability matrix P ( x ) .

Original: Note that in our setting, the mapping x (cid:55)→ α ( x ) , P ( x ) is unknown to the decision maker.
Revised: Note that in our setting, the mapping x (cid:55)→ α ( x ) , P ( x ) is unknown to the decision maker.

Original: Thus, weneed to offer different price vector x to learn this mapping.
Revised: Thus, weneed to offer different price vector x to learn this mapping.

Original: And we also assume this mapping isnonparametric.
Revised: And we also assume this mapping isnonparametric.

Original: For the sake of illustration, we give the following linear mapping as an example.
Revised: For the sake of illustration, we give the following linear mapping as an example.

Original: [Equation]
Revised: [Equation]

Original: Suppose the price vector x = (1 , 2 , 2 , 5) .
Revised: Suppose the price vector x = (1 , 2 , 2 , 5) .

Original: In this case, the total arrival rate α ( x ) = 100 and the routing matrix P ( x ) is given byP
Revised: In this case, the total arrival rate α ( x ) = 100 and the routing matrix P ( x ) is given by

Original: [Equation]

Original: 

Original: [Equation]
Revised: [Equation]

Original:  [Equation] p [Equation]
Revised: giving rise to

Original: x

Original: [Equation] x [Equation] + x
Revised: Product Demand:

Original: [Equation]

Original: x

Original: [Equation]

Original: [Equation]

Original: p

Original: [Equation]
Revised: [Equation]

Original: t

Original: [Equation]

Original: x 2 + x

Original: [Equation]

Original: x

Original: [Equation]

Original: + x

Original: [Equation]

Original: + x

Original: [Equation] + x 4 = [Equation] .p
Revised: Server Demand:

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation] + [Equation] x
Revised: Note that in this example, Assumption 3 also means that out of all the customers who have chosen

Original: [Equation]

Original: x [Equation] = [Equation]
Revised: EC2, the proportion of choosing SAP is 0.3 and the proportion of not choosing SAP is 0.7 (regardlessof database options).

Original: .p

Original: [Equation]

Original: p

Original: [Equation]

Original: p

Original: [Equation]

Original: =, giving rise to

Original: Product Demand:  [Equation] [Equation] α
Revised: We also consider two payment models, the lump sum payment model (presented in the mainmanuscript) and the service duration dependent payment model (presented in Appendix F).

Original: [Equation] 100 × 0 . [Equation] × 0 .
Revised: • ( Lump sum payment model.)

Original: [Equation]

Original: α 4 = 100 × 0 .

Original: 3 × 0 . 3 =, Server Demand:   λ 1 = 100 × 0 . 7 =λ 2 = 100 × 0 .
Revised: A customer of product j ∈ J pays a lump sum

Original: 3 =λ 3 = 100 × 1 =λ 4 = 100 × 0 . 3 =.
Revised: x j to the ﬁrm.

Original: Note that in this example, Assumption 3 also means that out of all the customers who have chosen EC2, the proportion of choosing SAP is 0.3 and the proportion of not choosing SAP is 0.7 (regardlessof database options).
Revised: We provide justiﬁcation for this model as follows:

Original: C Technical results in Section 3
Revised: If a customers chooses product j , then the customer will pay x j to the ﬁrm regardless of how long the services over resources are. In our illustrative example, each product can be quoted a ﬁxed price (i

Revised: e., not service durationdependent), which could be seen a preset “cloud solution package”.

Revised: The customers mayprefer or beneﬁt from this ﬁxed price (contract) from their own project/accounting/cash-ﬂowmanagement.

Revised: Under this model, even though the customers are not sensitive to service

Revised: durations, the cloud platform is nevertheless sensitive to service durations since they affect

Revised: how many products/services can be completed within a ﬁnite horizon.

Revised: • ( Service duration dependent payment model.)

Revised: A customer of product j ∈ J pays theamount that is equal to the per-unit-time x j multiplied by the total service duration.

Revised: This

Revised: payment scheme is duration dependent (e

Revised: g., how much time is spent using EC2 in ourillustrative example).

Revised: The model and analytical details are presented in Appendix F.

Revised: C Technical results in Section

Original: Proof of Proposition 1.
Revised: Proof of Proposition 1.

Original: We show that the decisions variables ( π, a, Y ) associated with any admissible policy must satisfy theconstraints of the linear program (1), and the expected revenue of this admissible policy is exactly thecorresponding objective value in (1).
Revised: We show that the decisions variables ( π, a, Y ) associated with any admissible policy must satisfy theconstraints of the linear program (1), and the expected revenue of this admissible policy is exactly thecorresponding objective value in (1).

Original: Consider an arbitrary policy ˆ π .
Revised: Consider an arbitrary policy ˆ π .

Original: We obtain the values of decisionvariables by the following steps.
Revised: We obtain the values of decisionvariables by the following steps.

Original: Variable ˆ a y t takes the value of probability that the system has ycustomers at the beginning of period t under the policy
Revised: Variable ˆ a y t takes the value of probability that the system has ycustomers at the beginning of period t under the policy

Original: ˆ π for y ∈ { 0 , 1 , . . .
Revised: ˆ π for y ∈ { 0 , 1 , . . .

Original: , ∞} I and t = 1 , . . .
Revised: , ∞} I and t = 1 , . . .

Original: , T .
Revised: , T .

Original: Therefore, constraints (1b) are immediately satisﬁed.
Revised: Therefore, constraints (1b) are immediately satisﬁed.

Original: Variable ˆ π xy t = 1 if the policy ˆ π chooses pricex at state ( y , t ) , otherwise, ˆ π xy t = 0 , for y ∈ { 0 , 1 , . . .
Revised: Variable ˆ π xy t = 1 if the policy ˆ π chooses pricex at state ( y , t ) , otherwise, ˆ π xy t = 0 , for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T , y ∈ X .
Revised: , T , y ∈ X .

Original: As a result,constraints (1c) hold because in the left-hand side we have one variable equal to 1 and others allzero.
Revised: As a result,constraints (1c) hold because in the left-hand side we have one variable equal to 1 and others allzero.

Original: Variable ˆ Y y t takes the value of expected revenue collected before the end of period T from thecustomers who arrive during period t , when the system has y customers at the beginning of periodt , for y ∈ { 0 , 1 , . . .
Revised: Variable ˆ Y y t takes the value of expected revenue collected before the end of period T from thecustomers who arrive during period t , when the system has y customers at the beginning of periodt , for y ∈ { 0 , 1 , . . .

Original: , ∞} I .
Revised: , ∞} I .

Original: Consider arbitrary y and t , we denote the price chosen at state ( y , t ) bythe policy ˆ π as ˆ x .
Revised: Consider arbitrary y and t , we denote the price chosen at state ( y , t ) bythe policy ˆ π as ˆ x .

Original: Then, by plugging in ˆ π ˆ xy t = 1 , in the right-hand side of constraints (1d), we have(cid:126)α T x x , where (cid:126)α T x denotes the average number of arrived customers for consuming products duringperiod t and ˆ x is the revenue vector of these products.
Revised: Then, by plugging in ˆ π ˆ xy t = 1 , in the right-hand side of constraints (1d), we have(cid:126)α T x x , where (cid:126)α T x denotes the average number of arrived customers for consuming products duringperiod t and ˆ x is the revenue vector of these products.

Original: Therefore, (cid:126)α T x x denotes the expected revenuecollected from customers who arrive during period t (where assuming all arrived customers beingserved successfully).
Revised: Therefore, (cid:126)α T x x denotes the expected revenuecollected from customers who arrive during period t (where assuming all arrived customers beingserved successfully).

Original: Consequently, we can conclude that the right-hand side of (1d) is as large as theleft-hand side, because the right-hand side may also compute the revenue collected after the end ofperiod T .
Revised: Consequently, we can conclude that the right-hand side of (1d) is as large as theleft-hand side, because the right-hand side may also compute the revenue collected after the end ofperiod T .

Original: Hence, constraints (1d) are also satisﬁed.
Revised: Hence, constraints (1d) are also satisﬁed.

Original: By the value assignment steps of ˆ a y t and ˆ Y y t ,for y ∈ { 0 , 1 , . . .
Revised: By the value assignment steps of ˆ a y t and ˆ Y y t ,for y ∈ { 0 , 1 , . . .

Original: , ∞} I , t = 1 , . . .
Revised: , ∞} I , t = 1 , . . .

Original: , T , one has that the corresponding LP objective value at solution(ˆ π, ˆ a, ˆ J ) is Y ˆ π .
Revised: , T , one has that the corresponding LP objective value at solution(ˆ π, ˆ a, ˆ J ) is Y ˆ π .

Original: Therefore, we have Y LP ≥ Y π , for any admissible policy π , and thus also for theoptimal state-dependent policy π ∗ .
Revised: Therefore, we have Y LP ≥ Y π , for any admissible policy π , and thus also for theoptimal state-dependent policy π ∗ .

Original: By deﬁning ˜ x = argmax x ∈X (cid:126)α T x x , we ﬁnd an optimal solution (˜ π, ˜ a, ˜ Y ) to the LP in (1) as follows.
Revised: By deﬁning ˜ x = argmax x ∈X (cid:126)α T x x , we ﬁnd an optimal solution (˜ π, ˜ a, ˜ Y ) to the LP in (1) as follows.

Original: [Equation]
Revised: [Equation]

Original: where y 

Revised: where y

Original: =
Revised: =

Original: [0] J .
Revised: [0] J .

Original: Then, we can compute Y LP = T · (cid:126)α T ˜ x
Revised: Then, we can compute Y LP = T · (cid:126)α T ˜ x

Original: ˜ x .
Revised: ˜ x .

Original: Note that (cid:126)α T x x is the stationary revenuerate of price x , denoted as r ( (cid:126)α x , x ) .
Revised: Note that (cid:126)α T x x is the stationary revenuerate of price x , denoted as r ( (cid:126)α x , x ) .

Original: Q.E.D.
Revised: Q.E.D.

Original: D Technical 
Revised: D Technical results in Section 4.

Original: results in Section 4.

Original: Proof of Proposition 2.
Revised: Proof of Proposition 2.

Original: The proof is based on the concentration bound for sub-exponential random variables.
Revised: The proof is based on the concentration bound for sub-exponential random variables.

Original: For productj ∈ J off , their demand is off, i.e., α j ( x ) = 0 , j ∈ J off ( x ) .
Revised: For productj ∈ J off , their demand is off, i.e., α j ( x ) = 0 , j ∈ J off ( x ) .

Original: In this proof, we only consider one price Algorithm and thus we use α j to represent α j ( x ) for notation simplicity.
Revised: In this proof, we only consider one priceand thus we use α j to represent α j ( x ) for notation simplicity.

Original: We can rewrite the right-hand-side as:
Revised: We can rewrite the right-hand-side as:

Original: P   (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:126)α T x x − (cid:88) j ∈J on ( x ) x j ¯ d j ( x ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:88)

Original: [Equation]
Revised: [Equation]

Original: We can derive that for one product j ∈ J on ( x ) : P   (cid:12)(cid:12)(cid:12)(cid:12)
Revised: [Table]

Original: α j xj −
Revised: ( x

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Revised:  .

Revised: We can derive that for one product j ∈ J on ( x ) :

Revised: P 

Original: [Equation]
Revised: [Equation]

Original: The exponential arrival time intervals are also sub-exponential random variables with parameters( 4 α 2 j , 2 α j ) , denoted by SE ( 4 α 2 j , 2 α j ) , see Lemma 1 in the appendix.
Revised: The exponential arrival time intervals are also sub-exponential random variables with parameters( 4 α 2 j , 2 α j ) , denoted by SE ( 4 α 2 j , 2 α j ) , see Lemma 1 in the appendix.

Original: Consider the concentration inequalityfor i.i.d.
Revised: Consider the concentration inequality for i.i.d.

Original: exponential variables ξ i , i = 1 , . . .
Revised: exponential variables ξ i , i = 1 , . . .

Original: n with mean 1 /λ :P
Revised: n with mean 1 /λ :

Original: [Equation]

Original: λ − n n (cid:88) i = ξ

Original: i

Original: [Equation]
Revised: [Equation]

Original: By the assumption n j ( x ) ≥ 8 log( J 14 T ) , we can compute
Revised: By the assumption n j ( x ) ≥ 8 log( J 14 T ) , we can compute

Original: [Equation]
Revised: [Equation]

Original: and thus we can apply the ﬁrst inequality in (6) to (5).
Revised: and thus we can apply the ﬁrst inequality in (6) to (5).

Original: Therefore, we can derive
Revised: Therefore, we can derive

Original: P

Original: [Equation]
Revised: [Equation]

Original: Then, we can compute a lower bound of (4) as
Revised: Then, we can compute a lower bound of (4) as

Original: [Equation]
Revised: [Equation]

Revised: ≥ 1 − 2 J

Revised: J · T 4 = 1 −

Revised: T 4 .

Revised: Q.E.D.

Original: Lemma 1.
Revised: Lemma 1.

Original: (Sub-exponential property, Lemma 3 in Jia et al.
Revised: (Sub-exponential property, Lemma 3 in Jia et al.

Original: If a random variable X follows anexponential distribution with mean 1 /λ , then the random variable X − 1 λ is ( 4 λ 2 , 2 λ ) -sub-exponential.
Revised: If a random variable X follows anexponential distribution with mean 1 /λ , then the random variable X − 1 λ is ( 4 λ 2 , 2 λ ) -sub-exponential.

Original: E Technical results in Section 5
Revised: E Technical results in Section 5

Original: Proof of Proposition 3.
Revised: Proof of Proposition 3.

Original: In this proof, we analyze the coupling time of nodes layer by layer.
Revised: In this proof, we analyze the coupling time of nodes layer by layer.

Original: We deﬁne events as follows (all starting from empty state).
Revised: We deﬁne events as follows (all starting from empty state).

Original: A t = { the Jackson Network reaches the steady-state by time t } ,
Revised: [Equation] the steady [Equation]

Original: A t,l = { all layer-l nodes reach the steady-state by time t } ,
Revised: state

Original: A it =
Revised: [Equation]

Original: { the node i reaches the steady-state by time t } .
Revised: time [Equation] steady [Equation]

Original: First, we discuss the coupling analysis of an M/M/c queue, which is the basis for completing thisproof.
Revised: First, we discuss the coupling analysis of an M/M/c queue, which is the basis for completing thisproof.

Original: With the assumption log > 4 , the probability that an M/M/c queue reaches the steady-statefrom empty state after (log T ) 2 periods of time is bounded by 1 − 2 /T 2 , see Lemma 3 (Proposition4 in Jia et al.
Revised: With the assumption log > 4 , the probability that an M/M/c queue reaches the steady-statefrom empty state after (log T ) 2 periods of time is bounded by 1 − 2 /T 2 , see Lemma 3 (Proposition4 in Jia et al.

Original: This results are derived following the law of total probability, the ﬁrst-orderstochastic dominance between busy times of an M/M/c queue, and the concentration inequality forindependent samples.
Revised: This results are derived following the law of total probability, the ﬁrst-orderstochastic dominance between busy times of an M/M/c queue, and the concentration inequality forindependent samples.

Original: We denote node i is a prerequisite node of i (cid:48) , i < i (cid:48) if there exists a productrequires to use nodes i and i (cid:48) .
Revised: We denote node i is a prerequisite node of i (cid:48) , i < i (cid:48) if there exists a productrequires to use nodes i and i (cid:48) .

Original: Thus, it is easy to show that the prerequisite nodes of a Layer- l nodemust have smaller layer indices.
Revised: Thus, it is easy to show that the prerequisite nodes of a Layer- l nodemust have smaller layer indices.

Original: In our Jackson Network, node i (cid:48) reaches its steady-state indicates that the arrival process of nodei (cid:48) is a stable Poisson process, and therefore it requires that all the prerequisite nodes also reach thesteady-state.
Revised: In our Jackson Network, node i (cid:48) reaches its steady-state indicates that the arrival process of nodei (cid:48) is a stable Poisson process, and therefore it requires that all the prerequisite nodes also reach thesteady-state.

Original: In this proof, we analyze the nodes layer by layer.
Revised: In this proof, we analyze the nodes layer by layer.

Original: By treating the nodes in the samelayer together, we can derive that
Revised: By treating the nodes in the samelayer together, we can derive that

Original: [Equation]
Revised: [Equation]

Original: In short, we derive a lower bound of this mixing probability by requiring the Layer- l nodes reachthe steady-state within the ﬁrst L × τ periods, conditional on that Layer- l (cid:48) nodes also reach thesteady-state within the ﬁrst l (cid:48) × τ periods for l (cid:48) < l , or equivalently, all prerequisite nodes of Layer- lnodes reach the steady-state within ( l − 1) × τ periods.
Revised: In short, we derive a lower bound of this mixing probability by requiring the Layer- l nodes reachthe steady-state within the ﬁrst L × τ periods, conditional on that Layer- l (cid:48) nodes also reach thesteady-state within the ﬁrst l (cid:48) × τ periods for l (cid:48) < l , or equivalently, all prerequisite nodes of Layer- lnodes reach the steady-state within ( l − 1) × τ periods.

Original: Layer-1 Nodes.
Revised: Layer-1 Nodes.

Original: In the ﬁrst τ periods, by applying the probability bound for an M/M/c queue, wecan easily conclude that all Layer-1 nodes (the correspondent M/M/c queues) reach the steady-statewith probability 1 − 2 T 3 .
Revised: In the ﬁrst τ periods, by applying the probability bound for an M/M/c queue, wecan easily conclude that all Layer-1 nodes (the correspondent M/M/c queues) reach the steady-statewith probability 1 − 2 T 3 .

Original: Recall I 1 is the collection of all Layer-1 nodes and |I 1 | is the cardinality.
Revised: Recall I 1 is the collection of all Layer-1 nodes and |I 1 | is the cardinality.

Original: Therefore we can compute that the probability that all Layer-1 nodes reach the steady-state by ﬁrst τperiods is
Revised: Therefore we can compute that the probability that all Layer-1 nodes reach the steady-state by ﬁrst τperiods is

Original: [Equation]
Revised: [Equation]

Original: because the queueing evolution of all Layer-1 nodes are independent with each other and thus we cansimplify multiply the probability together, in the ﬁrst equality in the above expression.
Revised: because the queueing evolution of all Layer-1 nodes are independent with each other and thus we cansimplify multiply the probability together, in the ﬁrst equality in the above expression.

Original: Layer-2 Nodes.
Revised: Layer-2 Nodes.

Original: We then analyze the Layer-2 nodes.
Revised: We then analyze the Layer-2 nodes.

Original: Consider one arbitrary layer-2 node, denoted byi without loss of generality.
Revised: Consider one arbitrary layer-2 node, denoted byi without loss of generality.

Original: We then focus on the corresponding queue of this node i conditional onthat all Layer-1 nodes have reached steady-state.
Revised: We then focus on the corresponding queue of this node i conditional onthat all Layer-1 nodes have reached steady-state.

Original: We ﬁrstly show a virtual queue can reach steadystate within the next τ periods with probability at least 1 − 2 T 2 .
Revised: We ﬁrstly show a virtual queue can reach steadystate within the next τ periods with probability at least 1 − 2 T 2 .

Original: Then we show that an alternativequeue of the actual queue takes a shorter time to reach steady-state and thus the actual queue can alsoreach steady-state within the next τ periods with probability at least 1 − 2 T 2 .•
Revised: Then we show that an alternativequeue of the actual queue takes a shorter time to reach steady-state and thus the actual queue can alsoreach steady-state within the next τ periods with probability at least 1 − 2 T 2 .•

Original: Virtual queue.
Revised: Virtual queue.

Original: We consider a virtual queue which rejects all customers arrived before τperiods.
Revised: We consider a virtual queue which rejects all customers arrived before τperiods.

Original: Then by Lemma 3, this virtual queue reaches the steady-state with probability1 − 2 T 2 after the next τ periods, i.e., after the ﬁrst 2 τ periods.•
Revised: Then by Lemma 3, this virtual queue reaches the steady-state with probability 1 − 2 T 2 after the next τ periods, i.e., after the ﬁrst 2 τ periods.

Original:  Alternative queue.
Revised: • Alternative queue.

Original: Denote the probability that the number of customers in Node i at theend of period τ is less than N by P i ( N ) , including the customers both using the serveror in the waiting queue of node i .
Revised: Denote the probability that the number of customers in Node i at the end of period τ is less than N by P i ( N ) , including the customers both using the serveror in the waiting queue of node i .

Original: The actual queue accepts customers during the ﬁrst τperiods with a changing rate from an initial rate to the stable arrival rate
Revised: The actual queue accepts customers during the ﬁrst τperiods with a changing rate from an initial rate to the stable arrival rate

Original: λ i (conditionalon all Layer-1 nodes reach steady-state and thus provides stable Poisson arrivals for nodeswith high layer indices).
Revised: λ i (conditionalon all Layer-1 nodes reach steady-state and thus provides stable Poisson arrivals for nodeswith high layer indices).

Original: Denote the demand of all products to the Jackson Network byα .
Revised: Denote the demand of all products to the Jackson Network by α .

Original: The stable arrival rate λ i = αp si + α
Revised: The stable arrival rate λ i = αp si + α

Original: (cid:80) i (cid:48) :(
Revised: (cid:80) i (cid:48) :(

Original: i (cid:48) ,i ∈ E ) p si (cid:48) p i (cid:48) i , where αp si is the initial rate,i.e., the demand rate of products that request node i as the ﬁrst service, and αp si (cid:48)
Revised: i (cid:48) ,i ∈ E ) p si (cid:48) p i (cid:48) i , where αp si is the initial rate, i.e., the demand rate of products that request node i as the ﬁrst service, and αp si (cid:48)

Original: p i (cid:48)
Revised: p i (cid:48)

Original: i is thestable demand ﬂowing from prerequisite node i (cid:48) to node i .
Revised: i is thestable demand ﬂowing from prerequisite node i (cid:48) to node i .

Original: Consider an alternative queuewhich accepts customers at the stable rate
Revised: Consider an alternative queuewhich accepts customers at the stable rate

Original: λ i .
Revised: λ i .

Original: If the queue starts from time 0 , then theprobability that this virtual queue has customers with a size less than N at the end of periodτ is smaller than P i ( N ) .
Revised: If the queue starts from time 0 , then theprobability that this virtual queue has customers with a size less than N at the end of periodτ is smaller than P i ( N ) .

Original: This is because this alternative queue accepts more customerswhile the service times are all following a same distribution.
Revised: This is because this alternative queue accepts more customers while the service times are all following a same distribution.

Original: Then we postpone the startingpoint of this alternative queue to a time stamp between 0 and τ such that the probabilitythat this alternative queue has customers with a size less than N at the end of period τis the same as P i ( N ) .
Revised: Then we postpone the startingpoint of this alternative queue to a time stamp between 0 and τ such that the probabilitythat this alternative queue has customers with a size less than N at the end of period τis the same as P i ( N ) .

Original: Note that this time stamps is exist, which can be easily proved bythe memoryless property of the exponential service time assumptions.
Revised: Note that this time stamps is exist, which can be easily proved by the memoryless property of the exponential service time assumptions.

Original: Then we have thefollowing two results.
Revised: Then we have the following two results.

Original: First, this alternative queue has a same evolution path as the actualqueue.
Revised: First, this alternative queue has a same evolution path as the actualqueue.

Original: Second, because this alternative queue starts accepting customers earlier than time τ ,and thus it will reach steady-state earlier than the virtual queue.
Revised: Second, because this alternative queue starts accepting customers earlier than time τ ,and thus it will reach steady -state earlier than the virtual queue.

Original: We provide an illustration of the actual, alternative, and virtual queues in Figure 3.
Revised: We provide an illustration of the actual, alternative, and virtual queues in Figure 3.

Original: To conclude, we have the actual queue can reach the steady-state with a higher probability after theﬁrst 2 τ periods.
Revised: To conclude, we have the actual queue can reach the steady-state with a higher probability after theﬁrst 2 τ periods.

Original: Similarly, the queueing evolution of all Layer-2 nodes are independent with eachother and thus we can simplify multiply the probability together.
Revised: Similarly, the queueing evolution of all Layer-2 nodes are independent with eachother and thus we can simplify multiply the probability together.

Original: Therefore, we have that for all
Revised: Therefore, we have that for all

Original: Layer-2 nodes
Revised: Layer-2 nodes

Original: [Equation]
Revised: [Equation]

Original: Layers with Higher Indices.
Revised: Layers with Higher Indices.

Original: Then we apply the same analysis steps of the Layer-2 nodes to nodesin layers with higher indices.
Revised: Then we apply the same analysis steps of the Layer-2 nodes to nodesin layers with higher indices.

Original: Conditional on the events that all nodes in layers with lower indiceshave reached the steady-state by the end of ( l − 1) τ periods, we can compute that the probability theall nodes of layer l can reach steady-state in the next τ period with a probability:
Revised: Conditional on the events that all nodes in layers with lower indiceshave reached the steady-state by the end of ( l − 1) τ periods, we can compute that the probability the all nodes of layer l can reach steady-state in the next τ period with a probability:

Original: [Equation]
Revised: [Equation]

Original:  to 
Revised: With the results in (9), (10), and (11), we are ready to bound the probability in (8):

Original: [Equation]
Revised: [Equation]

Original: We refer to Figure 4 for a direct representation of the above proof sketch, which shows the mixingtime analysis of an illustrative example.
Revised: We refer to Figure 4 for a direct representation of the above proof sketch, which shows the mixingtime analysis of an illustrative example.

Original: [Figure]
Revised: Q.E.D.

Original: Proof of Proposition 4.
Revised: Proof of Proposition 4.

Original: Similar to the proof of Proposition 4, we analyze the mixing time of nodes layer by layer.
Revised: Similar to the proof of Proposition 4, we analyze the mixing time of nodes layer by layer.

Original: Before conducting the proof, we introduce an existing work for an M/M/c queue mixing time.
Revised: Before conducting the proof, we introduce an existing work for an M/M/c queue mixing time.

Original: We similarly deﬁne the following events when the Jackson Network starts to operate under a new price starting from the steady-state of the old price:
Revised: We similarly deﬁne the following events when the Jackson Network starts to operate under a new price starting from the steady-state of the old price:

Original: [Equation]
Revised: [Equation]

Original: Lemma 4 says that for one M/M/c queue where Assumption 5 holds, then the probability that thequeue reaches a new steady-state from a old steady-state within 2 τ time periods after the price changeis bounded by 1 − 4 /T 2 .
Revised: Lemma 4 says that for one M/M/c queue where Assumption 5 holds, then the probability that thequeue reaches a new steady-state from a old steady-state within 2 τ time periods after the price changeis bounded by 1 − 4 /T 2 .

Original: To derive this result, the authors decompose possible price changes intothe following two cases.
Revised: To derive this result, the authors decompose possible price changes intothe following two cases.

Original: The ﬁrst is when the arrival rate changes from a higher value to a lowervalue and the second is vice versa.
Revised: The ﬁrst is when the arrival rate changes from a higher value to a lowervalue and the second is vice versa.

Original: For each case, the authors further consider two sub-cases: the unitservice rate increase or decrease.
Revised: For each case, the authors further consider two sub-cases: the unitservice rate increase or decrease.

Original: Consider the Jackson Network system in our problem.
Revised: Consider the Jackson Network system in our problem.

Original: We can bound the probability by:
Revised: We can bound the probability by:

Original: [Equation]
Revised: [Equation]

Original: The above inequality is derived following the same steps in (7) and we neglect the details here.
Revised: The above inequality is derived following the same steps in (8) and we neglect the details here.

Original: Assumption 5 allows us to apply Lemma 4 when we analyze one speciﬁc node, i.e., one speciﬁc
Revised: Assumption 5 allows us to apply Lemma 4 when we analyze one speciﬁc node, i.e., one speciﬁc

Original: M/M/c queue.
Revised: M/M/c queue.

Original: The structure of the following analysis is similar to the structure of the proof of
Revised: The structure of the following analysis is similar to the structure of the proof of

Original: Proposition 3, we ﬁrst analyze Layer-1 node and then the nodes with higher layer indices.
Revised: Proposition 3, we ﬁrst analyze Layer-1 node and then the nodes with higher layer indices.

Original: For nodeswith higher layer indices, we apply the aforementioned proposition to a virtual queue, and then showthis virtual queue takes longer than an alternative system with the actual queue.
Revised: For nodeswith higher layer indices, we apply the aforementioned proposition to a virtual queue, and then showthis virtual queue takes longer than an alternative system with the actual queue.

Original: Layer-1 Nodes.
Revised: Layer-1 Nodes.

Original: In the ﬁrst 2 τ periods, by applying the probability bound for an M/M/c queue withprice change (Lemma 4), we can easily conclude that all Layer-1 nodes (the correspondent M/M/cqueues) reach the steady-state with probability 1 − 4 T 2 .
Revised: In the ﬁrst 2 τ periods, by applying the probability bound for an M/M/c queue withprice change (Lemma 4), we can easily conclude that all Layer-1 nodes (the correspondent M/M/cqueues) reach the steady-state with probability 1 − 4 T 2 .

Original: Consider the probabilistic independency of thecoupling process among Layer-1 nodes, the probability that all Layer-1 nodes reach the steady-stateby ﬁrst 2 τ periods is
Revised: Consider the probabilistic independency of thecoupling process among Layer-1 nodes, the probability that all Layer-1 nodes reach the steady-stateby ﬁrst 2 τ periods is

Original: [Equation]
Revised: [Equation]

Original: Layer-2 Nodes and Nodes with Higher Layer Indices.
Revised: Layer-2 Nodes and Nodes with Higher Layer Indices.

Original: We then analyze nodes with a layer index 2or higher.
Revised: We then analyze nodes with a layer index 2or higher.

Original: Consider one arbitrary Layer-l node, denoted by i without loss of generality.
Revised: Consider one arbitrary Layer-l node, denoted by i without loss of generality.

Original: We then focuson the corresponding queue of this node i conditional on that all prerequisites nodes have reachedsteady-state.
Revised: We then focuson the corresponding queue of this node i conditional on that all prerequisites nodes have reachedsteady-state.

Original: We ﬁrstly show a virtual queue can reach steady-state within the next 2 τ periods withprobability at least 1 − 4 T 2 .
Revised: We ﬁrstly show a virtual queue can reach steady-state within the next 2 τ periods withprobability at least 1 − 4 T 2 .

Original: Then we show that an alternative queue of the actual queue takes a shortertime to reach steady-state and thus the actual queue can also reach steady-state within the next 2 τperiods with probability at least  − 4 T 2 .
Revised: Then we show that an alternative queue of the actual queue takes a shortertime to reach steady-state and thus the actual queue can also reach steady-state within the next 2 τperiods with probability at least 1 − 4 T 2 .

Original: • Virtual queue.
Revised: • Virtual queue.

Original: We consider a virtual queue which accepts customers under the old queueing
Revised: We consider a virtual queue which accepts customers under the old queueing

Original: parameters in the ﬁrst 2( l − 1) τ periods and then accepts customers under the new queueing
Revised: parameters in the ﬁrst 2( l − 1) τ periods and then accepts customers under the new queueing

Original: parameter in the next 2 τ periods.
Revised: parameter in the next 2 τ periods.

Original: With Assumption5, we can directly apply the results forone M/M/c queue to reach steady-state starting from a steady-state and conclude that this
Revised: With Assumption we can directly apply the results for one M/M/c queue to reach steady-state starting from a steady-state and conclude that this

Original: virtual queue reaches the steady-state with probability 1 − 4 T 2 after the next 2 τ periods, i.e.,after the ﬁrst 2 lτ periods.
Revised: virtual queue reaches the steady-state with probability 1 − 4 T 2 after the next 2 τ periods, i e., after the ﬁrst 2 lτ periods

Original: • Alternative queue.
Revised: • Alternative queue.

Original: Denote the probability that in Node i at the end of 2( l − 1) τ periods, thenumber of old customers is less than N o and the number of new customers is less than N n by
Revised: Denote the probability that in Node i at the end of 2( l − 1) τ periods, the number of old customers is less than N o and the number of new customers is less than N n by

Original: P i ( N o , Nn ) , including the customers both using the server or in the waiting queue of node i . Consider an alternative queue which starts accepting new customers under the new queueingsystem parameter from time
Revised: P i ( N o , N

Original: and denote the corresponding probability for having less than N o
Revised: n ) , including the customers both using the server or in the waiting queue of node i .

Original:  number of old customers and less than N n new customers at the end of 2( l − 1) τperiods by P (cid:48)
Revised: Consider an alternative queue which starts accepting new customers under the new queueingsystem parameter from time 0 and denote the corresponding probability for having lessthan N o number of old customers and less than N n new customers at the end of 2( l − 1) τperiods by P (cid:48)

Original: i ( N o , N n ) .
Revised: i ( N o , N n ) .

Original: By the analysis in Proposition 3, we know there is a mismatchbetween P
Revised: By the analysis in Proposition 3, we know there is a mismatchbetween P

Original: i
Revised: i

Original: ( N o , N n ) .
Revised: ( N o , N n ) .

Original: Then we postpone the starting point of this alternative queue to atime stamp between 0 and 2( L − 1) τ such that P (cid:48) i ( N o , N n )
Revised: Then we postpone the starting point of this alternative queue to atime stamp between 0 and 2( L − 1) τ such that P (cid:48) i ( N o , N n )

Original: = P i ( N o , N n ) .
Revised: = P i ( N o , N n ) .

Original: Note that thistime stamps is always exist, which can be easily proved by the memoryless property of theexponential service time assumptions because we can always ﬁnd this stamp backwardsfrom any customer state at time 2( l − 1) τ .
Revised: Note that this time stamps is always exist, which can be easily proved by the memoryless property of theexponential service time assumptions because we can always ﬁnd this stamp backwardsfrom any customer state at time 2( l − 1) τ .

Original: Then we have the following two results.
Revised: Then we have the following two results.

Original: First, this alternative queue has a same evolution path as the actual queue.
Revised: First,this alternative queue has a same evolution path as the actual queue.

Original: Second, because thisalternative queue starts accepting new customers earlier than time 2( l − 1) τ , and thus it willreach steady-state earlier than the virtual queue.
Revised: Second, because thisalternative queue starts accepting new customers earlier than time 2( l − 1) τ , and thus it willreach steady-state earlier than the virtual queue.

Original: To conclude, we have the actual queue can reach the steady-state with a higher probability after theﬁrst 2 τ periods.
Revised: To conclude, we have the actual queue can reach the steady-state with a higher probability after theﬁrst 2 τ periods.

Original: Therefore , conditional on the events that all nodes in layers with lower indices havereached the steady-state by the end of 2( l − 1) τ periods, we can compute that the probability the allnodes of layer l can reach steady-state in the next 2 τ period with a probability:
Revised: Therefore , conditional on the events that all nodes in layers with lower indices havereached the steady-state by the end of 2( l − 1) τ periods, we can compute that the probability the allnodes of layer l can reach steady-state in the next 2 τ period with a probability:

Original: [Equation]
Revised: [Equation]

Original: With the results in (12) and (13), we are ready to bound the probability in (11):
Revised: With the results in (13) and (14), we are ready to bound the probability in (12):

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: We bound the relaxed regret during the Warm-up and Learning Phases separately.
Revised: We bound the relaxed regret during the Warm-up and Learning Phases separately.

Original: We furtherdecompose the relaxed regret during the Learning Phase into two parts.
Revised: We furtherdecompose the relaxed regret during the Learning Phase into two parts.

Original: The ﬁrst part is the regret ofsuboptimality, which is induced by not offering the state-independent optimal price in certain timeperiods.
Revised: The ﬁrst part is the regret ofsuboptimality, which is induced by not offering the state-independent optimal price in certain timeperiods.

Original: We bound the loss due to suboptimality with the help of Proposition 2.
Revised: We bound the loss due to suboptimality with the help of Proposition 2.

Original: The second part isthe regret of nonstationarity, which represents the revenue loss induced by changing offered prices.
Revised: The second part isthe regret of nonstationarity, which represents the revenue loss induced by changing offered prices.

Original: We bound the loss due to nonstationarity with the help of Propositions 3 and 4.
Revised: We bound the loss due to nonstationarity with the help of Propositions 3 and 4.

Original: The structure of thisproof is similar to that of Theorem 1 in Jia et al.
Revised: The structure of thisproof is similar to that of Theorem 1 in Jia et al.

Original: (2020), where the authors consider a single reusableproduct setting.
Revised: (2020), where the authors consider a single reusableproduct setting.

Original: Regret of the Warm-up Phase.
Revised: Regret of the Warm-up Phase.

Original: The difference between Y LPWarm-up and Y RNRM-UCBWarm-up is at most linearon the length of the Warm-up Phase, which is O ( X log( J 1 / 4 T )) .
Revised: The difference between Y LPWarm-up and Y RNRM-UCBWarm-up is at most linearon the length of the Warm-up Phase, which is O ( X log( J 1 / 4 T )) .

Original: Therefore we can directly have
Revised: Therefore we can directly have

Original: Regret Warm-up ( RNRM
Revised: Regret Warm-

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Regret of nonstationarity during the Learning Phase.
Revised: Regret of nonstationarity during the Learning Phase.

Original: We denote the offered price of batch m in
Revised: We denote the offered price of batch m in

Original: RNRM-UCB as x m .
Revised: RNRM-UCB as x m .

Original: Therefore, the expected revenue collected in batch m is
Revised: Therefore, the expected revenue collected in batch m is

Original: [Equation]
Revised: [Equation]

Original: where Y RNRM-UCB m, 0 ∼ 2 Lτ ( x m ) denotes the expected revenue collected in the ﬁrst 2 Lτ periods and
Revised: where Y RNRM-UCB m, 0 ∼ 2 Lτ ( x m ) denotes the expected revenue collected in the ﬁrst 2 Lτ periods and

Original: Y RNRM-UCB m, 2 Lτ ∼
Revised: Y RNRM-UCB m, 2 Lτ ∼

Original: I m Lτ ( x m ) denotes the expected revenue collected from (2 Lτ + 1) th period to the endof batch m .
Revised: I m Lτ ( x m ) denotes the expected revenue collected from (2 Lτ + 1) th period to the endof batch m .

Original: We deﬁne event C 2 Lτ as that the network service system reaches the steady-state underthe new price within the ﬁrst 2 Lτ periods and deﬁne ¯ C 2 Lτ as the complementary event of C 2 Lτ .
Revised: We deﬁne event C 2 Lτ as that the network service system reaches the steady-state underthe new price within the ﬁrst 2 Lτ periods and deﬁne ¯ C 2 Lτ as the complementary event of C 2 Lτ .

Original: By
Revised: By

Original: Propositions 3 and 4, we have P [ C 2 Lτ ] ≥ 1 − 4 I/T 2 .
Revised: Propositions 3 and 4, we have P [ C 2 Lτ ] ≥ 1 − 4 I/T 2 .

Original: Consequently, we deﬁne Y RNRM-UCB m, C 2 Lτ ( x m ) tobe the expected value of Y RNRM-UCB m, 2 Lτ ∼
Revised: Consequently, we deﬁne Y RNRM-UCB m, C 2 Lτ ( x m ) tobe the expected value of Y RNRM-UCB m, 2 Lτ ∼

Original: I m Lτ ( x m ) conditional on C 2 Lτ and Y RNRM-UCB m, ¯ C 2 Lτ ( x m ) similarly.
Revised: I m Lτ ( x m ) conditional on C 2 Lτ and Y RNRM-UCB m, ¯ C 2 Lτ ( x m ) similarly.

Original: Asa result, we can express the expected revenue during batch m as:
Revised: Asa result, we can express the expected revenue during batch m as:

Original: [Equation]
Revised: [Equation]

Original: Regret of suboptimality during the Learning Phase.
Revised: Regret of suboptimality during the Learning Phase.

Original: Recall the upper and lower conﬁdence bounds of the revenue rate deﬁned in Deﬁnition 2.
Revised: Recall the upper and lower conﬁdence bounds of the revenue rate deﬁned in Deﬁnition 2.

Original: It is clear that
Revised: It is clear that

Original: [Equation]
Revised: [Equation]

Original: For batch m , we can compute the difference in the revenue rates between the optimal price andselected price x m as
Revised: For batch m , we can compute the difference in the revenue rates between the optimal price andselected price x m as

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds because U m ( x m ) ≥ U m (˜ x ) by the algorithm construction.
Revised: where the last inequality holds because U m ( x m ) ≥ U m (˜ x ) by the algorithm construction.

Original: Then the cumulative regret during the Learning Phase is
Revised: Then the cumulative regret during the Learning Phase is

Original: [Equation]
Revised: [Equation]

Original: By (16), we have
Revised: By (17), we have

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst analyze the middle term of (18).
Revised: We ﬁrst analyze the middle term of (19).

Original: Let M x denote the batches choosing price x and T x thenumber of periods choosing price x , i.e., T x = (cid:80) m
Revised: Let M x denote the batches choosing price x and T x thenumber of periods choosing price x , i.e., T x = (cid:80) m

Original: ∈M x
Revised: ∈M x

Original: I m Lτ .
Revised: I m Lτ .

Original: Then,
Revised: Then,

Original: M (cid:88) m =

Original: [Equation]

Original: O

Original: [Equation]

Original: x

Original: [Equation]
Revised: [Equation]

Original: x

Original: [Equation]

Original: (cid:112)

Original: [Equation] (19e) [Equation] where (19c) by Lemma 2, (19e) is derived by the number of the batches, and (19f) follows Jensen’sinequality.
Revised: where (20c) by Lemma 2, (20e) is derived by the number of the batches, and (20f) follows Jensen’sinequality.

Original: The number of batches is log 2 (cid:16) 1 + T
Revised: The number of batches is log 2 (cid:16) 1 + T

Original: − 8 X log
Revised: − 8 X log

Original: ( J 1 / 4 T ) 2 L log( T )(cid:17) =
Revised: ( J 1 / 4 T ) 2 L log( T )(cid:17) [Equation]

Original: O (cid:0) log( T ) (cid:1) .
Revised: O (cid:0) log( T ) (cid:1) .

Original: Below we analyze the ﬁrst term of (18), and the same logic applies to the third term, of which weomit the details here.
Revised: Below we analyze the ﬁrst term of (19), and the same logic applies to the third term, of which weomit the details here.

Original: [Equation]
Revised: [Equation]

Original: ) where (20e) is because U m (˜ x ) > 0 by Deﬁnition 2, and (20f) follows Proposition 2.
Revised: 21f) where (21e) is because U m (˜ x ) > 0 by Deﬁnition 2, and (21f) follows Proposition 2.

Original: Given that O (cid:0) n Mj ( x ) (cid:1) =
Revised: Given that O (cid:0) n Mj ( x ) (cid:1) =

Original: O ( T x ) , for any product j ∈ J on ( x ) , we have (cid:80) Mm =
Revised: O ( T x ) , for any product j ∈ J on ( x ) , we have (cid:80) Mm =

Original: 1 O ( Lτ ) =
Revised: 1 O ( Lτ ) =

Original: O (cid:0) L (log( T )) 3 (cid:1) .
Revised: O (cid:0) L (log( T )) 3 (cid:1) .

Original: Combining the three terms together,
Revised: Combining the three terms together,

Original: [Equation]
Revised: [Equation]

Original: Total Regret of RNRM-UCB algorithm.
Revised: Total Regret of RNRM-UCB algorithm.

Original: By adding up the relaxed regret during the Warm-up and 702 Learning Phases, we have the regret of RNRM-UCB algorithm as:[Equation] 22c
Revised: By adding up the relaxed regret during the Warm-up and 722 Learning Phases, we have the regret of RNRM-UCB algorithm as:

Original: [Equation]
Revised: [Equation]

Original: Q.
Revised: Q.

Original: [Equation]
Revised: [Equation]

Original: .D
Revised: .D

Original: [Equation]
Revised: [Equation]

Original: Proof of 
Revised: Proof of Theorem

Original: Theorem 2.

Original: The analysis here is similar to that in the proof of Theorem 1.
Revised: The analysis here is similar to that in the proof of Theorem 1.

Original: We ﬁrst bound the Bayesian regret ofthe Warm-up Phase and then for the Learning Phase.
Revised: We ﬁrst bound the Bayesian regret ofthe Warm-up Phase and then for the Learning Phase.

Original: Bayesian regret in the Warm-up Phase.
Revised: Bayesian regret in the Warm-up Phase.

Original: Similarly, we can bound the Bayesian regret in the
Revised: Similarly, we can bound the Bayesian regret in the

Original: Warm-up Phase as
Revised: Warm-up Phase as

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Bayesian regret in the Learning Phase.
Revised: Bayesian regret in the Learning Phase.

Original: In batch m , the chosen price x m and the state-independentoptimal price ˜ x are identically distributed conditional on F m , which indicates that E (cid:2)
Revised: In batch m , the chosen price x m and the state-independentoptimal price ˜ x are identically distributed conditional on F m , which indicates that E (cid:2)

Original: U m ( x m ) −
Revised: U m ( x m ) −

Original: U m (˜ x ) |F m (cid:3) = 

Revised: U m (˜ x ) |F m (cid:3) =

Original: (e.g., see the proof of Lemma 3.7 in Slivkins (2019)).
Revised: (e.g., see the proof of Lemma 3.7 in Slivkins (2019)).

Original: Thus, the Bayesian regret in batch m is:
Revised: Thus, the Bayesian regret in batch m is:

Original: [Equation]
Revised: [Equation]

Original: where (24g) is derived following the analysis of nonstationarity in (15).
Revised: where (25g) is derived following the analysis of nonstationarity in (16).

Original: Derived from Proposition 2, for some γ > 0 , two properties hold (see, e
Revised: Derived from Proposition 2, for some γ > 0 , two properties hold (see, e

Original: [Equation]
Revised: [Equation]

Original: g., Slivkins 2019):
Revised: g., Slivkins 2019):

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Following these two properties of the stationary revenue rate in (25)
Revised: Following these two properties of the stationary revenue rate in (26)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and similarly,
Revised: and similarly,

Original: [Equation]
Revised: [Equation]

Original: Summing over all batches m, ∀ m = 1 , . .
Revised: Summing over all batches m, ∀ m = 1 , . .

Original: , M , we have the cumulative relaxed Bayesian regret bythe end of the Learning Phase as:
Revised: , M , we have the cumulative relaxed Bayesian regret bythe end of the Learning Phase as:

Original: [Equation]
Revised: [Equation]

Original: where (28e) is by Lemma 2 and (28f) is by Jensen’s inequality.
Revised: where (29e) is by Lemma 2 and (29f) is by Jensen’s inequality.

Original: Regret of RNRM-TS algorithm.
Revised: Regret of RNRM-TS algorithm.

Original: By adding up the relaxed regret during the Warm-up and Learning
Revised: By adding up the relaxed regret during the Warm-up and Learning

Original: Phases, we have the regret of RNRM-TS algorithm as:
Revised: Phases, we have the regret of RNRM-TS algorithm as:

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: For any price x ∈ X ,(cid:88)
Revised: For any price x ∈ X ,(cid:88)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: Plugging the deﬁnition of Rad m ( x ) , ∀ m = 1 , . .
Revised: Plugging the deﬁnition of Rad m ( x ) , ∀ m = 1 , . .

Original: , M , one has:
Revised: , M , one has:

Original: [Equation]
Revised: [Equation]

Original: 31g) where and (31b) is by n mj ( x ) =
Revised: 32g) where and (32b) is by n mj ( x ) =

Original: O ( I m Lτ ) , ∀ j ∈ J on ( x ) and (31e) is by Jensen’s inequality.
Revised: O ( I m Lτ ) , ∀ j ∈ J on ( x ) and (32e) is by Jensen’s inequality.

Original: Lemma 3.
Revised: Lemma 3.

Original: (Coupling probability of M/M/c queue starting from empty state, Proposition 4 in
Revised: (Coupling probability of M/M/c queue starting from empty state, Proposition 4 in

Original: Jia et al. 2020)
Revised: Jia et al. 2020)

Original: For τ = (log T ) 2 , if an M/M/c queue starts from empty state at time 0 , then theprobability that the system reaches the steady-state within τ is bounded by 1 − 2 T 2 .
Revised: For τ = (log T ) 2 , if an M/M/c queue starts from empty state at time 0 , then theprobability that the system reaches the steady-state within τ is bounded by 1 − 2 T 2 .

Original: Lemma 4.
Revised: Lemma 4.

Original: (Coupling probability of M/M/c queue when price changes, Proposition 5 in Jia et al.2020)
Revised: (Coupling probability of M/M/c queue when price changes, Proposition 5 in Jia et al.2020)

Original: For τ = (log T ) 2 , if an M/M/c queue starts from steady-state and changes system parameter attime 0 , then the probability that the system reaches the steady-state within 2 τ is bounded by 1 − 4 T 2 .
Revised: For τ = (log T ) 2 , if an M/M/c queue starts from steady-state and changes system parameter attime 0 , then the probability that the system reaches the steady-state within 2 τ is bounded by 1 − 4 T 2 .

Revised: F Service Duration Dependent Payment Model

Revised: We consider a new variant payment model where the customer pays the product’s per-unit-time pricemultiplied by its total service duration.

Revised: The ﬁrm then needs to decide the per-unit-time price for eachproduct j ∈ J .

Revised: Under this payment setting, we modify the deﬁnition of the steady-state revenue rate,derive a new concentration bound of historical observations and actual system parameters, and thencorrespondingly develop new UCB of the revenue rate of each price.

Revised: With those modiﬁed deﬁnitions,we present a new RNRM-UCB variant and show that it achieves a same regret bound.

Revised: Data collection.

Revised: Similar to n j , ˆ d jk ( x ) , and ¯ d j ( x ) , we also collect service time observations overresources.

Revised: We use n ri ( x ) to denote the number of customers who successfully consume resource iunder price x .

Revised: We use ˆ g ik ( x ) to denote the service time of customer k over resource i under the pricex .

Revised: We use ¯ g i ( x ) to denote the empirical mean of the service time and ¯ g i ( x ) =

Revised: (cid:80) n ri ( x ) k

Original: {+=
+}
Revised: {+=
+}

Revised: ˆ g ik ( x ) /n ri ( x ) ,for

Revised: i ∈

Revised: I .

Revised: We further assume that there is a known upper bound r max = max x,i,j α j /µ i , n min j ( x )

Revised: =min { n j ( x ) , min i : a ji =1 n si ( x ) } .

Revised: Revenue rate.

Revised: Under this duration-dependent payment model, we offer unit price x ∈ R J to products

Revised: J .

Revised: Customer k who requests to use product j pays the per-unit-time price multiplied by the totalservice duration, i.e., x j (cid:80) i : a ji

Revised: =1 ˆ g ik ( x ) , to the ﬁrm.

Revised: Different than the lump sum payment model,the steady-state revenue rate for this setting is r v ( x ) =

Original: {+(cid:80) j ∈J on ( x ) α j x j (cid:80) i : a ji =1 
+}
Revised: {+(cid:80) j ∈J on ( x ) α j x j (cid:80) i : a ji =1 
+}

Revised: /µ i ( x ) .

Revised: Proposition 5.

Revised: For any price x ∈ X , if the number of arrival time observations for each turn-on product n j ( x ) ≥ 8 log( L 14 J 14 T ) for j ∈ J on ( x ) and the number of service time observations for each resource n si ( x ) ≥ 8 log( L 1 4 J 1 4 T ) ∀ i ∈ { i ∈ I : (cid:80) j ∈J on ( x ) a ji ≥ 1 } , then we have

Revised: P 

Revised: [Equation]

Revised: (cid:113)

Revised: [Equation]

Revised: where r max is a known upper bound with r max = max x,i,j α j /µ i , n min j ( x )

Revised: =min { n j ( x ) , min i : a ji =1 n si ( x ) } , and L is the number of layers.

Revised: Proof of Proposition 5.

Revised: The above probability can be bounded by the product of the following probabilities from below.

Revised: Firstly we can bound that for i, j with a ji = 1 :

Revised: [Table]

Revised: 

Revised: [Table]

Revised: The ﬁrst and second probability in the above equation can be bounded with the same technique as that we used in (7) with Lemma 1 and thus we have

Revised: [Table]

Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: LJT

Revised: [Equation]

Revised: and

Revised: P

Revised: [Equation]

Revised: 

Revised: [Equation]

Revised: Therefore, we have the above probability can be bounded by

Revised: [Equation]

Revised: Then, for product j ∈ J on ( x ) , we can further bound

Revised: P 

Revised: [Equation]

Revised: ≥

Revised: [Equation]

Revised: i :

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: P 

Revised: [Equation]

Revised: µ

Revised: [Equation]

Revised: ≤ (

Revised: [Equation]

Revised: because (cid:80) i ∈I a ji ≤ L ∀ j ∈ J .

Revised: Now we are ready to derive the concentration bound.

Revised: [Table]

Revised: According to the above proposition

Revised: [Table]

Revised: we can deﬁne the conﬁdence radius as follows.

Revised: (We use the superscript v to denote this duration-dependent payment model.)

Revised: [Equation]

Revised: Therefore, the upper and lower conﬁdence bound of the revenue rate associated with each price is

Revised: U vt ( x

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: Algorithm design.

Revised: The algorithmic framework remains unchanged except for two modiﬁcations.(i)

Revised: In Step 6, the algorithm also requires the number of service time observations to be at least8 log( L 14 J 14 T ) .

Revised: (ii)

Revised: In Steps 13 and 16, the algorithm uses the new UCB deﬁnition.

Revised: Algorithm 3 RNRM-UCB (Duration-Dependent Payment Model.)

Revised: 1: Input

Revised: T

Revised: X

Revised: service

Revised: turn

Revised: off price.

Revised: Choose argmax x ∈ U vm − 1 ( x ) .

Revised: Offer x m in batch m , i.e., for I m Lτ periods.

Revised: Record inter-arrival time ˆ d jk ( x m ) for arriving customers of product j ∈ J on ( x m ) .

Revised: Compute and update U vm ( x m ) .

Revised: 17: end for

Revised: Regret analysis.

Revised: We change the constraints (1d) in the LP benchmark to

Revised: [Equation]

Revised: Under this duration-dependent payment setting, we can follow the same analysis technique to showthat this LP model also gives a valid upper bound of the optimal revenue.

Revised: The results of Propositions1, 3, and 4 and Theorem 1 remain the same for this duration-dependent payment model.