Original: Beyond Central Limit Theorem for Higher-Order Inference in Batched Bandits
Revised: Beyond Central Limit Theorem for Higher-Order Inference in Batched Bandits

Original: Abstract
Revised: Abstract

Original: Adaptive experiments have been gaining traction in a variety of domains, which stimulates a growing literature focusing on post-experimental statistical inference on data collected from such designs.
Revised: Adaptive experiments have been gaining traction in a variety of domains, which stimulates a growing literature focusing on post-experimental statistical inference on data collected from such designs.

Original: Prior work constructs confidence intervals mainly based on two types of methods: (i) martingale concentration inequalities and (ii) asymptotic approximation to distribution of test statistics; this work contributes to the second kind.
Revised: Prior work constructs confidence intervals mainly based on two types of methods: (i) martingale concentration inequalities and (ii) asymptotic approximation to distribution of test statistics; this work contributes to the second kind.

Original: The current asymptotic approximation methods however mostly rely on first-order limit theorems, which can have a slow convergence in a data-poor regime.
Revised: The current asymptotic approximation methods however mostly rely on first-order limit theorems, which can have a slow convergence in a data-poor regime.

Original: Besides, established results often rely on conditions that noises behave well, which can be problematic when the real-world instances are heavy-tailed or asymmetric.
Revised: Besides, established results often rely on conditions that noises behave well, which can be problematic when the real-world instances are heavy-tailed or asymmetric.

Original: In this paper, we propose a higher-order asymptotic expansion formula for inference on adaptively collected data, which generalizes normal approximation to the distribution of standard test statistics.
Revised: In this paper, we propose a higher-order asymptotic expansion formula for inference on adaptively collected data, which generalizes normal approximation to the distribution of standard test statistics.

Original: Our theorem relaxes assumptions on the noise distribution and benefits a higher-order approximation in the distributional distance to accommodate small sample sizes.
Revised: Our theorem relaxes assumptions on the noise distribution and benefits a higher-order approximation in the distributional distance to accommodate small sample sizes.

Original: We complement our results by promising empirical performances in simulations.
Revised: We complement our results by promising empirical performances in simulations.

Original:  Introduction
Revised:  Introduction

Original: Adaptive experimental designs such as bandit algorithms have received increasing popularity in many applications [13, 1, 7].
Revised: Adaptive experimental designs such as bandit algorithms have received increasing popularity in many applications [13, 1, 7].

Original: Instead of fixing a randomization rule, the experimenter progressively updates the data collection mechanism in response to past observations, so as to reduce experimental costs or optimize sample efficiency to test hypotheses.
Revised: Instead of fixing a randomization rule, the experimenter progressively updates the data collection mechanism in response to past observations, so as to reduce experimental costs or optimize sample efficiency to test hypotheses.

Original: With the increasing availability of data collected from such designs, we would like to understand the best practice to conduct post-experimental statistical inference.
Revised: With the increasing availability of data collected from such designs, we would like to understand the best practice to conduct post-experimental statistical inference.

Original: Specifically, we seek to evaluate alternative treatment assignment policies using data collected from batched multi-armed bandit algorithms and construct confidence intervals around the estimations.
Revised: Specifically, we seek to evaluate alternative treatment assignment policies using data collected from batched multi-armed bandit algorithms and construct confidence intervals around the estimations.

Original: We would like to do so in a data-efficient and robust way and accommondate regimes where the sample size may be small, and the noise may be heavy-tailed or asymmetric.
Revised: We would like to do so in a data-efficient and robust way and accommondate regimes where the sample size may be small, and the noise may be heavy-tailed or asymmetric.

Original: There are two types of approaches of inference on adaptive data in the current literature.
Revised: There are two types of approaches of inference on adaptive data in the current literature.

Original: The first kind is based on martingale concentration inequalities that result in valid confidence intervals for arbitrary stopping times, but such methods are often overly conservative and lose statistical power especially when sample size is small.
Revised: The first kind is based on martingale concentration inequalities that result in valid confidence intervals for arbitrary stopping times, but such methods are often overly conservative and lose statistical power especially when sample size is small.

Original: Alternatively, asymptotic approximation-based methods construct estimators that have desirable limiting distributions, which are then translated to asymptotically valid confidence intervals [5, 16, 8, 17].
Revised: Alternatively, asymptotic approximation-based methods construct estimators that have desirable limiting distributions, which are then translated to asymptotically valid confidence intervals [5, 16, 8, 17].

Original: We view our work as complimentary to this line of research.
Revised: We view our work as complimentary to this line of research.

Original: Closest to our setting of batched multi-armed bandits, [16] first construct a studentized statistic per batch and then aggregate them to obtain an estimator with asymptotically normal guarantees, which retains even with nonstationarity.
Revised: Closest to our setting of batched multi-armed bandits, [16] first construct a studentized statistic per batch and then aggregate them to obtain an estimator with asymptotically normal guarantees, which retains even with nonstationarity.

Original: Despite the growing research efforts, the current asymptotic approximation-based methods for post-experimental inference lack two things:
Revised: Despite the growing research efforts, the current asymptotic approximation-based methods for post-experimental inference lack two things:

Original: Robustness to distributional assumptions.
Revised: Robustness to distributional assumptions.

Original: Prior work has mainly relied on assumptions on sub-gaussian or symmetric noises, which is however problematic since such desirable properties may not be realized in many real-world applications.
Revised: Prior work has mainly relied on assumptions on sub-gaussian or symmetric noises, which is however problematic since such desirable properties may not be realized in many real-world applications.

Original: For example, treatment effect heterogeneity or truncation induces asymmetry in the noises.
Revised: For example, treatment effect heterogeneity or truncation induces asymmetry in the noises.

Original: Thus methods predicated upon such assumptions might not yield correct statistical properties in cases of heavy-tailed or asymmetric noises.
Revised: Thus methods predicated upon such assumptions might not yield correct statistical properties in cases of heavy-tailed or asymmetric noises.

Original: Sample size considerations.
Revised: Sample size considerations.

Original: The current asymptotic approximation methods for adaptive inference all rely on first-order limit theorems, a.k.a.
Revised: The current asymptotic approximation methods for adaptive inference all rely on first-order limit theorems, a.k.a.

Original: central limit theorem.
Revised: central limit theorem.

Original: This convergence is slow in a data-poor regime, which is typical if we want to evaluate sub-optimal arms using bandit data.
Revised: This convergence is slow in a data-poor regime, which is typical if we want to evaluate sub-optimal arms using bandit data.

Original: Bandit algorithms are often geared towards minimizing cumulative regret, which leads to meager data collection for arms that do not perform well during the first part of the experiment.
Revised: Bandit algorithms are often geared towards minimizing cumulative regret, which leads to meager data collection for arms that do not perform well during the first part of the experiment.

Original: This issue exacerbates as the number of arms increases.
Revised: This issue exacerbates as the number of arms increases.

Original: To provide intuition, Example 1 shows that batched OLS [16] generates a test statistic that is far from being normal with limited sample size and asymmetric noise.
Revised: To provide intuition, Example 1 shows that batched OLS [16] generates a test statistic that is far from being normal with limited sample size and asymmetric noise.

Original: Example 1.
Revised: Example 1.

Original: Consider a two-batch multi-armed bandit experiment run under an ϵ -greedy algorithm, where the noise is distributed as standardized Γ(2 , 1) with mean 0 and variance 1 .
Revised: Consider a two-batch multi-armed bandit experiment run under an ϵ -greedy algorithm, where the noise is distributed as standardized Γ(2 , 1) with mean 0 and variance 1 .

Original: Figure 1 presents the Monte Carlo distribution of the test statistic based on batched OLS [16], which is shown to be far from being normal due to the asymmetric and heavy-tailed noise.
Revised: Figure 1 presents the Monte Carlo distribution of the test statistic based on batched OLS [16], which is shown to be far from being normal due to the asymmetric and heavy-tailed noise.

Original: How should we tackle these problems?
Revised: How should we tackle these problems?

Original: The confidence sequence approach has an appeal of timeuniform valid inference, but past simulation studies suggest that they may be conservative with small sample size [16, 5].
Revised: The confidence sequence approach has an appeal of timeuniform valid inference, but past simulation studies suggest that they may be conservative with small sample size [16, 5].

Original: This paper provides an alternative method: if the first-order approximation is insufficient, why not resort to higher-orders?
Revised: This paper provides an alternative method: if the first-order approximation is insufficient, why not resort to higher-orders?

Original: With given sample size, our method aims to capture higher-order departure of the test statistic from being normally distributed and translate it into a form of an asymptotic expansion density.
Revised: With given sample size, our method aims to capture higher-order departure of the test statistic from being normally distributed and translate it into a form of an asymptotic expansion density.

Original: This asymptotic expansion formula formally guarantees an error bound of o ( n − ( p − 2) / 2 ) , where p is a user-specified integer.
Revised: This asymptotic expansion formula formally guarantees an error bound of o ( n − ( p − 2) / 2 ) , where p is a user-specified integer.

Original: This asymptotic expansion is used to provide higher-order adjusted z-scores for test statistics, which should be instance-dependent with respect to parameters of the experiment such as sample size.
Revised: This asymptotic expansion is used to provide higher-order adjusted z-scores for test statistics, which should be instance-dependent with respect to parameters of the experiment such as sample size.

Original: This paper provides the first attempt on higher-order approximation in the context of adaptive experiments.
Revised: This paper provides the first attempt on higher-order approximation in the context of adaptive experiments.

Original: As we will see, the framework of [15] that was used for a very different purpose (Jump-diffusion process in random environment ) can be applied batchwise to the context of batch-wise adaptive data collection, justified by the back propogation formula that provides precise estimate of the approximation error.
Revised: As we will see, the framework of [15] that was used for a very different purpose (Jump-diffusion process in random environment ) can be applied batchwise to the context of batch-wise adaptive data collection, justified by the back propogation formula that provides precise estimate of the approximation error.

Original: Besides, we also modify his framework to incorporate random measurable functions, possibly discontinuous, which corresponds to the assignment policy in the context of adaptive experiments.
Revised: Besides, we also modify his framework to incorporate random measurable functions, possibly discontinuous, which corresponds to the assignment policy in the context of adaptive experiments.

Original: The rest of the paper is organized as follows.
Revised: The rest of the paper is organized as follows.

Original: Section 2 formulates the problem and provides preliminaries.
Revised: Section 2 formulates the problem and provides preliminaries.

Original: The main results are provided in Section 3, where we estimate an arm value based on an asymptotic expansion density that generalizes the commonly-made normal approximation.
Revised: The main results are provided in Section 3, where we estimate an arm value based on an asymptotic expansion density that generalizes the commonly-made normal approximation.

Original: Section 4 offers preliminary numerical experiments.
Revised: Section 4 offers preliminary numerical experiments.

Original: Finally, Section 5 concludes.
Revised: Finally, Section 5 concludes.

Original:  Setup 2.1 Batched Bandit Framework
Revised:  Setup 2.1 Batched Bandit Framework

Original: Suppose the agent interacts with n individuals at batch s , and the experiment has S
Revised: Suppose the agent interacts with n individuals at batch s , and the experiment has S

Original: = { 1 , ..., S } batches in total.
Revised: = { 1 , ..., S } batches in total.

Original: At batch s , the i -th individual is identified with an element j =
Revised: At batch s , the i -th individual is identified with an element j =

Original: ( s, i s ) .
Revised: ( s, i s ) .

Original: We write J n = { ( s, i s ; i s ∈ I ns , s ∈ S } , where I ns = { 1 , ..., n s } .
Revised: We write J n = { ( s, i s ; i s ∈ I ns , s ∈ S } , where I ns = { 1 , ..., n s } .

Original: Let s ( j ) =
Revised: Let s ( j ) =

Original: s for each individual j ∈ J ns .
Revised: s for each individual j ∈ J ns .

Original: The batch size n is the parameter that drives the asymptotic theory we will develop.
Revised: The batch size n is the parameter that drives the asymptotic theory we will develop.

Original: Given a probability space (Ω , F , P ) , for the individual j ∈ J ns the agent selects an action/treatment according to an arm assignment policy (to be shortly introduced), which is expressed by a k sdimensional vector A j =
Revised: Given a probability space (Ω , F , P ) , for the individual j ∈ J ns the agent selects an action/treatment according to an arm assignment policy (to be shortly introduced), which is expressed by a k sdimensional vector A j =

Original: ( A j,k s ) k s ∈K s , where K s is the action space of size k s in batch s .
Revised: ( A j,k s ) k s ∈K s , where K s is the action space of size k s in batch s .

Original: Each entry
Revised: Each entry

Original: A j,k s takes values in { 1 , 0 } and
Revised: A j,k s takes values in { 1 , 0 } and

Original: (cid:80) k s ∈K s A j,k s = 1 .
Revised: (cid:80) k s ∈K s A j,k s = 1 .

Original: The agent then observes a reward R j from the action A j , which is written as follows,
Revised: The agent then observes a reward R j from the action A j , which is written as follows,

Original: [Equation]
Revised: [Equation]

Original: where β s ∈ R k s represents the underlying effect of the actions, ˙
Revised: where β s ∈ R k s represents the underlying effect of the actions, ˙

Original: ϵ j is an exogenous mean-zero noise that is sampled i.i.d.
Revised: ϵ j is an exogenous mean-zero noise that is sampled i.i.d.

Original: each time, and the star ⋆ denotes the matrix transpose 1 .
Revised: each time, and the star ⋆ denotes the matrix transpose 1 .

Original: For batch s , let A ns
Revised: For batch s , let A ns

Original: =
Revised: =

Original: ( A j ; j ∈ J ns ) represent the action set and ϵ ns =
Revised: ( A j ; j ∈ J ns ) represent the action set and ϵ ns =

Original: ( ϵ j ; j ∈ J ns ) , where ϵ j is an r s -dimensional random vector, e.g., ϵ j
Revised: ( ϵ j ; j ∈ J ns ) , where ϵ j is an r s -dimensional random vector, e.g., ϵ j

Original: =
Revised: =

Original: ˙ ϵ j , ˙ ϵ 2 j − σ 2 ) ⋆ .
Revised: ˙ ϵ j , ˙ ϵ 2 j − σ 2 ) ⋆ .

Original: For j ∈ J n s , let W n j be a d s × r s random matrix measurable with respect to σ [ A ns ( j ) ]
Revised: For j ∈ J n s , let W n j be a d s × r s random matrix measurable with respect to σ [ A ns ( j ) ]

Original: Write W ns =
Revised: Write W ns =

Original: ( W nj ) j ∈ J ns .
Revised: ( W nj ) j ∈ J ns .

Original: Consider a weighted sum
Revised: Consider a weighted sum

Original: [Equation]
Revised: [Equation]

Original: This Z ns captures the most statistical quantities of interest in estimating the value of a single arm or arm difference for batch s .
Revised: This Z ns captures the most statistical quantities of interest in estimating the value of a single arm or arm difference for batch s .

Original: For example, using data collected only from batch s , the scaled error of the OLS estimator has the following form:
Revised: For example, using data collected only from batch s , the scaled error of the OLS estimator has the following form:

Original: [Equation]
Revised: [Equation]

Original: where N ns,k denotes the number of action k being selected in batch s .
Revised: where N ns,k denotes the number of action k being selected in batch s .

Original: We next model the bandit algorithm that determines the policy for each batch.
Revised: We next model the bandit algorithm that determines the policy for each batch.

Original: Let L s be a measurable space for s ∈ S .
Revised: Let L s be a measurable space for s ∈ S .

Original: We consider a σ [ A ns ] -measurable random map L ns : Ω → L s for every ( n, s ) ∈ N × S .
Revised: We consider a σ [ A ns ] -measurable random map L ns : Ω → L s for every ( n, s ) ∈ N × S .

Original: The variables L ns − 1 will be used for making a criterion for selection of an action in batch s .
Revised: The variables L ns − 1 will be used for making a criterion for selection of an action in batch s .

Original: We write
Revised: We write

Original: L ns =
Revised: L ns =

Original: ( L n 1 , ..., L ns ) for ( L s ) s ∈ S 2 .
Revised: ( L n 1 , ..., L ns ) for ( L s ) s ∈ S 2 .

Original: In the batched bandits, how the agent updates the data collection policy c s (the distribution of A ns ) from a measurable set C s is determined by the average effects of actions at batch s − 1 .
Revised: In the batched bandits, how the agent updates the data collection policy c s (the distribution of A ns ) from a measurable set C s is determined by the average effects of actions at batch s − 1 .

Original: We choose the policy c s by the distribution q (cid:0) ( L ns − 1 , Z ns − 1 ) , dc s (cid:1) over the action space.
Revised: We choose the policy c s by the distribution q (cid:0) ( L ns − 1 , Z ns − 1 ) , dc s (cid:1) over the action space.

Original: Example 2 ( ϵ -Greedy algorithm) .
Revised: Example 2 ( ϵ -Greedy algorithm) .

Original: The ϵ -Greedy algorithm having C 2 = { c (1)2 , c (2)2 } with
Revised: The ϵ -Greedy algorithm having C 2 = { c (1)2 , c (2)2 } with

Original: [Equation]
Revised: [Equation]

Original: for some function h 1 .
Revised: for some function h 1 .

Original: Example 3 (Thomspon sampling) .
Revised: Example 3 (Thomspon sampling) .

Original: A Thompson sampling algorithm is realized as q
Revised: A Thompson sampling algorithm is realized as q

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: with some constants a 1 , a 2 expressing the clipping constraint commonly assumed in batched bandits[16] and some C 2 -valued function C 2 of ( l 1 , z 1 ) .
Revised: with some constants a 1 , a 2 expressing the clipping constraint commonly assumed in batched bandits[16] and some C 2 -valued function C 2 of ( l 1 , z 1 ) .

Original: Finally, we denote G ns = σ (cid:2)
Revised: Finally, we denote G ns = σ (cid:2)

Original: L ns ′ , Z ns ′ ; s ′ ≤ s (cid:3) , and let L = Π s ∈ S L s and d = (cid:80) s ∈ S d s .
Revised: L ns ′ , Z ns ′ ; s ′ ≤ s (cid:3) , and let L = Π s ∈ S L s and d = (cid:80) s ∈ S d s .

Original: We assume the following condition holds unless stated otherwise:
Revised: We assume the following condition holds unless stated otherwise:

Original: [Equation]
Revised: [Equation]

Original: For notational simplicity, we denote the conditional expectation E [ · |G ns − 1 ] as E s − 1 [ · ] and the conditional expectation E (cid:2) · |G ns − 1 ∨ σ [ A ns ] (cid:3) as E s − 1 , A ns [ · ] .
Revised: For notational simplicity, we denote the conditional expectation E [ · |G ns − 1 ] as E s − 1 [ · ] and the conditional expectation E (cid:2) · |G ns − 1 ∨ σ [ A ns ] (cid:3) as E s − 1 , A ns [ · ] .

Original: For w ns = ( w nj ) j ∈ J ns ∈ R d s r s n s ,
Revised: For w ns = ( w nj ) j ∈ J ns ∈ R d s r s n s ,

Original: P ns ( w ns , dz s ) =
Revised: P ns ( w ns , dz s ) =

Original: P (cid:80) j ∈ J ns w nj ϵ j ( dz s ) represents the distribution of (cid:80) j ∈ J ns w nj ϵ j 

Revised: P (cid:80) j ∈ J ns w nj ϵ j ( dz s ) represents the distribution of (cid:80) j ∈ J ns w nj ϵ j

Original: We assume a representation of a regular conditional distribution of ( L ns , W ns ) given G ns − 1 as
Revised: We assume a representation of a regular conditional distribution of ( L ns , W ns ) given G ns − 1 as

Original: [Equation]
Revised: [Equation]

Original: 6) for a probability distribution ν nc s = η ( L ns , W ns ) c s of ( L ns , W ns ) given c s .
Revised: 6) for a probability distribution ν nc s = η ( L ns , W ns ) c s of ( L ns , W ns ) given c s .

Original: 2.2 Off-policy Evaluation with Backwards Induction
Revised: 2.2 Off-policy Evaluation with Backwards Induction

Original: When the experiment finishes, we want to conduct hypothesis testing like the value of the arm β s ( j ) for scientific discovery.
Revised: When the experiment finishes, we want to conduct hypothesis testing like the value of the arm β s ( j ) for scientific discovery.

Original: We can generically define hypothesis testing problems mathematically as follows.
Revised: We can generically define hypothesis testing problems mathematically as follows.

Original: Given a measurable function f : L × R m → R , where m = (cid:80) s ∈ S m s , our estimand is the value of the expectation E n = E (cid:2) f (cid:0) L n S , Y nS (cid:1)(cid:3) , where Y ns =
Revised: Given a measurable function f : L × R m → R , where m = (cid:80) s ∈ S m s , our estimand is the value of the expectation E n = E (cid:2) f (cid:0) L n S , Y nS (cid:1)(cid:3) , where Y ns =

Original: Y ns ( L ns , Z ns ) is an m s -dimensional random variable.
Revised: Y ns ( L ns , Z ns ) is an m s -dimensional random variable.

Original: For standard hypotheses and confidence intervals, f is usually an indicator function for whether the test statistic is smaller than some particular threshold.
Revised: For standard hypotheses and confidence intervals, f is usually an indicator function for whether the test statistic is smaller than some particular threshold.

Original: If we know the true underlying density P ns ( w ns , dz s ) , we can iteratively calculate the desired expectation from a simple exercise analogous to dynamic programming, with a slight modification with a truncation 1 { L ns ∈A ns } , where A ns is a measurable set of L s .
Revised: If we know the true underlying density P ns ( w ns , dz s ) , we can iteratively calculate the desired expectation from a simple exercise analogous to dynamic programming, with a slight modification with a truncation 1 { L ns ∈A ns } , where A ns is a measurable set of L s .

Original: If the clipping constraint holds, E n almost equals to E n :=
Revised: If the clipping constraint holds, E n almost equals to E n :=

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Theorem1 formally proves this with the aid of the following lemma.
Revised: Theorem1 formally proves this with the aid of the following lemma.

Original: Lemma 1.
Revised: Lemma 1.

Original: For s ∈ S ,
Revised: For s ∈ S ,

Original: E s −(cid:20)
Revised: E s −(cid:20)

Original: f ns (cid:0) L ns , Z ns (cid:1) s (cid:89) s ′
Revised: f ns (cid:0) L ns , Z ns (cid:1) s (cid:89) s ′

Original: =
Revised: =

Original: 1 { L ns ′ ∈A ns ′ } (cid:21) =
Revised: 1 { L ns ′ ∈A ns ′ } (cid:21) =

Original: f ns − 1 ( L ns − 1 , Z ns − 1 ) s − 1 (cid:89) s ′
Revised: f ns − 1 ( L ns − 1 , Z ns − 1 ) s − 1 (cid:89) s ′

Original: = 1 { L ns ′ ∈A ns ′ } a.s.
Revised: = 1 { L ns ′ ∈A ns ′ } a.s.

Original: Here the product (cid:81) 0 s ′
Revised: Here the product (cid:81) 0 s ′

Original: =1 reads 1 .
Revised: =1 reads 1 .

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose that f (cid:0) L nS , Z nS (cid:1) is integrable.
Revised: Suppose that f (cid:0) L nS , Z nS (cid:1) is integrable.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: for s ∈ S , s
Revised: for s ∈ S , s

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 on its own does not have much practical relevance, since the true distribution P ns ( w ns , dz s ) is unknown.
Revised: Theorem 1 on its own does not have much practical relevance, since the true distribution P ns ( w ns , dz s ) is unknown.

Original: Therefore past research on off policy evaluation have aimed to approximate it by some random signed measure Ψ ns,p, w ns on R d depending on w ns .
Revised: Therefore past research on off policy evaluation have aimed to approximate it by some random signed measure Ψ ns,p, w ns on R d depending on w ns .

Original: All past attempts on asymptotic approximation claimed that the gaussian density was sufficient.
Revised: All past attempts on asymptotic approximation claimed that the gaussian density was sufficient.

Original: We claimed that can be problematic especially when the sample size is small and the noise does not behave well.
Revised: We claimed that can be problematic especially when the sample size is small and the noise does not behave well.

Original:  Main Result: Asymptotic Expansion for Batched Bandits
Revised:  Main Result: Asymptotic Expansion for Batched Bandits

Original: To assess the accuracy of the approximation, we need to estimate the difference between f n 0 and (cid:98) f n 0 , where (cid:98) f n 0 is the approximation to the f n 0 by replacing P ns ( w ns , dz s ) with Ψ ns,p, w ns for each batch.
Revised: To assess the accuracy of the approximation, we need to estimate the difference between f n 0 and (cid:98) f n 0 , where (cid:98) f n 0 is the approximation to the f n 0 by replacing P ns ( w ns , dz s ) with Ψ ns,p, w ns for each batch.

Original: More precisely, we define (cid:98) f ns ( l s , z s ) by (cid:98) f nS = f nS and
Revised: More precisely, we define (cid:98) f ns ( l s , z s ) by (cid:98) f nS = f nS and

Original: [Equation]
Revised: [Equation]

Original: for s ∈ S , if the integral (8) exists.
Revised: for s ∈ S , if the integral (8) exists.

Original: We shall only present the results with assumptions stated verbally due to the page limit and we refer the interested reader to [10] for directions for the full version.
Revised: We shall only present the results with assumptions stated verbally due to the page limit and we refer the interested reader to [10] for directions for the full version.

Original: The conditions we assume include (i) conditional i.i.d.
Revised: The conditions we assume include (i) conditional i.i.d.

Original: noise ϵ that hold in most sequential experiments, (ii) regularity conditions such as measurability or existence of moments, and (iii) clipping conditions that are usually implicitly or explicitly assumed in the past literature [15].
Revised: noise ϵ that hold in most sequential experiments, (ii) regularity conditions such as measurability or existence of moments, and (iii) clipping conditions that are usually implicitly or explicitly assumed in the past literature [15].

Original: The only condition that uniquely arises in the higher-order case is the well-known Cramér condition as follows, for any batch s ∈ S , there exists a constant B 0 such that
Revised: The only condition that uniquely arises in the higher-order case is the well-known Cramér condition as follows, for any batch s ∈ S , there exists a constant B 0 such that

Original: [Equation]
Revised: [Equation]

Original: which places regularity on the characteristic function of the noise distribution and validates the
Revised: which places regularity on the characteristic function of the noise distribution and validates the

Original: Fourier inversion to get the asymptotic expansion density.
Revised: Fourier inversion to get the asymptotic expansion density.

Original: This condition generally holds except for lattice distributions that we leave for future work.
Revised: This condition generally holds except for lattice distributions that we leave for future work.

Original: Theorem 2 (Informal) .
Revised: Theorem 2 (Informal) .

Original: Suppose the conditions stated above hold.
Revised: Suppose the conditions stated above hold.

Original: There exists a constant M ∗ such that E n = f n 0 admits the following estimates:
Revised: There exists a constant M ∗ such that E n = f n 0 admits the following estimates:

Original: [Equation]
Revised: [Equation]

Original: for all f ∈ (cid:98) D ( M , γ ) , a class of measurable functions with at most polynomial growth, where
Revised: for all f ∈ (cid:98) D ( M , γ ) , a class of measurable functions with at most polynomial growth, where

Original: [Equation]
Revised: [Equation]

Original: for s = 1 , ..., S − 1 .
Revised: for s = 1 , ..., S − 1 .

Original: Remark 1.
Revised: Remark 1.

Original: Theorem 2 suggests using (cid:98) f n 0 to approximate E n .
Revised: Theorem 2 suggests using (cid:98) f n 0 to approximate E n .

Original: Usually U ns +1 is uniformly bounded in n or grows as slowly as to be controlled by the factor n − δ 1 / 2 in the error bound of (9).
Revised: Usually U ns +1 is uniformly bounded in n or grows as slowly as to be controlled by the factor n − δ 1 / 2 in the error bound of (9).

Original: We also present an explicit form of the asymptotic expansion density that can be used to calculate the higher-order corrected Z-score for BOLS estimator [16] up to order p (which is a user-specified integer) when the variance is known.
Revised: We also present an explicit form of the asymptotic expansion density that can be used to calculate the higher-order corrected Z-score for BOLS estimator [16] up to order p (which is a user-specified integer) when the variance is known.

Original: We present the case when variance is estimated in [10].
Revised: We present the case when variance is estimated in [10].

Original: Proposition 1.
Revised: Proposition 1.

Original: The function d Ψ n s ,p, C s /dz ( z ) ( z = ( z 1 , ..., z k s ) ) is
Revised: The function d Ψ n s ,p, C s /dz ( z ) ( z = ( z 1 , ..., z k s ) ) is

Original: [Equation]
Revised: [Equation]

Original: with the summation taken up to order n − ( p − 2) / 2 s , where λ n s ,r, C s ,k = n ( r − 2) / 2 s · ( N ns,k /n s ) − ( r − 2) /
Revised: with the summation taken up to order n − ( p − 2) / 2 s , where λ n s ,r, C s ,k = n ( r − 2) / 2 s · ( N ns,k /n s ) − ( r − 2) /

Original: 2 κ r, C s ( ϵ ( s, 1) ) with the r -th order C s conditional cumulant
Revised: 2 κ r, C s ( ϵ ( s, 1) ) with the r -th order C s conditional cumulant

Original: κ r, C s ( ϵ ( s, 1) ) of ϵ ( s, 1) , and h r ( z ; Σ) = ( − 1) r ϕ
Revised: κ r, C s ( ϵ ( s, 1) ) of ϵ ( s, 1) , and h r ( z ; Σ) = ( − 1) r ϕ

Original: ( z ; 0 , Σ) − 1 ∂ rz ϕ
Revised: ( z ; 0 , Σ) − 1 ∂ rz ϕ

Original: ( z ; 0 , Σ) is the r -th order Hermite polynomial.
Revised: ( z ; 0 , Σ) is the r -th order Hermite polynomial.

Original: Remark 2.
Revised: Remark 2.

Original: Note that all elements of this density are known and calculable for batched bandits.
Revised: Note that all elements of this density are known and calculable for batched bandits.

Original: Even if we do not know the distribution and hence the moments of ϵ ( s, 1) , we derived an asymptotic expansion that is valid up to the first order, by plugging certain estimators in the moments.
Revised: Even if we do not know the distribution and hence the moments of ϵ ( s, 1) , we derived an asymptotic expansion that is valid up to the first order, by plugging certain estimators in the moments.

Original:  Simulations
Revised:  Simulations

Original: In the main paper, we only provide the numerical results for the setting described in Example 1.
Revised: Due to space restrictions, we only provide the numerical results for the setting described in Example 1.

Original: As figure 2 shows, asymptotic expansion has clear gains over the normal distribution in approximating the true underlying density.
Revised: As Figure 2 shows, asymptotic expansion has clear gains over the normal distribution in approximating the true underlying density.

Original: While the second order asymptotic expansion seems to have better approximation in most regions of the support, the difference may not be that large.
Revised: While the second order asymptotic expansion seems to have better approximation in most regions of the support, the difference may not be that large.

Original: When increasing the sample size for each batch from 30 to 50 as in figure 3, second order expansion seems to perform better.
Revised: When increasing the sample size for each batch from 30 to 50 as in Figure 3, second order expansion seems to perform better.

Original: While a more extensive numerical experiments is necessary to fully assess the marginal benefit, asymptotic expansion methods seems to be better in general than normal approximation based methods.
Revised: While a more extensive numerical experiments is necessary to fully assess the marginal benefit, asymptotic expansion methods seems to be better in general than normal approximation based methods.

Original:  Discussions
Revised:  Discussions

Original: In this paper, we have introduced, developed, and justified the first theory for higher-order statistical inference specialized for data collected by batched bandit algorithms.
Revised: In this paper, we have introduced, developed, and justified the first theory for higher-order statistical inference specialized for data collected by batched bandit algorithms.

Original: Interesting future work includes
Revised: Interesting future work includes

Original: Higher extending our higher-order theory to non-batched case and doing power analysis based on higherorder theory.
Revised: Higher extending our higher-order theory to non-batched case and doing power analysis based on higherorder theory.

Original: Having better type-I error control and powered experiments thanks to higher-order inference has a potential to extend the limits on the experimental design, enabling more aggressive algorithms or reducing the sample sizes necessary.
Revised: Having better type-I error control and powered experiments thanks to higher-order inference has a potential to extend the limits on the experimental design, enabling more aggressive algorithms or reducing the sample sizes necessary.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work was in part supported by Japan Science and Technology Agency CREST JPMJCR2115;
Revised: This work was in part supported by Japan Science and Technology Agency CREST JPMJCR2115;

Original: Japan Society for the Promotion of Science Grants-in-Aid for Scientific Research No.
Revised: Japan Society for the Promotion of Science Grants-in-Aid for Scientific Research No.

Original: 17H01702 (Scientific Research); and by a Cooperative Research Program of the Institute of Statistical Mathematics.
Revised: 17H01702 (Scientific Research); and by a Cooperative Research Program of the Institute of Statistical Mathematics.

Original: The authors thank Professor Kengo Kamatani for his valuable comments on numerical computations.
Revised: The authors thank Professor Kengo Kamatani for his valuable comments on numerical computations.