Original: O FF -P OLICY E VALUATION AND L EARNING FROM L OGGED B ANDIT F EEDBACK : E RROR R EDUCTION VIA S URROGATE P OLICY
Revised: O FF -P OLICY E VALUATION AND L EARNING FROM L OGGED B ANDIT F EEDBACK : E RROR R EDUCTION VIA S URROGATE P OLICY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: When learning from a batch of logged bandit feedback, the discrepancy between the policy to be learned and the off-policy training data imposes statistical and computational challenges.
Revised: When learning from a batch of logged bandit feedback, the discrepancy between the policy to be learned and the off-policy training data imposes statistical and computational challenges.

Original: Unlike classical supervised learning and online learning settings, in batch contextual bandit learning, one only has access to a collection of logged feedback from the actions taken by a historical policy, and expect to learn a policy that takes good actions in possibly unseen contexts.
Revised: Unlike classical supervised learning and online learning settings, in batch contextual bandit learning, one only has access to a collection of logged feedback from the actions taken by a historical policy, and expect to learn a policy that takes good actions in possibly unseen contexts.

Original: Such a batch learning setting is ubiquitous in online and interactive systems, such as ad platforms and recommendation systems.
Revised: Such a batch learning setting is ubiquitous in online and interactive systems, such as ad platforms and recommendation systems.

Original: Existing approaches based on inverse propensity weights, such as Inverse Propensity Scoring (IPS) and Policy Optimizer for Exponential
Revised: Existing approaches based on inverse propensity weights, such as Inverse Propensity Scoring (IPS) and Policy Optimizer for Exponential

Original: Models (POEM), enjoy unbiasedness but often suffer from large mean squared error.
Revised: Models (POEM), enjoy unbiasedness but often suffer from large mean squared error.

Original: In this work, we introduce a new approach named Maximum Likelihood Inverse Propensity Scoring (MLIPS) for batch learning from logged bandit feedback.
Revised: In this work, we introduce a new approach named Maximum Likelihood Inverse Propensity Scoring (MLIPS) for batch learning from logged bandit feedback.

Original: Instead of using the given historical policy as the proposal in inverse propensity weights, we estimate a maximum likelihood surrogate policy based on the logged action-context pairs, and then use this surrogate policy as the proposal.
Revised: Instead of using the given historical policy as the proposal in inverse propensity weights, we estimate a maximum likelihood surrogate policy based on the logged action-context pairs, and then use this surrogate policy as the proposal.

Original: We prove that MLIPS is asymptotically unbiased, and moreover, has a smaller nonasymptotic mean squared error than IPS.
Revised: We prove that MLIPS is asymptotically unbiased, and moreover, has a smaller nonasymptotic mean squared error than IPS.

Original: Such an error reduction phenomenon is somewhat surprising as the estimated surrogate policy is less accurate than the given historical policy.
Revised: Such an error reduction phenomenon is somewhat surprising as the estimated surrogate policy is less accurate than the given historical policy.

Original: Results on multi-label classiﬁcation problems and a large-scale ad placement dataset demonstrate the empirical effectiveness of MLIPS.
Revised: Results on multi-label classiﬁcation problems and a large-scale ad placement dataset demonstrate the empirical effectiveness of MLIPS.

Original: Furthermore, the proposed surrogate policy technique is complementary to existing error reduction techniques, and when combined, is able to consistently boost the performance of several widely used approaches.
Revised: Furthermore, the proposed surrogate policy technique is complementary to existing error reduction techniques, and when combined, is able to consistently boost the performance of several widely used approaches.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While maintaining online and interactive systems for information retrieval, news recommendation, ad placement, and e-commerce, we collect large batches of logs during testing phases and past deployment periods (Li et al., 2010).
Revised: While maintaining online and interactive systems for information retrieval, news recommendation, ad placement, and e-commerce, we collect large batches of logs during testing phases and past deployment periods (Li et al., 2010).

Original: Such logs contain rich information that can be used for many purposes.
Revised: Such logs contain rich information that can be used for many purposes.

Original: For example, the logs of an e-commerce system record the decisions (or actions) on which ads or recommendations are displayed to a user given a context, and the corresponding feedback (or reward), including whether the user clicks on any of the displayed items and whether a purchase occurs afterwards.
Revised: For example, the logs of an e-commerce system record the decisions (or actions) on which ads or recommendations are displayed to a user given a context, and the corresponding feedback (or reward), including whether the user clicks on any of the displayed items and whether a purchase occurs afterwards.

Original: Although the logs are collected when the online system is deployed with a historical algorithm (or policy), they are informative for future improvements and design of the system.
Revised: Although the logs are collected when the online system is deployed with a historical algorithm (or policy), they are informative for future improvements and design of the system.

Original: One application of such logs is to estimate the performance of new policies, also known as off-policy evaluation (Dud´ık et al., 2014;
Revised: One application of such logs is to estimate the performance of new policies, also known as off-policy evaluation (Dud´ık et al., 2014;

Original: Li et al., 2011; 2014).
Revised: Li et al., 2011; 2014).

Original: Furthermore, the use of logs allows for an accurate and more efﬁcient way to test and optimize new policies without expensive trial-and-error cycles in traditional A/B tests, which often take weeks.
Revised: Furthermore, the use of logs allows for an accurate and more efﬁcient way to test and optimize new policies without expensive trial-and-error cycles in traditional A/B tests, which often take weeks.

Original: Since the logged feedback only provides partial information, policy evaluation and optimization using logs are often formulated as batch contextual bandit learning.
Revised: Since the logged feedback only provides partial information, policy evaluation and optimization using logs are often formulated as batch contextual bandit learning.

Original: Different from the online learning setting, batch contextual bandit learning is statistically more challenging because the collected logs are off-policy, that is, they are generated by a historical policy that differs from the current policy we intend to evaluate or optimize.
Revised: Different from the online learning setting, batch contextual bandit learning is statistically more challenging because the collected logs are off-policy, that is, they are generated by a historical policy that differs from the current policy we intend to evaluate or optimize.

Original: To bridge this discrepancy, Inverse Propensity Scoring (IPS) has been introduced to construct an unbiased counterfactual estimator of policy performance using off-policy data (Bottou et al., 2013).
Revised: To bridge this discrepancy, Inverse Propensity Scoring (IPS) has been introduced to construct an unbiased counterfactual estimator of policy performance using off-policy data (Bottou et al., 2013).

Original: However, such an estimator tends to have a large mean squared error when the current policy and the historical policy are very different.
Revised: However, such an estimator tends to have a large mean squared error when the current policy and the historical policy are very different.

Original: To reduce the mean squared error, a number of regularization and control variate approaches have been introduced (e.g., Dud´ık et al.
Revised: To reduce the mean squared error, a number of regularization and control variate approaches have been introduced (e.g., Dud´ık et al.

Original: ; Hirano et al.
Revised: ; Hirano et al.

Original: ; Ionides (2008);
Revised: ; Ionides (2008);

Original: Li et al.
Revised: Li et al.

Original: Notably, Swaminathan &
Revised: Notably, Swaminathan &

Original: Joachims (2015a) proposed Policy Optimizer for Exponential Models (POEM), which introduces a mean squared error regularizer based on the counterfactual risk minimization principle.
Revised: Joachims (2015a) proposed Policy Optimizer for Exponential Models (POEM), which introduces a mean squared error regularizer based on the counterfactual risk minimization principle.

Original: Empirical evaluations also suggested the effectiveness of these approaches on policy evaluation and optimization.
Revised: Empirical evaluations also suggested the effectiveness of these approaches on policy evaluation and optimization.

Original: In this paper, we introduce a new yet simple parametric approach for mean squared error reduction in batch contextual bandit learning.
Revised: In this paper, we introduce a new yet simple parametric approach for mean squared error reduction in batch contextual bandit learning.

Original: Instead of using the given historical policy, we use linear models or neural network models to estimate a maximum likelihood surrogate policy using the logged actioncontext pairs.
Revised: Instead of using the given historical policy, we use linear models or neural network models to estimate a maximum likelihood surrogate policy using the logged actioncontext pairs.

Original: Then we use such a surrogate policy as the proposal distribution to obtain the inverse propensity weights in the off-policy estimator, as if the logs are generated by this surrogate policy.
Revised: Then we use such a surrogate policy as the proposal distribution to obtain the inverse propensity weights in the off-policy estimator, as if the logs are generated by this surrogate policy.

Original: This idea dates back to the “estimated sampler” technique in statistics (Delyon & Portier, 2016;
Revised: This idea dates back to the “estimated sampler” technique in statistics (Delyon & Portier, 2016;

Original: Henmi et al., 2007).
Revised: Henmi et al., 2007).

Original: We provide theoretical justiﬁcation for this approach, which is named Maximum
Revised: We provide theoretical justiﬁcation for this approach, which is named Maximum

Original: Likelihood Inverse Propensity Scoring (MLIPS).
Revised: Likelihood Inverse Propensity Scoring (MLIPS).

Original: In particular, we show that the proposed MLIPS estimator is asymptotically unbiased, and moreover, has a smaller nonasymptotic mean squared error than the IPS estimator.
Revised: In particular, we show that the proposed MLIPS estimator is asymptotically unbiased, and moreover, has a smaller nonasymptotic mean squared error than the IPS estimator.

Original: Such an error reduction effect is counterintuitive as we replace the known historical policy with a less accurate estimated surrogate policy, which is supposed to increase the mean squared error.
Revised: Such an error reduction effect is counterintuitive as we replace the known historical policy with a less accurate estimated surrogate policy, which is supposed to increase the mean squared error.

Original: We evaluate the MLIPS estimator on several multi-label classiﬁcation problems and a large-scale ad placement dataset to demonstrate its empirical effectiveness on mean squared error reduction for policy evaluation and optimization.
Revised: We evaluate the MLIPS estimator on several multi-label classiﬁcation problems and a large-scale ad placement dataset to demonstrate its empirical effectiveness on mean squared error reduction for policy evaluation and optimization.

Original: Furthermore, we also combine the
Revised: Furthermore, we also combine the

Original: MLIPS estimator with several existing approaches.
Revised: MLIPS estimator with several existing approaches.

Original: We ﬁnd that our surrogate policy technique is complementary to existing mean squared error reduction techniques and achieves consistently improved performance.
Revised: We ﬁnd that our surrogate policy technique is complementary to existing mean squared error reduction techniques and achieves consistently improved performance.

Original:  O FF -P OLICY E VALUATION AND L EARNING FROM L OGGED B ANDIT F EEDBACK
Revised:  O FF -P OLICY E VALUATION AND L EARNING FROM L OGGED B ANDIT F EEDBACK

Original: In this section, we introduce the background on off-policy evaluation and learning from logged bandit feedback.
Revised: In this section, we introduce the background on off-policy evaluation and learning from logged bandit feedback.

Original: In particular, we focus on IPS and present several existing mean squared error reduction and regularization approaches.
Revised: In particular, we focus on IPS and present several existing mean squared error reduction and regularization approaches.

Original: 2. I NVERSE P ROPENSITY S CORING
Revised: 2. I NVERSE P ROPENSITY S CORING

Original: The value function used to evaluate a target policy π is deﬁned as
Revised: The value function used to evaluate a target policy π is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Here x is the context drawn from the context distribution λ , which is independent of π , a is the action taken according to π given context x , and r is the reward (or feedback) received.
Revised: Here x is the context drawn from the context distribution λ , which is independent of π , a is the action taken according to π given context x , and r is the reward (or feedback) received.

Original: In the off-policy setting, we are not able to sample different actions from the target policy π and obtain reward, which is expensive and time-consuming in practice.
Revised: In the off-policy setting, we are not able to sample different actions from the target policy π and obtain reward, which is expensive and time-consuming in practice.

Original: Instead, we collect samples from a logging policy µ , that is, a ∼ µ ( · | x ) , and expect to use the logs to evaluate a target policy and further perform policy optimization.
Revised: Instead, we collect samples from a logging policy µ , that is, a ∼ µ ( · | x ) , and expect to use the logs to evaluate a target policy and further perform policy optimization.

Original: To bridge the discrepancy between π and µ , the IPS estimator, which is known as the importance sampling estimator, reweights each logged action-context pair as follows,
Revised: To bridge the discrepancy between π and µ , the IPS estimator, which is known as the importance sampling estimator, reweights each logged action-context pair as follows,

Original: [Equation]
Revised: [Equation]

Original: In the sequel, we use (cid:101) V to denote (cid:101) V IPS for notational simplicity.
Revised: In the sequel, we use (cid:101) V to denote (cid:101) V IPS for notational simplicity.

Original: It is worth noting that the IPS estimator is an unbiased estimator of V , and thus has been widely adopted as an objective function for optimizing the target policy π .
Revised: It is worth noting that the IPS estimator is an unbiased estimator of V , and thus has been widely adopted as an objective function for optimizing the target policy π .

Original: However, ρ ( x i , a i ) =
Revised: However, ρ ( x i , a i ) =

Original: π ( a i | x i ) /µ
Revised: π ( a i | x i ) /µ

Original: ( a i | x i ) in (2.2) may induce a large mean squared error, especially when the logging policy µ is very different from the target policy π .
Revised: ( a i | x i ) in (2.2) may induce a large mean squared error, especially when the logging policy µ is very different from the target policy π .

Original: As a result, the IPS estimator often leads to unsatisfactory policy optimization and poor generalization performance.
Revised: As a result, the IPS estimator often leads to unsatisfactory policy optimization and poor generalization performance.

Original: 2. M EAN S QUARED E RROR R EDUCTION AND R EGULARIZATION
Revised: 2. M EAN S QUARED E RROR R EDUCTION AND R EGULARIZATION

Original: To reduce the mean squared error of IPS, several thresholding approaches have been proposed and studied in the literature (e.g., Bottou et al.
Revised: To reduce the mean squared error of IPS, several thresholding approaches have been proposed and studied in the literature (e.g., Bottou et al.

Original: ; Cortes et al. (2010); Ionides (2008); Strehl et al.
Revised: ; Cortes et al. (2010); Ionides (2008); Strehl et al.

Original: One straightforward approach is Propensity Weight Capping (Ionides, 2008), which takes the form
Revised: One straightforward approach is Propensity Weight Capping (Ionides, 2008), which takes the form

Original: [Equation]
Revised: [Equation]

Original: where M > 1 is the threshold parameter.
Revised: where M > 1 is the threshold parameter.

Original: Although thresholding reduces the mean squared error when M is small, it introduces a large bias.
Revised: Although thresholding reduces the mean squared error when M is small, it introduces a large bias.

Original: More recently, Swaminathan & Joachims (2015a) proposed the POEM algorithm for batch learning from bandit feedback.
Revised: More recently, Swaminathan & Joachims (2015a) proposed the POEM algorithm for batch learning from bandit feedback.

Original: POEM is motivated by the structural risk minimization principle and a generalization bound that characterizes the mean squared error of (cid:101) V ( M ) IPS based on an empirical
Revised: POEM is motivated by the structural risk minimization principle and a generalization bound that characterizes the mean squared error of (cid:101) V ( M ) IPS based on an empirical

Original: Bernstein argument.
Revised: Bernstein argument.

Original: The POEM algorithm jointly optimizes (cid:101) V ( M ) IPS and its empirical standard deviation, which is also known as counterfactual risk minimization.
Revised: The POEM algorithm jointly optimizes (cid:101) V ( M ) IPS and its empirical standard deviation, which is also known as counterfactual risk minimization.

Original: Let u i π
Revised: Let u i π

Original: = min { M, π ( a i | x i ) /µ
Revised: = min { M, π ( a i | x i ) /µ

Original: ( a i | x i ) } · r i , ¯ u
Revised: ( a i | x i ) } · r i , ¯ u

Original: [Equation]
Revised: [Equation]

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =1 u iπ /n , and (cid:100)
Revised: =1 u iπ /n , and (cid:100)

Original: Var π
Revised: Var π

Original: ( (cid:101) V ( M ) IPS )
Revised: ( (cid:101) V ( M ) IPS )

Original: =
Revised: =

Original: ( (cid:80) ni
Revised: ( (cid:80) ni

Original: =1 ( u iπ − ¯ u π ) 2 ) / ( n − 1) , the POEM algorithm searches for a policy π that maximizes
Revised: =1 ( u iπ − ¯ u π ) 2 ) / ( n − 1) , the POEM algorithm searches for a policy π that maximizes

Original: [Equation]
Revised: [Equation]

Original: Here M > 1 and λ ≥ 0 are the thresholding and regularization parameters, respectively.
Revised: Here M > 1 and λ ≥ 0 are the thresholding and regularization parameters, respectively.

Original: Also note that when λ = 0 , the POEM objective in (2.4) reduces to (2.3).
Revised: Also note that when λ = 0 , the POEM objective in (2.4) reduces to (2.3).

Original: Motivated by the propensity overﬁtting problem of IPS and POEM, Swaminathan & Joachims (2015b) proposed to use control variates, which lead to the following self-normalized estimator
Revised: Motivated by the propensity overﬁtting problem of IPS and POEM, Swaminathan & Joachims (2015b) proposed to use control variates, which lead to the following self-normalized estimator

Original: [Equation]
Revised: [Equation]

Original: Note that (cid:101) V SN is shifted by a constant C if each r i is shifted to r i + C .
Revised: Note that (cid:101) V SN is shifted by a constant C if each r i is shifted to r i + C .

Original: In other words, (cid:101) V SN is equivariant (Hesterberg, 1995) and always lies within the range of r i .
Revised: In other words, (cid:101) V SN is equivariant (Hesterberg, 1995) and always lies within the range of r i .

Original: In contrast, the IPS estimator does not have such properties.
Revised: In contrast, the IPS estimator does not have such properties.

Original: Self-Normalized POEM (Norm-POEM), a variant of POEM, was also proposed by Swaminathan & Joachims (2015b).
Revised: Self-Normalized POEM (Norm-POEM), a variant of POEM, was also proposed by Swaminathan & Joachims (2015b).

Original: In particular, Norm-POEM searches for a policy that maximizes the normalized estimator regularized by its empirical standard deviation
Revised: In particular, Norm-POEM searches for a policy that maximizes the normalized estimator regularized by its empirical standard deviation

Original: [Equation]
Revised: [Equation]

Original: In addition, other techniques, such as control variates and doubly-robust estimators Dud´ık et al.
Revised: In addition, other techniques, such as control variates and doubly-robust estimators Dud´ık et al.

Original: (2014) have also been applied to reduce the mean squared error of the off-policy estimator.
Revised: (2014) have also been applied to reduce the mean squared error of the off-policy estimator.

Original:  E RROR R EDUCTION VIA S URROGATE P OLICY
Revised:  E RROR R EDUCTION VIA S URROGATE P OLICY

Original: Instead of directly using the logging policy propensities, we propose to reﬁt a surrogate policy and use it to obtain the inverse propensity weights in the IPS estimator deﬁned in (2.2).
Revised: Instead of directly using the logging policy propensities, we propose to reﬁt a surrogate policy and use it to obtain the inverse propensity weights in the IPS estimator deﬁned in (2.2).

Original: We assume that the logging policy µ ( a | x ) is parameterized by β ∈ R d and denote it by µ ( a | x ; β ) .
Revised: We assume that the logging policy µ ( a | x ) is parameterized by β ∈ R d and denote it by µ ( a | x ; β ) .

Original: In particular, we assume the logging policy with β = β ∗ generates the logged action-context pairs.
Revised: In particular, we assume the logging policy with β = β ∗ generates the logged action-context pairs.

Original: Although we have the access to µ ( a | x ; β ∗ ) , we choose to use the maximum likelihood estimator of β ∗ based on { ( a i , x i ) } ni =1 as a surrogate policy parameter, which is denoted as (cid:98) β , to replace β ∗ and obtain a more accurate estimator of the value function V deﬁned in (2.1).
Revised: Although we have the access to µ ( a | x ; β ∗ ) , we choose to use the maximum likelihood estimator of β ∗ based on { ( a i , x i ) } ni =1 as a surrogate policy parameter, which is denoted as (cid:98) β , to replace β ∗ and obtain a more accurate estimator of the value function V deﬁned in (2.1).

Original: In speciﬁc, we deﬁne the MLIPS estimator of V as
Revised: In particular, we deﬁne the MLIPS estimator of V as

Original: [Equation]
Revised: [Equation]

Original: where { ( a i , x i ) } ni =1 are the logged action-context pairs, which are independently sampled from µ ( a | x ; β ∗ ) .
Revised: where { ( a i , x i ) } ni =1 are the logged action-context pairs, which are independently sampled from µ ( a | x ; β ∗ ) .

Original: Unlike the IPS estimator (cid:101) V in (2.2), the MLIPS estimator (cid:98) V in (3.1) is biased in general.
Revised: Unlike the IPS estimator (cid:101) V in (2.2), the MLIPS estimator (cid:98) V in (3.1) is biased in general.

Original: However, (cid:98) V is asymptotically unbiased, and moreover, in the following subsection we show that (cid:98) V achieves a smaller mean squared error than (cid:101) V when the logging distribution µ ( a | x ; β ) is properly chosen.
Revised: However, (cid:98) V is asymptotically unbiased, and moreover, in the following subsection we show that (cid:98) V achieves a smaller mean squared error than (cid:101) V when the logging distribution µ ( a | x ; β ) is properly chosen.

Original: 3. T HEORETICAL A NALYSIS
Revised: 3. T HEORETICAL A NALYSIS

Original: In this section, for the simplicity of analysis, we assume that the reward r is determined by a and x , that is, r = r ( a, x ) .
Revised: In this section, for the simplicity of analysis, we assume that the reward r is determined by a and x , that is, r = r ( a, x ) .

Original: For notational simplicity, in the subsequent analysis we use the following equivalent deﬁnitions of (cid:101) V and (cid:98) V based on joint distributions,
Revised: For notational simplicity, in the subsequent analysis we use the following equivalent deﬁnitions of (cid:101) V and (cid:98) V based on joint distributions,

Original: [Equation]
Revised: [Equation]

Original: since we have
Revised: since we have

Original: [Equation]
Revised: [Equation]

Revised: For the ease of notation, we denote E ( a,x ) ∼ µ ( · , · ; β ∗ ) [ · ] by E [ · ] .

Revised: Also, we denote by (cid:107) · (cid:107) the (cid:96) 2 -norm of a vector or the spectral norm of a matrix.

Revised: For a function f ( β ) , we denote by ∂f ( β ∗ ) /∂β

Revised: its derivative evaluated at β = β ∗ , which is ∂f ( β ) /∂β | β = β ∗ .

Original: Before we lay out the main theory, we ﬁrst introduce the following deﬁnitions.
Revised: Before we lay out the main theory, we ﬁrst introduce the following deﬁnitions.

Original: Deﬁnition 3.1 (Score Function and Fisher Information) .
Revised: Deﬁnition 3.1 (Score Function and Fisher Information) .

Original: The score function S ( a, x ; β ) ∈ R d and the
Revised: The score function S ( a, x ; β ) ∈ R d and the

Original: Fisher Information matrix I ( β ) ∈
Revised: Fisher Information matrix I ( β ) ∈

Original: R
Revised: R

Original: d × d are deﬁned as
Revised: d × d are deﬁned as

Original: [Equation]
Revised: [Equation]

Original: It is worth noting that the reformulation from µ ( a | x ; β ) to µ ( a, x ; β ) does not change the score function or the Fisher information matrix, since the marginal distribution of the context x in (3.2), namely λ ( x ) , does not depend on β , and hence its partial derivative with respect to β is zero.
Revised: It is worth noting that the reformulation from µ ( a | x ; β ) to µ ( a, x ; β ) does not change the score function or the Fisher information matrix, since the marginal distribution of the context x in (3.2), namely λ ( x ) , does not depend on β , and hence its partial derivative with respect to β is zero.

Original: Deﬁnition 3.2 (Deviation Function) .
Revised: Deﬁnition 3.2 (Deviation Function) .

Original: The difference between π ( a, x ) /µ ( a, x ; β ) · r ( a, x ) and the true value V of the target policy, which is deﬁned in (2.1), is denoted by
Revised: The difference between π ( a, x ) /µ ( a, x ; β ) · r ( a, x ) and the true value V of the target policy, which is deﬁned in (2.1), is denoted by

Original: [Equation]
Revised: [Equation]

Original: In the sequel, we introduce two assumptions and a condition on an event.
Revised: In the sequel, we introduce two assumptions and a condition on an event.

Original: For the ease of notation, we denote E ( a,x ) ∼ µ ( · , · ; β ∗ ) [ · ] by E [ · ] .

Original: Also, we denote by (cid:107) · (cid:107) the (cid:96) 2 -norm of a vector or the spectral norm of a matrix.

Original: The next assumption ensures that the Fisher information matrix of the joint logging distribution µ ( a, x ; β ∗ ) is well behaved.
Revised: The next assumption ensures that the Fisher information matrix of the joint logging distribution µ ( a, x ; β ∗ ) is well behaved and the identiﬁability of the model is thus ensured.

Original: Assumption 3.3 (Nonsingular Fisher Information) .
Revised: Assumption 3.3 (Nonsingular Fisher Information) .

Original: For the joint distribution µ ( · , · ; β ∗ ) , we assume that its Fisher information matrix satisﬁes
Revised: For the joint distribution µ ( · , · ; β ∗ ) , we assume that its Fisher information matrix satisﬁes

Original: [Equation]
Revised: [Equation]

Original: The next assumption ensures the score and deviation functions are sufﬁciently smooth.
Revised: The next assumption ensures the score and deviation functions are sufﬁciently smooth.

Original: Assumption 3.4 (Smoothness of Score and Deviation) .
Revised: Assumption 3.4 (Smoothness of Score and Deviation) .

Original: For the deviation function D V ( r, a, x ; β ) deﬁned in (3.4), we assume that[Equation] = O (
Revised: For the deviation function D V ( r, a, x ; β ) deﬁned in (3.4), we assume that

Original: [Equation]
Revised: [Equation]

Original: ) , (3.6) where the second equality holds for any β .
Revised: where the second equality holds for any β .

Original: Meanwhile, for the score function S ( a, x ; β ) deﬁned in (3.3), we assume that for any v ∈ R d such that (cid:107) v (cid:107) = 1 , it holds that
Revised: Meanwhile, for the score function S ( a, x ; β ) deﬁned in (3.3), we assume that for any v ∈ R d such that (cid:107) v [Equation] = 1 , it holds that

Original: [Equation]
Revised: [Equation]

Original: where ∂ 2 S ( a, x ; β ) /∂β 2 is a tensor consisting of the third-order derivative of log µ ( a, x ; β ) with respect to β .
Revised: where ∂ 2 S ( a, x ; β ) /∂β 2 is a tensor consisting of the third-order derivative of log µ ( a, x ; β ) with respect to β .

Original: Here ( ∂ 2 S ( a, x ; β 0 ) /∂β 2 )( v 1 , v 2 , · ) is the bilinear map R d × R d (cid:55)→ R d induced by the tensor, which maps ( v 1 , v 2 ) to a vector in R d .
Revised: Here ( ∂ 2 S ( a, x ; β 0 ) /∂β 2 )( v 1 , v 2 , · ) is the bilinear map R d × R d (cid:55)→ R d induced by the tensor, which maps ( v 1 , v 2 ) to a vector in R d .

Original:  Before introducing the condition, we deﬁne several events to simplify the presentation.
Revised: The Assumptions 3.3-3.4 are satisﬁed, under necessary regularity conditions, by many popular models such as the multinomial logistic regression model which will be illustrated in detail in § D. Before introducing the condition, we deﬁne several events to simplify the presentation.

Original: In the following, we ﬁrst deﬁne an event on the estimation error of the maximum likelihood estimator (cid:98) β .
Revised: In the following, we ﬁrst deﬁne an event on the estimation error of the maximum likelihood estimator (cid:98) β .

Original: Deﬁnition 3.5 (Maximum Likelihood Estimation Error) .
Revised: Deﬁnition 3.5 (Maximum Likelihood Estimation Error) .

Original: For the maximum likelihood estimator (cid:98) β and the true parameter β ∗ , we deﬁne the event E β ( t β ) as
Revised: For the maximum likelihood estimator (cid:98) β and the true parameter β ∗ , we deﬁne the event E β ( t β ) as

Original: [Equation]
Revised: [Equation]

Original: Next we deﬁne several events on the concentration behavior of the score function deﬁned in (3.3) and the deviation function deﬁned in (3.4) as well as their derivatives.
Revised: Next we deﬁne several events on the concentration behavior of the score function deﬁned in (3.3) and the deviation function deﬁned in (3.4) as well as their derivatives.

Original: Deﬁnition 3.6 (Concentration of Score) .
Revised: Deﬁnition 3.6 (Concentration of Score) .

Original: For the score function with true parameter β ∗ , we deﬁne the following events,
Revised: For the score function with true parameter β ∗ , we deﬁne the following events,

Original: [Equation]
Revised: [Equation]

Original: Also, we deﬁne
Revised: Also, we deﬁne

Revised: E ST

Original: [Equation]
Revised: [Equation]

Original: (cid:27)
Revised: t

Original: [Equation]
Revised: [Equation]

Original:  where ( ∂ 2 S ( a, x ; β ) /∂β 2 )( v, v, · ) is deﬁned in Assumption 3.4.
Revised: = [Equation] (cid:27) , where ( ∂ 2 S ( a, x ; β ) /∂β 2 )( v, v, · ) is deﬁned in Assumption 3.4.

Original: Then we deﬁne the joint event
Revised: Then we deﬁne the joint event

Original: E S ( t S ) =
Revised: E S ( t S ) =

Original: E SG ( t SG ) ∩ E SH ( t SH )
Revised: E SG ( t SG ) ∩ E SH ( t SH )

Original: ∩ E ST ( t ST ) , where t S =
Revised: ∩ E ST ( t ST ) , where t S =

Original: ( t SG , t SH , t ST ) .
Revised: ( t SG , t SH , t ST ) .

Original: Deﬁnition 3.7 (Concentration of Deviation) .
Revised: Deﬁnition 3.7 (Concentration of Deviation) .

Original: For deviation function with true parameter β ∗ , namely
Revised: For deviation function with true parameter β ∗ , namely

Original: D V ( r, a, x ; β ∗ ) , we deﬁne
Revised: D V ( r, a, x ; β ∗ ) , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Also, we write
Revised: Also, we write

Original: [Equation]
Revised: [Equation]

Revised: (cid:21)(cid:13)(cid:13)(cid:13)(cid:13)

Original:  Then we deﬁne the joint event E D ( t D ) =
Revised: [Equation] (cid:27) . Then we deﬁne the joint event E D ( t D ) =

Original: E DG ( t DG ) ∩ E DH ( t DH ) , where t D
Revised: E DG ( t DG ) ∩ E DH ( t DH ) , where t D

Original: =
Revised: =

Original: ( t DG , t DH ) .
Revised: ( t DG , t DH ) .

Original: We now present a condition on the overall tail behavior of the probability of the events deﬁned above.
Revised: We now present a condition on the overall tail behavior of the probability of the events deﬁned above.

Original: Condition 3.8 (Concentration Tail Behavior) .
Revised: Condition 3.8 (Concentration Tail Behavior) .

Original: Let t (cid:63) =
Revised: Let t (cid:63) =

Original: ( t β , t S , t D ) and
Revised: ( t β , t S , t D ) and

Original: [Equation]
Revised: [Equation]

Original: it holds that P ( E ( t (cid:63) )) ≥ 1 − f ( t ) , where t = (cid:107) t (cid:63) (cid:107) ∞ for some decreasing tail function f ( t ) such that (cid:82) ∞ 

Revised: it holds that P ( E ( t (cid:63) )) ≥ 1 − f ( t ) , where t = (cid:107) t (cid:63) (cid:107) ∞ for some decreasing tail function f ( t ) such that (cid:82) ∞

Original: t k d f ( t ) is ﬁnite and goes to zero as n → ∞ for any nonnegative integer k and decreases along k .
Revised: t k d f ( t ) is ﬁnite and goes to zero as n → ∞ for any nonnegative integer k and decreases along k .

Original: Here f ( t ) also depends on the dimension d and the sample size n .
Revised: Here f ( t ) also depends on the dimension d and the sample size n .

Original: Condition 3.8 states that, in addition to the upper bound on the estimation error deﬁned in Deﬁnition 3.5, the sample versions of the score function deﬁned in (3.3) and the deviation function deﬁned in (3.4) as well as their derivatives all well concentrate to their population versions.
Revised: Condition 3.8 states that, in addition to the upper bound on the estimation error deﬁned in Deﬁnition 3.5, the sample versions of the score function deﬁned in (3.3) and the deviation function deﬁned in (3.4) as well as their derivatives all well concentrate to their population versions.

Original: In particular, the condition on the integrability of the tail function f ( t ) characterizes that the tail probability of concentration decays faster than the polynomial rate.
Revised: In particular, the condition on the integrability of the tail function f ( t ) characterizes that the tail probability of concentration decays faster than the polynomial rate.

Original: As we will illustrate using the multinomial logistic regression model in § D, such a superpolynomial tail behavior is typically guaranteed by exponential concentration inequalities such as Bernstein-type inequalities and their matrix variants.
Revised: As we will illustrate using the multinomial logistic regression model in § D, such a superpolynomial tail behavior is typically guaranteed by exponential concentration inequalities such as Bernstein-type inequalities and their matrix variants.

Original: The following theorem compares the mean squared errors of the MLIPS and IPS estimators deﬁned in (3.1) and (2.2).
Revised: The following theorem compares the mean squared errors of the MLIPS and IPS estimators deﬁned in (3.1) and (2.2).

Original: Recall that ζ DG , ζ ST , and ζ DH are deﬁned in Assumption 3.4.
Revised: Recall that ζ DG , ζ ST , and ζ DH are deﬁned in Assumption 3.4.

Original: Since ζ DG , ζ ST , and ζ DH are population quantities, they do not scale with the sample size n .
Revised: Since ζ DG , ζ ST , and ζ DH are population quantities, they do not scale with the sample size n .

Original: Theorem 3.9 (Mean Squared Error Reduction) .
Revised: Theorem 3.9 (Mean Squared Error Reduction) .

Original: Under Condition 3.8 and Assumptions 3.3 and 3.4, the MLIPS (cid:98) V is asymptotically unbiased and it holds that
Revised: Under Condition 3.8 and Assumptions 3.3 and 3.4, the MLIPS (cid:98) V is asymptotically unbiased and it holds that

Original: [Equation]
Revised: [Equation]

Original: Reduction of MSE
Revised: Reduction of MSE

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and ξ ( n )=
Revised: and ξ ( n )

Original: [Equation]
Revised: [Equation]

Revised: E [

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See § A for a detailed proof.
Revised: See § A for a detailed proof.

Revised: It is worth noting that the key to our guarantee for MSE reduction lies in the fact that

Revised: [Equation]

Revised: , which is actually an orthogonality relation between ( (cid:101) V − V ) − (1 /n ) · (cid:80) ni =1 Π( r i , a i , x i ; β ∗ ) and (1 /n ) · (cid:80) ni

Revised: =1 Π( r i , a i , x i ; β ∗ ) .

Original: We interpret  Theorem 3.9 as follows.
Revised: We interpret the MSE reduction guarantee in Theorem 3.9 as follows.

Original: For the simplicity of discussion, for now we assume that the dimension d does not scale with the sample size n .
Revised: For the simplicity of discussion, for now we assume that the dimension d does not scale with the sample size n .

Original: The 1 /n · Var(Π( r, x, a ; β ∗ )) term on the right-hand side of (3.13) characterizes the reduction of the mean squared error.
Revised: The 1 /n · Var(Π( r, x, a ; β ∗ )) term on the right-hand side of (3.13) characterizes the reduction of the mean squared error.

Original: For example, in the multinomial logistic regression model, the ξ ( n ) term is roughly of the order 1 /n 3 / 2 .
Revised: For example, in the multinomial logistic regression model, the ξ ( n ) term is roughly of the order 1 /n 3 / 2 .

Original: As a consequence, for a sufﬁciently large n , we have
Revised: As a consequence, for a sufﬁciently large n , we have

Original: [Equation]
Revised: [Equation]

Original: In other words, the mean squared error of the MLIPS estimator is at least smaller than that of the
Revised: In other words, the mean squared error of the MLIPS estimator is at least smaller than that of the

Original: IPS estimator by a margin of 1 / (2 n ) · Var(Π( r, x, a ; β ∗ )) , where Var(Π( r, x, a ; β ∗ )) is a population quantity that does not scale with n .
Revised: IPS estimator by a margin of 1 / (2 n ) · Var(Π( r, x, a ; β ∗ )) , where Var(Π( r, x, a ; β ∗ )) is a population quantity that does not scale with n .

Original: In comparison, the mean squared error is also of the order 1 /n in the multinomial logistic regression model.
Revised: In comparison, the mean squared error is also of the order 1 /n in the multinomial logistic regression model.

Original: That is to say, the reduction effect does not vanish even when n → ∞ .
Revised: That is to say, the reduction effect does not vanish even when n → ∞ .

Original: See § D for more discussions and a more detailed illustration of Theorem 3.9 using the multinomial logistic regression model.
Revised: See § D for more discussions and a more detailed illustration of Theorem 3.9 using the multinomial logistic regression model.

Original: 3. P RACTICAL C ONSIDERATIONS
Revised: 3. P RACTICAL C ONSIDERATIONS

Original: In practice we may not know the class oflogging policies that generate the logs.
Revised: Throughout the theoretical analysis, we assume that the parametrization of the logging policy is known.

Original: However,  we can use universal function approximators such as neural networks to estimate such surrogate policies. 
Revised: However, in practice we may not know the class of logging policies that generate the logs, in which cases we can use universal function approximators such as neural networks to estimate such surrogate policies. The parametrization of the logging distribution may be misspeciﬁed when we use those approximators. However, when universal function approximators such as neural networks are used, the approximation error often diminishes with an increasing number of layers and neurons (see, for example, Schmidt-Hieber (2017); Telgarsky (2016); Yarotsky (2017)).

Revised: Such a diminishing approximation error should enter the Taylor expansions in Lemma A.1-A.2.

Original: In our experiments, we found that neural network policies worked very well on both standard multi-label classiﬁcation datasets and a large-scale ad placement dataset.
Revised: In our experiments, we found that neural network policies worked very well on both standard multi-label classiﬁcation datasets and a large-scale ad placement dataset.

Original: Due to its simplicity, MLIPS can be easily implemented without much extra effort.
Revised: Due to its simplicity, MLIPS can be easily implemented without much extra effort.

Original: Furthermore,
Revised: Furthermore,

Original: MLIPS is orthogonal to other existing approaches for mean squared error reduction in policy evaluation and optimization.
Revised: MLIPS is orthogonal to other existing approaches for mean squared error reduction in policy evaluation and optimization.

Original: Hence, it can often be combined straightforwardly with them.
Revised: Hence, it can often be combined straightforwardly with them.

Original: In our experiments, we observed that it can be used to further improve POEM for policy optimization.
Revised: In our experiments, we observed that it can be used to further improve POEM for policy optimization.

Original: Another advantage of our approach is that it still works even when we have no access to the logging policy, which often happens in real world due to various reasons.
Revised: Another advantage of our approach is that it still works even when we have no access to the logging policy, which often happens in real world due to various reasons.

Original: In those cases, IPS may not work at all, but our approach exhibits superior performance, as shown in our experiments.
Revised: In those cases, IPS may not work at all, but our approach exhibits superior performance, as shown in our experiments.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically evaluate the effectiveness of the proposed method on a number of datasets.
Revised: In this section, we empirically evaluate the effectiveness of the proposed method on a number of datasets.

Original: We ﬁrst check the reduction of the mean squared error enabled by the MLIPS estimator.
Revised: We ﬁrst check the reduction of the mean squared error enabled by the MLIPS estimator.

Original: Then we apply IPS, MLIPS, POEM and MLPOEM to policy optimization and compare their performance on several datasets.
Revised: Then we apply IPS, MLIPS, POEM and MLPOEM to policy optimization and compare their performance on several datasets.

Original: 4. S ETTINGS
Revised: 4. S ETTINGS

Original: Datasets.
Revised: Datasets.

Original: First, we adopt the same experimental design as the one speciﬁed in Swaminathan &
Revised: First, we adopt the same experimental design as the one speciﬁed in Swaminathan &

Original: Joachims (2015a) to generate batch contextual bandit learning datasets from supervised learning datasets.
Revised: Joachims (2015a) to generate batch contextual bandit learning datasets from supervised learning datasets.

Original: Four multi-label datasets are selected from the LibSVM repository (see Swaminathan &
Revised: Four multi-label datasets are selected from the LibSVM repository (see Swaminathan &

Original: Joachims (2015a) for more details).
Revised: Joachims (2015a) for more details).

Original: To construct a batch contextual bandit learning dataset, we take a supervised learning dataset D ∗ =
Revised: To construct a batch contextual bandit learning dataset, we take a supervised learning dataset D ∗ =

Original: { ( x i , y ∗ i ) } ni =1 , simulate using a logging policy µ by sampling y i ∼ µ ( · | x i ) , and collect the reward as the negative Hamming distance − ∆( y ∗ i , y i ) between the supervised label and the sampled label.
Revised: { ( x i , y ∗ i ) } ni =1 , simulate using a logging policy µ by sampling y i ∼ µ ( · | x i ) , and collect the reward as the negative Hamming distance − ∆( y ∗ i , y i ) between the supervised label and the sampled label.

Original: We use a Conditional Random Field (CRF) model (Sutton &
Revised: We use a Conditional Random Field (CRF) model (Sutton &

Original: McCallum, 2012) trained on 5% of the data as the logging policy µ .
Revised: McCallum, 2012) trained on 5% of the data as the logging policy µ .

Original: We repeat this procedure four times to generate the entire bandit feedback dataset D = { ( [Equation]
Revised: We repeat this procedure four times to generate the entire bandit feedback dataset D = { ( x i , y i , r i = − ∆( y ∗ i , y i ) , µ ( y i | x i )) } ni =1 .

Original: Both the training sets and held-out testing sets are constructed in the same way as suggested in
Revised: Both the training sets and held-out testing sets are constructed in the same way as suggested in

Original: Swaminathan & Joachims (2015a).
Revised: Swaminathan & Joachims (2015a).

Original: In addition to these standard datasets, we also evaluate the effectiveness of our method on a large-scale ad placement dataset.
Revised: In addition to these standard datasets, we also evaluate the effectiveness of our method on a large-scale ad placement dataset.

Original: The dataset is collected by Criteo, a leader in the display advertising industry (Lefortier et al., 2016).
Revised: The dataset is collected by Criteo, a leader in the display advertising industry (Lefortier et al., 2016).

Original: For this dataset, we consider an ad display scenario where a policy selects the products to be displayed on a website when a user arrives.
Revised: For this dataset, we consider an ad display scenario where a policy selects the products to be displayed on a website when a user arrives.

Original: Provided the candidate set of products and impression context, we hope the policy to place ads such that the aggregated click-through-rate (CTR) or the number of clicks is maximized.
Revised: Provided the candidate set of products and impression context, we hope the policy to place ads such that the aggregated click-through-rate (CTR) or the number of clicks is maximized.

Original: We consider a subset of the dataset, which is used in the NIPS 2017 Criteo Ad Placement Challenge.
Revised: We consider a subset of the dataset, which is used in the NIPS 2017 Criteo Ad Placement Challenge.

Original: Each ad is represented by a 74000 -dimensional feature vector.
Revised: Each ad is represented by a 74000 -dimensional feature vector.

Original: For each context (or impression), the logged action of selecting a candidate ad, a binary reward value on whether the displayed product is clicked by the user, and the probability that the logging policy picks such a speciﬁc candidate are recorded.
Revised: For each context (or impression), the logged action of selecting a candidate ad, a binary reward value on whether the displayed product is clicked by the user, and the probability that the logging policy picks such a speciﬁc candidate are recorded.

Original: More details can be found at CrowdAI (2017).
Revised: More details can be found at CrowdAI (2017).

Original: Surrogate policy estimation.
Revised: Surrogate policy estimation.

Original: To ﬁt the surrogate policy, we apply two models, a simple logistic regression model and a one-hidden-layer perceptron with ten ReLU activation units, both with (cid:96) 2regularization.
Revised: To ﬁt the surrogate policy, we apply two models, a simple logistic regression model and a one-hidden-layer perceptron with ten ReLU activation units, both with (cid:96) 2regularization.

Original: Here ﬁve-fold cross validation is used to tune the regularization parameter.
Revised: Here ﬁve-fold cross validation is used to tune the regularization parameter.

Original: We use the
Revised: We use the

Original: L-BFGS algorithm (Liu & Nocedal, 1989) for optimization.
Revised: L-BFGS algorithm (Liu & Nocedal, 1989) for optimization.

Original: Policy optimization.
Revised: Policy optimization.

Original: We consider IPS, POEM (Swaminathan & Joachims, 2015a), and NormPOEM (Swaminathan & Joachims, 2015b).
Revised: We consider IPS, POEM (Swaminathan & Joachims, 2015a), and NormPOEM (Swaminathan & Joachims, 2015b).

Original: We compare their performance with MLIPS as well as the combination of our surrogate policy technique with POEM (MLPOEM).
Revised: We compare their performance with MLIPS as well as the combination of our surrogate policy technique with POEM (MLPOEM).

Original: There are several hyperparameters in POEM, Norm-POEM, and MLPOEM, including the thresholding parameter and the regularization parameter.
Revised: There are several hyperparameters in POEM, Norm-POEM, and MLPOEM, including the thresholding parameter and the regularization parameter.

Original: We use ﬁve-fold cross validation on the training set to tune them.
Revised: We use ﬁve-fold cross validation on the training set to tune them.

Original: For reward maximization, following the procedures of POEM, we initialize the model weights to zeros, use mini-batch AdaGrad (Duchi et al., 2011) with a batch size of 100 and step size η = 1 .
Revised: For reward maximization, following the procedures of POEM, we initialize the model weights to zeros, use mini-batch AdaGrad (Duchi et al., 2011) with a batch size of 100 and step size η = 1 .

Original: On all datasets, we repeat the experiments ten times to report the average generalization performance.
Revised: On all datasets, we repeat the experiments ten times to report the average generalization performance.

Original: 4. M EAN S QUARED E RROR R EDUCTION IN P OLICY E STIMATION
Revised: 4. M EAN S QUARED E RROR R EDUCTION IN P OLICY E STIMATION

Original: LYRL dataset. First, we check whether the MLIPS estimator has a smaller mean squared error than the vanilla IPS estimator.
Revised: First, we check whether the MLIPS estimator has a smaller mean squared error than the vanilla IPS estimator.

Original: Instead of comparing the estimation of the value function, we focus on comparing the performance of MLIPS and IPS on off-policy gradient estimation, whose accuracy is crucial to policy optimization.
Revised: Instead of comparing the estimation of the value function, we focus on comparing the performance of MLIPS and IPS on off-policy gradient estimation, whose accuracy is crucial to policy optimization.

Original: We use the LYRL dataset (Swaminathan & Joachims, 2015a) and combine the training and testing sets to obtain a large bandit feedback dataset.
Revised: We use the LYRL dataset (Swaminathan & Joachims, 2015a) and combine the training and testing sets to obtain a large bandit feedback dataset.

Original: Given a reasonably accurate policy weight vector (after training with 10000 mini-batch updates), we compute its IPS-based gradient estimator on the entire dataset to get a rough “ground-truth” gradient.
Revised: Given a reasonably accurate policy weight vector (after training with 10000 mini-batch updates), we compute its IPS-based gradient estimator on the entire dataset to get a rough “ground-truth” gradient.

Original: We further construct a series of data subsets to evaluate the gradient estimation with smaller sample sizes, with 2 − 1 / 2 , 2 − 2 / 2 , . . .
Revised: We further construct a series of data subsets to evaluate the gradient estimation with smaller sample sizes, with 2 − 1 / 2 , 2 − 2 / 2 , . . .

Original: , 2 − k/ 2 of the entire dataset.
Revised: , 2 − k/of the entire dataset.

Original: For the neural network estimate of surrogate policy (NN-surrogate), we ﬁrst ﬁt a logistic regression model and a one-hidden-layer perceptron with ten ReLU activation units to estimate the surrogate policy.
Revised: For the neural network estimate of surrogate policy (NN-surrogate), we ﬁrst ﬁt a logistic regression model and a one-hidden-layer perceptron with ten ReLU activation units to estimate the surrogate policy.

Original: After this step, we compute the Euclidean distance between the “ground-truth” gradient and the gradient estimators that correspond to smaller sample sizes.
Revised: After this step, we compute the Euclidean distance between the “ground-truth” gradient and the gradient estimators that correspond to smaller sample sizes.

Original: Average Euclidean distances and standard deviations are computed over twenty trials.
Revised: Average Euclidean distances and standard deviations are computed over twenty trials.

Original: Results are illustrated in Figure 1.
Revised: Results are illustrated in Figure 1.

Original: We observe that the MLIPS estimator gives much better gradient estimation than the original IPS estimator.
Revised: We observe that the MLIPS estimator gives much better gradient estimation than the original IPS estimator.

Original: In particular, the relative improvement is signiﬁcant especially when the sample size is small.
Revised: In particular, the relative improvement is signiﬁcant especially when the sample size is small.

Original: It is also interesting to see that the difference between the linear surrogate policy and the neural network surrogate policy is very small, possibly because the actual logging CRF policy is linear.
Revised: It is also interesting to see that the difference between the linear surrogate policy and the neural network surrogate policy is very small, possibly because the actual logging CRF policy is linear.

Original: On the Criteo dataset, in which the logging policy is unknown and possibly very complicated, we ﬁnd that the neural network surrogate policy works better than the linear policy.
Revised: On the Criteo dataset, in which the logging policy is unknown and possibly very complicated, we ﬁnd that the neural network surrogate policy works better than the linear policy.

Original: 4. G ENERALIZATION P ERFORMANCE OF P OLICY O PTIMIZATION
Revised: 4. G ENERALIZATION P ERFORMANCE OF P OLICY O PTIMIZATION

Original: To evaluate the effectiveness of our method on policy optimization, we train logistic regression policies with IPS, POEM, Norm-POEM objective functions, and the IPS and POEM versions with maximum likelihood surrogate policies (MLIPS and MLPOEM).
Revised: To evaluate the effectiveness of our method on policy optimization, we train logistic regression policies with IPS, POEM, Norm-POEM objective functions, and the IPS and POEM versions with maximum likelihood surrogate policies (MLIPS and MLPOEM).

Original: As mentioned before, we use both the logistic regression model and the one-hidden-layer perceptron model to ﬁt the surrogate policies, which are respectively marked with “-Lin” and “-NN” sufﬁxes in the names of the training methods.
Revised: As mentioned before, we use both the logistic regression model and the one-hidden-layer perceptron model to ﬁt the surrogate policies, which are respectively marked with “-Lin” and “-NN” sufﬁxes in the names of the training methods.

Original: Results are summarized in Table 1.
Revised: Results are summarized in Table 1.

Original: The test set Hamming losses for all methods are averaged over ten runs.
Revised: The test set Hamming losses for all methods are averaged over ten runs.

Original: On each run, both MLIPS and MLPOEM consistently and signiﬁcantly outperforms their corresponding original versions. 
Revised: On each run, both MLIPS and MLPOEM consistently and signiﬁcantly outperforms their corresponding original versions. [Table]

Original: [Table]
Revised: [Table]

Original: As a control, we study the scenario where the logging policy is not available.
Revised: As a control, we study the scenario where the logging policy is not available.

Original: We train the policies using IPS while treating the logging policy to be uniform.
Revised: We train the policies using IPS while treating the logging policy to be uniform.

Original: The performance of such a variant of IPS (thelast row in the table withalgorithm name “IPS-Uniform”) is signiﬁcantly worse than our method, which also does not require knowing the logging policy.
Revised: The performance of such a variant of IPS (the row in the table with name “IPS-Uniform”) is signiﬁcantly worse than our method, which also does not require knowing the logging policy.

Original: Notably, when compared against (Norm-)POEM, the stateof-the-art algorithm on these datasets,
Revised: Notably, when compared against (Norm-)POEM, the stateof-the-art algorithm on these datasets,

Original: our
Revised: our

Original: MLPOEM algorithm can augment the generalization performance on most datasets.
Revised: MLPOEM algorithm can augment the generalization performance on most datasets.

Original: This indicates that our surrogate policy technique provides a complementary mean squared error reduction effect for existing methods in this ﬁeld.
Revised: This indicates that our surrogate policy technique provides a complementary mean squared error reduction effect for existing methods in this ﬁeld.

Original: Finally, it is worth noting that the methods with neural network surrogate polices and linear policies are not too much different in terms of performance on these datasets, possibly because the logging policies used to generate these datasets are linear.
Revised: Finally, it is worth noting that the methods with neural network surrogate polices and linear policies are not too much different in terms of performance on these datasets, possibly because the logging policies used to generate these datasets are linear.

Original: 4.3. C RITEO A D P LACEMENT D ATASET
Revised: 4.3. C RITEO A D P LACEMENT D ATASET

Original: We also evaluate the performance of our method on the preprocessed data used in the NIPS
Revised: We also evaluate the performance of our method on the preprocessed data used in the NIPS

Original: Workshop Criteo ad placement challenge.
Revised: Workshop Criteo ad placement challenge.

Original: Because there is no information about the logging policy used to generate the data, it is reasonable to use a complex universal function approximator to ﬁt the surrogate policy.
Revised: Because there is no information about the logging policy used to generate the data, it is reasonable to use a complex universal function approximator to ﬁt the surrogate policy.

Original: We implement a neural network model with 21 tanh activation units for both the surrogate policy and the target policy.
Revised: We implement a neural network model with 21 tanh activation units for both the surrogate policy and the target policy.

Original: Due to the massive size of this dataset and its high-dimensional and sparse feature representation, we implement the (cid:96) 2 -regularized IPS and MLIPS in C++ with multi-threading.
Revised: Due to the massive size of this dataset and its high-dimensional and sparse feature representation, we implement the (cid:96) 2 -regularized IPS and MLIPS in C++ with multi-threading.

Original: Both the maximum likelihood estimation and policy optimization are performed using the L-BFGS algorithm.
Revised: Both the maximum likelihood estimation and policy optimization are performed using the L-BFGS algorithm.

Original: We split the dataset into training ( 80% ) and testing ( 20% ) sets.
Revised: We split the dataset into training ( 80% ) and testing ( 20% ) sets.

Original: The regularization parameter is chosen by ﬁve-fold cross-validation on the training set.
Revised: The regularization parameter is chosen by ﬁve-fold cross-validation on the training set.

Original: Evaluation is performed by computing the inverse propensity weights as suggested by the NIPS challenge.
Revised: Evaluation is performed by computing the inverse propensity weights as suggested by the NIPS challenge.

Original: In comparison with
Revised: In comparison with

Original: IPS, which obtains 0 .
Revised: IPS, which obtains 0 .

Original: 556 on the training set and 0 .
Revised: 556 on the training set and 0 .

Original: 551 on the testing set, our method obtains 0 .on
Revised: 551 on the testing set, our method obtains 0 .on

Original: the training set and 0 .
Revised: the training set and 0 .

Original: 572 on the testing set.
Revised: 572 on the testing set.

Original: An implementation of our method ( 0 .
Revised: An implementation of our method ( 0 .

Original: 576 on the training set and 0 .
Revised: 576 on the training set and 0 .

Original: 568 on the testing set, which are evaluated on the hold out data for the competition) is also ranked among very top (prize winning teams) of the NIPS 2017 challenge leaderboard.
Revised: 568 on the testing set, which are evaluated on the hold out data for the competition) is also ranked among very top (prize winning teams) of the NIPS 2017 challenge leaderboard.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce a new but simple approach for mean squared error reduction in policy evaluation and optimization.
Revised: We introduce a new but simple approach for mean squared error reduction in policy evaluation and optimization.

Original: Theoretical analysis illustrates that the proposed MLIPS estimator is asymptotically unbiased, and moreover, has a smaller mean squared error than the classical IPS estimator.
Revised: Theoretical analysis illustrates that the proposed MLIPS estimator is asymptotically unbiased, and moreover, has a smaller mean squared error than the classical IPS estimator.

Original: Experimental results on several large-scale datasets also demonstrate the empirical effectiveness of the proposed estimator.
Revised: Experimental results on several large-scale datasets also demonstrate the empirical effectiveness of the proposed estimator.

Original: Our technique can also be combined with existing approaches and further improves the performance.
Revised: Our technique can also be combined with existing approaches and further improves the performance.

Original: A P ROOF OF T HEOREM 3. 
Revised: A P ROOF OF T HEOREM 3.

Original: The proof of Theorem 3.9 is established based on the following two lemmas.
Revised: The proof of Theorem 3.9 is established based on the following two lemmas.

Original: First, we introduce a lemma that relates the errors of the MLIPS and IPS estimators.
Revised: First, we introduce a lemma that relates the errors of the MLIPS and IPS estimators.

Original: Recall that, as deﬁned in
Revised: Recall that, as deﬁned in

Original: Deﬁnitions 3.5-3.7 and Condition 3.8, we have t (cid:63) =
Revised: Deﬁnitions 3.5-3.7 and Condition 3.8, we have t (cid:63) =

Original: ( t β , t S , t D ) where t S =
Revised: ( t β , t S , t D ) where t S =

Original: ( t SG , t SH , t ST ) and t D =
Revised: ( t SG , t SH , t ST ) and t D =

Original: ( t DG , t DH ) .
Revised: ( t DG , t DH ) .

Original: Also, recall that ζ DG , ζ DH , and ζ ST are deﬁned in Assumption 3.4.
Revised: Also, recall that ζ DG , ζ DH , and ζ ST are deﬁned in Assumption 3.4.

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For the MLIPS estimator (cid:98) V deﬁned in (3.1) and the IPS estimator (cid:101) V deﬁned in (2.2), under Assumption 3.4 and conditioning on the event E ( t (cid:63) ) deﬁned in (3.12), we have
Revised: For the MLIPS estimator (cid:98) V deﬁned in (3.1) and the IPS estimator (cid:101) V deﬁned in (2.2), under Assumption 3.4 and conditioning on the event E ( t (cid:63) ) deﬁned in (3.12), we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See § B for a detailed proof.
Revised: See § B for a detailed proof.

Original: The following lemma characterizes the (cid:98) β − β ∗ term on the right-hand side of (A.1), which is the estimation error of (cid:98) β .
Revised: The following lemma characterizes the (cid:98) β − β ∗ term on the right-hand side of (A.1), which is the estimation error of (cid:98) β .

Original: Lemma A.2.
Revised: Lemma A.2.

Original: For the maximum likelihood estimator (cid:98) β and the true parameter β ∗ , we have
Revised: For the maximum likelihood estimator (cid:98) β and the true parameter β ∗ , we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Here λ S ∈ [0 , 1] , I − 1 ( β ∗ ) is the inverse of the Fisher information matrix deﬁned in (3.3), and ( ∂ 2 S ( a, x ; β ∗ ) /∂β 2 )( v 1 , v 2 , · ) is the bilinear map deﬁned in Assumption 3.4.
Revised: Here λ S ∈ [0 , 1] , I − 1 ( β ∗ ) is the inverse of the Fisher information matrix deﬁned in (3.3), and ( ∂ 2 S ( a, x ; β ∗ ) /∂β 2 )( v 1 , v 2 , · ) is the bilinear map deﬁned in Assumption 3.4.

Original: Proof.
Revised: Proof.

Original: See § C for a detailed proof.
Revised: See § C for a detailed proof.

Original: Lemma A.2 allows us to more precisely characterize the (cid:98) β − β ∗ term in (A.1).
Revised: Lemma A.2 allows us to more precisely characterize the (cid:98) β − β ∗ term in (A.1).

Original: Now we are ready to prove Theorem 3.9. Proof.
Revised: Now we are ready to prove Theorem 3.9. Proof of Theorem 3.9.

Original: By combining Lemmas A.1 and A.2, under Assumptions 3.3 and 3.4 and conditioning on the event E ( t (cid:63) ) deﬁned in (3.12), we have
Revised: By combining Lemmas A.1 and A.2, under Assumptions 3.3 and 3.4 and conditioning on the event E ( t (cid:63) ) deﬁned in (3.12), we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: For the asymptotic expectation of the MLIPS estimator (cid:98) V  ,  as (cid:101) V is an unbiased estimator of V , we have
Revised: First, we show that the MLIPS (cid:101) V is asymptotically unbiased. Note that we have E [ S ( a, x ; β ∗ )] = 0 , the expectation of the second term on the right hand side of (A.3) is zero. Thus, as (cid:101) V is an unbiased estimator of V , we have

Original: [Equation]
Revised: [Equation]

Original: By Condition 3.8 and the deﬁnition of η ( t (cid:63) ) in (A.4), we have
Revised: By Condition 3.8 and the deﬁnition of η ( t (cid:63) ) in (A.4), we have

Original: [Equation]
Revised: [Equation]

Original: A [Equation] [Equation] where t = (cid:107) t (cid:63) (cid:107) ∞ .
Revised: where t = (cid:107) t (cid:63) (cid:107) ∞ .

Original: As deﬁned in Condition 3.8, (cid:82) ∞ 

Revised: As deﬁned in Condition 3.8, (cid:82) ∞

Original: t k d f ( t ) goes to zero when n → ∞ for any k .
Revised: t k d f ( t ) goes to zero when n → ∞ for any k .

Original: Therefore, when n → ∞ , from (A.6) we have
Revised: Therefore, when n → ∞ , from (A.6) we have

Original: [Equation]
Revised: [Equation]

Original: which gives lim n →∞ E [ η ( t (cid:63) )]
Revised: which gives lim n →∞ E [ η ( t (cid:63) )]

Original: = 0 .
Revised: = 0 .

Original: Thus, we have lim n →∞ [ (cid:98) V − V ] = 0 , which means that the
Revised: Thus, we have lim n →∞ [ (cid:98) V − V ] = 0 , which means that the

Original: MLIPS estimator (cid:98) V is asymptotically unbiased.
Revised: MLIPS estimator (cid:98) V is asymptotically unbiased.

Revised: Next, we proceed to prove the MSE reducing effect of (cid:98) V .

Original: For the mean squared error of the MLIPS estimator (cid:98) V , we have
Revised: For the mean squared error of the MLIPS estimator (cid:98) V , we have

Original: [Equation]
Revised: [Equation]

Original:  where for term (ii) we use the law of total expectation.
Revised: 8) where for term (ii) we use the law of total expectation.

Original: In the sequel, we characterize terms (i)-(iii) respectively.
Revised: In the sequel, we characterize terms (i)-(iii) respectively.

Original: Term (i) in (A.8) : Note that by Deﬁnition 3.2 we have
Revised: Term (i) in (A.8) : Note that by Deﬁnition 3.2 we have

Original: [Equation]
Revised: [Equation]

Original: i=
Revised: i

Original:  D V ( r i , a i , x i ; β ∗ ) , where D V ( r, a, x ; β ) =
Revised: [Equation] D V ( r i , a i , x i ; β ∗ ) , where D V ( r, a, x ; β ) =

Original: π ( a, x ) µ ( a, x ; β ) · r ( a, x ) − V.
Revised: π ( a, x ) µ ( a, x ; β ) · r ( a, x ) − V.

Original: We reformulate the ﬁrst term on the right-hand side of (A.8) as
Revised: We reformulate the ﬁrst term on the right-hand side of (A.8) as

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Revised: r i , a i , x i ; β ∗ ) (cid:19) · (cid:18) n (cid:88) i =

Revised: Π( r i , a i , x i ; β ∗ ) (cid:19)(cid:21) .

Original:  Then by the fact that E [ S ( a, x ; β ∗ )] = 0 , for the second term on the right-hand side of (A.9) we have
Revised: (A.9) Then by the fact that E [ S ( a, x ; β ∗ )] = 0 , for the second term on the right-hand side of (A.9) we have

Original: [Equation] (A [Equation]
Revised: E (cid:2) Π( r, a, x ; β ∗ ) (cid:3) = E (cid:2) D V ( r, a, x ; β ∗ ) · S ( a, x ; β ∗ ) (cid:62) (cid:3) · I − 1 ( β ∗ ) · E (cid:2) S ( a, x ; β ∗ ) (cid:3) = 0 .

Original: [Equation]
Revised: (A.10)

Original: Meanwhile, we have
Revised: Meanwhile, we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: where the second equality follows from (A.10).
Revised: = [Equation] = n [Equation]

Original: Hence
Revised: E

Original: [Equation]
Revised: [Equation]

Revised: , (A.11) where the second equality follows from (A.10).

Original:  we obtain from (A.9) and (A.11)
Revised: Hence, we obtain from (A.9) and (A.11)

Original: that
Revised: that

Revised: E (cid:20)(cid:18) ( (cid:101) V −

Original: [Equation]
Revised: [Equation]

Original: Note that we have the following equality for the Fisher information matrix I ( β ∗ ) ,[Equation] ) /∂β.
Revised: Note that we have the following equality for the Fisher information matrix I ( β ∗ ) ,

Original: [Equation]
Revised: [Equation]

Original: Then by the deﬁnition of Π( r, a, x ; β ∗ ) in (3.14), we have
Revised: Then by the deﬁnition of Π( r, a, x ; β ∗ ) in (3.14), we have

Original: [Equation]
Revised: [Equation]

Original: which by (A.13) and the deﬁnition of Π( r, a, x ; β ∗ ) implies
Revised: which by (A.13) and the deﬁnition of Π( r, a, x ; β ∗ ) implies

Original: [Equation]
Revised: [Equation]

Original: Note that (A.14) implies for the third term on the right-hand side of (A.12), we have
Revised: Note that (A.14) implies for the third term on the right-hand side of (A.12), we have

Original: [Equation]
Revised: [Equation]

Original: where the second equality follows from (A.10).
Revised: where the second equality follows from (A.10).

Original: Plugging (A.15) into (A.12) [Equation] we obtain
Revised: Plugging (A.15) into (A.12), we obtain

Original: [Equation]
Revised: [Equation]

Original: Term (ii) in (A.8) :
Revised: Term (ii) in (A.8) :

Original: By Condition 3.8 and the deﬁnition of η ( t (cid:63) ) in (A.4), we have
Revised: By Condition 3.8 and the deﬁnition of η ( t (cid:63) ) in (A.4), we have

Original: [Equation]
Revised: [Equation]

Original:  where t = (cid:107) t (cid:63) (cid:107) ∞ .
Revised: 17) where t = (cid:107) t (cid:63) (cid:107) ∞ .

Original: As deﬁned in Condition 3.
Revised: As deﬁned in Condition 3.

Original: [Equation]
Revised: [Equation]

Original: (cid:82) ∞ 

Revised: (cid:82) ∞

Original: t k d f ( t ) is a decreasing function of k .
Revised: t k d f ( t ) is a decreasing function of k .

Original: Therefore, from (A.17) we have
Revised: Therefore, from (A.17) we have

Original: [Equation]
Revised: [Equation]

Original: Term (iii) in (A.8) :
Revised: Term (iii) in (A.8) :

Original: By the law of total expectation
Revised: By the law of total expectation, we have

Original: [Equation]
Revised: [Equation]

Original: we have

Original: [Equation]

Original: For the ﬁrst term on the right-hand side of (A.19), by the Cauchy-Schwartz inequality for expectation, we have
Revised: For the ﬁrst term on the right-hand side of (A.19), by the Cauchy-Schwartz inequality for expectation, we have

Original: [Equation]
Revised: [Equation]

Original: where the second equality follows from (A.18).
Revised: where the second equality follows from (A.18).

Original: Now we consider the second term on the right-hand side of (A.19).
Revised: Now we consider the second term on the right-hand side of (A.19).

Original: Note that by the deﬁnition of Π( r, a, x ; β ) in (A.3)
Revised: Note that by the deﬁnition of Π( r, a, x ; β ) in (A.3), we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: (A

Original: [Equation]
Revised: [Equation]

Original: ) (cid:62) (cid:3)(cid:13)(cid:13) · (cid:107) I − 1 ( β ∗ ) (cid:107) · (cid:13)(cid:13)(cid:13)(cid:13) 1 n n (cid:88) i =

Original: S ( a i , x i ; β ∗ ) (cid:13)(cid:13)(cid:13)(cid:13) , (A.21) where by (A.23) we have that, conditioning on the event E ( t (cid:63) ) deﬁned in (3.12), (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: n (cid:88) i = S
Revised: where by (A.23) we have that, conditioning on the event E ( t (cid:63) ) deﬁned in (3.12),

Original: [Equation]
Revised: [Equation]

Original: a i , x
Revised: n

Original: [Equation]
Revised: [Equation]

Original: ) (cid:13)(cid:13)(cid:13)(cid:13) ≤ t SG .

Original: Under Assumption 3.4, we prove in Lemma B.1 and (B.2) that
Revised: Under Assumption 3.4, we prove in Lemma B.1 and (B.2) that

Original: [Equation]
Revised: [Equation]

Original: Note that in the event E SG ( t SG ) deﬁned in (3.9), we have E [ S ( a, x ; β ∗ )]
Revised: Note that in the event E SG ( t SG ) deﬁned in (3.9), we have E [ S ( a, x ; β ∗ )]

Original: = 0 .
Revised: = 0 .

Original: Hence, equivalently we have
Revised: Hence, equivalently we have

Original: [Equation]
Revised: [Equation]

Original: Combining (A.4), (A.21), (A.22), (A.23), and Assumption 3.3, under Condition 3.8 we have 
Revised: Combining (A.4), (A.21), (A.22), (A.23), and Assumption 3.3, under Condition 3.8 we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: where t = (cid:107) t (cid:63) (cid:107) ∞ .
Revised: where t = (cid:107) t (cid:63) (cid:107) ∞ .

Original: By (A.24), using a similar argument to the one used to obtain (A.18)
Revised: By (A.24) [Equation] using a similar argument to the one used to obtain (A.18)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Revised: Thus

Original: Thus, combining (A.20) and (A.25), for (A.19) we obtain
Revised: [Equation] combining (A.20) and (A.25) [Equation] for (A.19) we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that term (ii) in (A.8), which is characterized by (A.18), is dominated by the last term on the right-hand side of (A.25), since (cid:82) ∞ 

Revised: Note that term (ii) in (A.8), which is characterized by (A.18), is dominated by the last term on the right-hand side of (A.25), since (cid:82) ∞

Original: t k d f ( t ) decreases along k under Condition 3.8.
Revised: t k d f ( t ) decreases along k under Condition 3.8.

Original: Plugging (A.16), (A.18), and (A.26) into (A.8), we obtain
Revised: Plugging (A.16), (A.18), and (A.26) into (A.8), we obtain

Original: E(cid:2) [Equation] n [Equation]
Revised: E

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of Theorem 3.9.
Revised: which concludes the proof of Theorem 3.9.

Original: B P ROOF OF L EMMA A. 
Revised: B P ROOF OF L EMMA A.

Original: Before proving the Lemma A.1, we ﬁrst introduce the following lemma on the gradient of the deviation function D V ( r, a, x ; β ) .
Revised: Before proving the Lemma A.1, we ﬁrst introduce the following lemma on the gradient of the deviation function D V ( r, a, x ; β ) .

Original: Lemma B.1.
Revised: Lemma B.1.

Original: For the score and deviation functions deﬁned in (3.3) and (3.4), we have
Revised: For the score and deviation functions deﬁned in (3.3) and (3.4), we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Deﬁnition 3.2, we have
Revised: By Deﬁnition [Equation] [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Then using the fact that E (cid:2) S ( a, x ; β ∗ ) (cid:3) = 0 , we have
Revised: Then using the fact that E (cid:2) S ( a, x ; β ∗ ) (cid:3) = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: where the second equality follows from (3.3) and the last equality follows from (B.1).
Revised: where the second equality follows from (3.3) and the last equality follows from (B.1).

Original: Hence, we conclude the proof of Lemma B.1.
Revised: Hence, we conclude the proof of Lemma B.1.

Original: Note that by Lemma B.1 and Assumption 3.4, we also have the following equality,
Revised: Note that by Lemma B.1 and Assumption 3.4, we also have the following equality,

Original: [Equation]
Revised: [Equation]

Original: Now we proceed to prove Lemma A.1. Proof.
Revised: Now we proceed to prove Lemma A.1. Proof of Lemma A.1.

Original: Recall that we have
Revised: Recall that we have

Original: [Equation]
Revised: [Equation]

Original: by which we have
Revised: by which we have

Original: [Equation]
Revised: [Equation]

Original: B
Revised: a i

Original: [Equation]
Revised: [Equation]

Original:  Then by applying the Taylor expansion to the second equality in (B.3), we have
Revised: (B.3) Then by applying the Taylor expansion to the second equality in (B.3), we have

Original: [Equation]
Revised: [Equation]

Original: B

Original: [Equation]

Original: [Equation]

Original: where β D = λ D β ∗
Revised: where β D = λ D β ∗

Original: + (1 − λ D ) (cid:98) β for some λ D ∈ [0 , 1] , and the last equality follows from Assumption 3.4 and the fact that we condition on the event E ( t (cid:63) ) deﬁned in (3.12).
Revised: + (1 − λ D ) (cid:98) β for some λ D ∈ [0 , 1] , and the last equality follows from Assumption 3.4 and the fact that we condition on the event E ( t (cid:63) ) deﬁned in (3.12).

Original: By Lemma B.1, for the second term on the right-hand side of (B.4), we have
Revised: By Lemma B.1, for the second term on the right-hand side of (B.4), we have

Original: [Equation]
Revised: [Equation]

Original: 5) where
Revised: ) where

Original: [Equation]
Revised: [Equation]

Original: Meanwhile, conditioning on the event E ( t (cid:63) ) deﬁned in (3.12), we have
Revised: Meanwhile, conditioning on the event E ( t (cid:63) ) deﬁned in (3.12), we have

Original: [Equation]
Revised: [Equation]

Original: Combining (B.6), (B.5), and (B.4), we have
Revised: Combining (B.6), (B.5), and (B.4), we have

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of Lemma A.1.
Revised: which concludes the proof of Lemma A.1.

Original: C P ROOF OF L EMMA A. 
Revised: C P ROOF OF L EMMA A.

Original: Proof.
Revised: Proof.

Original: First, recall that we have
Revised: First, recall that we have

Original: [Equation]
Revised: [Equation]

Original: Since (cid:98) β is the maximum likelihood estimator of the true parameter β ∗ based on { ( a i , x i ) } ni =1 , we have
Revised: Since (cid:98) β is the maximum likelihood estimator of the true parameter β ∗ based on { ( a i , x i ) } ni =1 , we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: we expand the above sum to obtain
Revised: we expand the above sum to obtain

Original: [Equation]
Revised: [Equation]

Original: C [Equation] where β S = λ S β ∗ + (1 − λ S ) (cid:98) β for some λ S ∈ [0 , 1] .
Revised: where β S = λ S β ∗ + (1 − λ S ) (cid:98) β for some λ S ∈ [0 , 1] .

Original: Then by the deﬁnition of ∆ S in (A.2)
Revised: Then by the deﬁnition of ∆ S in (A.2), we combine (C.1) and (C.2) to obtain

Original: [Equation]
Revised: [Equation]

Original: we combine (C.1) and (C.2) to obtain

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since we have
Revised: Since we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: , which is invertible by Assumption 3.3, by rearranging the terms in (C.3) we obtain (cid:98) β 
Revised: , which is invertible by Assumption 3.3, by rearranging the terms in (C.3) we obtain (cid:98) β − β

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof of Lemma A.2.
Revised: which concludes the proof of Lemma A.2.

Original: D A PPLICATION TO M ULTINOMIAL L OGISTIC R EGRESSION
Revised: D A PPLICATION TO M ULTINOMIAL L OGISTIC R EGRESSION

Original: In this section, we consider the setting where the logging distribution µ ( a | x ; β ) is parametrized by the multinomial logistic regression model.
Revised: In this section, we consider the setting where the logging distribution µ ( a | x ; β ) is parametrized by the multinomial logistic regression model.

Original: We assume that x ∈ R p and a ∈ [ m ] .
Revised: We assume that x ∈ R p and a ∈ [ m ] .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: β 1 , . ..
Revised: β 1 , . .

Original: [Equation]
Revised: [Equation]

Original: 1 ∈ R p
Revised: 1 ∈ R p

Original: [Equation]
Revised: [Equation]

Original: Thus, the dimensiond of the parameter β is  p ( m − 1) .
Revised: Thus, the dimension of the parameter β is d = p ( m − 1) .

Original: The logging distribution µ ( a | x ; β ) is parametrized by
Revised: The logging distribution µ ( a | x ; β ) is parametrized by

Original: [Equation]
Revised: [Equation]

Revised: We denote by λ max ( · ) and λ min ( · ) the largest and the smallest eigenvalue of a matrix, respectively.

Revised: Throughout this section, we assume that the following regularity condition holds.

Revised: Assumption D.1.

Revised: We assume that, for all i ∈ [ m − 1] , β i is bounded by (cid:107) β

Revised: i (cid:107) ≤ B .

Revised: Also, we assume that r ( a, x ) is bounded for all a and x .

Revised: Furthermore, recall that in (2.1) we have x ∼ ν , we assume that x is bounded by (cid:107) x (cid:107) ≤ X and for its covariance matrix Σ

Revised: =

Revised: E x ∼ ν

Revised: [ xx (cid:62) ] , we have

Original:  Then the score function S ( a, x ; β ) ∈
Revised: [Equation] Then the score function S ( a, x ; β ) ∈

Original: R p ( m − 1) takes the form
Revised: R p ( m − 1) takes the form

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: where 1 ( · ) is the indicator function, while the Fisher information matrix I ( β ) ∈
Revised: where 1 ( · ) is the indicator function, while the Fisher information matrix I ( β ) ∈

Original: R p ( m − 1) × p ( m − 1) is a block matrix whose ( j, k ) -th block, namely [ I ( β )] j,k , takes the form
Revised: R p ( m − 1) × p ( m − 1) is a block matrix whose ( j, k ) -th block, namely [ I ( β )] j,k , takes the form

Original: [Equation]
Revised: [Equation]

Original: for j, k ∈ [ m − 1] .
Revised: for j, k ∈ [ m − 1] .

Revised: We give lower and upper bounds for eigenvalues of I ( β ∗ ) in the following lemma.

Revised: Lemma D.2 (Non-singular Fisher Information Matrix) .

Original: Meanwhile, the deviation function D V (r, a, x ; β ) takes the form
Revised: Under Assumption D.1, the Fisher information matrix I ( β ) is non-singular. Furthermore, for the eigenvalues of I ( β ) , we have [Equation] Proof.

Revised: For the ease of notation

Revised: [Equation]

Revised: we let π i = µ ( i | x, β ) for i ∈ [ m − 1] , π m = 1 − (cid:80) m − 1 i =

Revised: π i and v = ( v 1 , . .

Revised: . , v m − 1 ) (cid:62) with v i ∈ R p .

Revised: First, for x and β i , i ∈ [ m − 1] satisfying Assumption D.1, we have

Revised: [Equation]

Revised: Then, by (D.4), we have

Revised: [Equation]

Revised: Note that (cid:80) mk =1 π k = 1 , we have

Revised: [Equation]

Revised: Thus

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: where the last two equalities are consequences of (D.6) and the Assumption D.1, respectively.

Revised: This is saying that the Fisher information matrix I ( β ) is positive deﬁnite.

Revised: For the largest eigenvalue of I ( β ) , we have

Revised: [Equation]

Revised: which, together with (D.9)

Revised: [Equation]

Revised: concludes the proof

Revised: [Equation]

Revised: Meanwhile

Revised: [Equation]

Revised: the deviation function D V ( r, a, x ; β ) takes the form

Original: [Equation]
Revised: [Equation]

Original: The gradient of the deviation function takes the form
Revised: The gradient of the deviation function takes the form

Original: [Equation]
Revised: [Equation]

Original:   .
Revised: 

Original: [Equation]
Revised: [Equation]

Original:  The Hessian of the deviation function is a block matrix in R p ( m − 1) × p ( m − 1) , whose ( j, k ) -th block takes the form 
Revised: D [Equation] [Equation] The Hessian of the deviation function is a block matrix in R p ( m − 1) × p ( m − 1) , whose ( j, k ) -th block takes the form (cid:20)

Original: [Equation]
Revised: [Equation]

Revised: D

Original:  for j, k ∈ [ m − 1] .
Revised: [Equation] for j, k ∈ [ m − 1] .

Original: Note that by (D.2)-(D.3) and (D.5) -(D.6) for the population quantities in Assumptions 3.3-3.4 are all bounded as long as x and β are bounded.
Revised: Note that by (D.3)-(D.4) and (D.11) -(D.12) for the population quantities, which do not scale with n , in Assumptions 3.3-3.4 are all bounded under Assumption D.1.

Original: Hence, in the multinomial logistic regression model deﬁned in (D.1) with bounded x and β  we have
Revised: Hence, in the multinomial logistic regression model deﬁned in (D.1) with bounded x and β as stated in Assumption D.1, when d is ﬁxed, we have

Original: [Equation]
Revised: [Equation]

Original:  where t is deﬁned in Condition 3.12 and M is some constant that does not scale with n .
Revised: 13) where t is deﬁned in Condition 3.12 and M is some constant that does not scale with n .

Original: The subsequent lemmas establish the tail behaviors for the events deﬁned in Deﬁnitions 3.5-3.7, which together yield a concrete form of the tail function f ( t ) deﬁned in Condition 3.8.
Revised: The subsequent lemmas establish the tail behaviors for the events deﬁned in Deﬁnitions 3.5-3.7, which together yield a concrete form of the tail function f ( t ) deﬁned in Condition 3.8.

Original: Lemma D.1 (Maximum Likelihood Estimation Error) .
Revised: Lemma D.3 (Maximum Likelihood Estimation Error) .

Original: For the event E β ( t β ) deﬁned in (3.8), we have
Revised: Under the Assumption D.1, for the event E β ( t β ) deﬁned in (3.8), we have

Original: [Equation]
Revised: [Equation]

Original: where C β is a positive constant.
Revised: where C β is a positive constant independent of n .

Revised: Proof.

Revised: Let (cid:98) (cid:96) n ( β ) be the negative log-likelihood function (cid:98) (cid:96) n ( β ) =

Revised: − (1 /n ) ·

Revised: (cid:80) ni

Revised: =1 log µ ( a i , x i ; β ) , where µ ( a, x ; β ) is deﬁned in (D.1).

Original:  Lemma D.2 (Concentration of Score) .
Revised: By Lemma D.2, we have λ min ≥ 2 e − 4 XB σ/m 2 , which gives P [Equation] min

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: where

Revised: [Table]

Revised: is

Revised: [Table]

Revised: , by matrix Bernstein inequality (see, for example, Tropp (2012);

Revised: Vershynin (2010)), we have

Revised: [Equation]

Revised: where the second inequality holds for a constant C (cid:48) β when n sufﬁciently large.

Revised: Combining (D.14) and (D.15), we know that (cid:98) (cid:96) n ( β ) is e − 4 XB σ/m 2 -strongly convex with at least probability 1 − C (cid:48)

Revised: β exp( d − nt 2 β ) .

Revised: The rest of the proof considers the case where the e − 4 XB σ/m 2 -strong convexity holds for (cid:98) (cid:96) n ( β ) , which means that we have

Revised: [Equation]

Revised: By deﬁnition of MLE, (cid:98) β is the minimizer for (cid:98) (cid:96) n ( β ) , which means (cid:98) (cid:96) n ( β ∗ ) ≥ (cid:98) (cid:96) n ( (cid:98) β ) .

Revised: Therefore, by

Revised: Cauchy-Schwarz inequality, we have

Revised: [Equation]

Revised: , which gives

Revised: [Equation]

Revised: Note that E [ S ( a, x ; β ∗ )]

Revised: = 0 , E [ S ( a, x ; β ∗ ) S ( a, x ; β ∗ ) (cid:62) ]

Revised: = I ( β ∗ ) and ∇ β (cid:98) (cid:96) n ( β ∗ ) =

Revised: (1 /n ) · (cid:80) ni =1 S ( a i , x i ; β ∗ ) , by Bernstein’s inequality (see, for example, Tropp (2012)), we have

Revised: [Equation]

Revised: for any w such that (cid:107) w (cid:107) = 1 , where the second inequality holds for a constant C (cid:48)(cid:48) β when n is large enough.

Revised: Let G d = { w ∈ R d : (cid:107) w (cid:107) = 1 } , by taking a union bound over w ∈ G d in (D.18), we have

Revised: P

Revised: [Equation]

Revised: [Equation]

Revised: Plugging (D.19) into (D.17) and recall that (D.16) holds with at least a probability of 1 − C (cid:48)

Revised: β exp( d − nt 2 β ) , we have

Revised: [Equation]

Original: For the logistic regression model in (D.1), with the event E SG ( t SG ) , E SH ( t SH ) and E ST ( t ST ) deﬁned in Deﬁnition 3.6, we have
Revised: which concludes the proof with C β = C (cid:48) β [Equation]

Original: P (cid:0) E SG ( t SG )
Revised: C (cid:48)(cid:48)

Original: [Equation]
Revised: [Equation]

Original: SG
Revised: Next, we state two Lemmas on the concentration of score and deviation.

Original: [Equation] exp [Equation] nt
Revised: The proofs for them are quite standard in matrix concentration theories.

Original: [Equation]
Revised: Lemma D.4 (Concentration of Score) .

Original:  SG ) , 
Revised: Under the Assumption D.1, for the logistic regression model in (D.1), with the event E SG ( t SG ) , E SH ( t SH ) and E ST ( t ST ) deﬁned in Deﬁnition 3.6, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where C SG , C SH , and C ST are positive constants.
Revised: where C SG , C SH , and C ST are positive constants independent of n .

Revised: Proof.

Revised: Following the arguments made in (D.15) and (D.19), the exponentially decaying tails are immediate.

Original: Lemma D.3 (Concentration of Deviation) .
Revised: Lemma D.5 (Concentration of Deviation) .

Revised: Suppose the Assumption D.1 holds and r ( a, x ) ≤ R for all a and x .

Original: For the logistic regression model (D.1), with the event E DG ( t DG ) and E DH ( t DH ) deﬁned in 3.7, we have
Revised: Then, for the logistic regression model (D.1), with the event E DG ( t DG ) and E DH ( t DH ) deﬁned in 3.7, we have

Original: [Equation]
Revised: [Equation]

Original: where C DG and C DH are positive constants.
Revised: where C DG and C DH are positive constants independent of n .

Original:  Recall  the event E ( t (cid:63) ) with t (cid:63)
Revised: Proof. Recall that we have π i ≥ e − 2 XB /m when the Assumption D.1 holds. We have [Equation]

Revised: ∂β

Revised: [Equation]

Revised: and, by similar arguments in the proof of Lemma D.2, we have

Revised: [Equation]

Revised: Then, the ﬁrst inequality in (D.20) follows by similar arguments as in (D.17)-(D.18).

Revised: Next, again by similar arguments in the proof of Lemma D.2, we also have

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: Then, similar as (D.15), the second inequality in (D.20) follows by matrix Bernstein inequality.

Revised: Recall the event E ( t (cid:63) ) with t (cid:63)

Original: = (cid:107) t (cid:107) ∞ is deﬁned in Condition 3.8.
Revised: = (cid:107) t (cid:107) ∞ is deﬁned in Condition 3.8.

Original: By Lemmas D.1-D.3, the tail function f ( t ) takes the form
Revised: By Lemmas D.3-D.5, the tail function f ( t ) takes the form

Original: [Equation]
Revised: [Equation]

Original: 8) where C = max { C β , C SG , C SH , C DG , C DH , C DT } > 0 .
Revised: D [Equation] 21) where C = max { C β , C SG , C SH , C DG , C DH , C DT } > 0 .

Original: In the sequel, we use (D.7) and the tail function in (D.8) to characterize the reduction of the mean squared error induced by the multinomial logistic regression model.
Revised: In the sequel, we use (D.13) and the tail function in (D.21) to characterize the reduction of the mean squared error induced by the multinomial logistic regression model.

Original: First, let t 

Revised: First, let t

Original: = ( T + 1) (cid:112) d/n with some constant T > 0 , we have the following bound on the tail probability of t based on (D.8),
Revised: = ( T + 1) (cid:112) d/n with some constant T > 0 , we have the following bound on the tail probability of t based on (D.21),

Original: [Equation]
Revised: [Equation]

Original: Then recall that in Theorem 3.9 we have
Revised: Then recall that in Theorem 3.9 we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where by (D.9) [Equation] we have
Revised: where by (D.22), we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: which decays exponentially with d and is therefore dominated by the bound given in (D.10).
Revised: which decays exponentially with d and is therefore dominated by the bound given in (D.23).

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: Using the same arguments in derivation of (D.12) [Equation] we also have
Revised: Using the same arguments in derivation of (D.25), we also have

Original: [Equation]
Revised: [Equation]

Original: Hence [Equation] we obtain
Revised: Hence, we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that the mean squared error term E [( (cid:101) V − V ) 2 ] is of the order 1 /n , which implies that ξ ( n ) is of the order ( d/n ) 3 / 2 .
Revised: Note that the mean squared error term E [( (cid:101) V − V ) 2 ] is of the order 1 /n , which implies that ξ ( n ) is of the order ( d/n ) 3 / 2 .

Original: Thus, (3.15) is satisﬁed when n is sufﬁciently large, which means that a reduction of the order 1 /n on the mean squared error is achieved by the multinomial logistic regression model when the sample size n is sufﬁciently large.
Revised: Thus, (3.15) is satisﬁed when n is sufﬁciently large, which means that a reduction of the order 1 /n on the mean squared error is achieved by the multinomial logistic regression model when the sample size n is sufﬁciently large.

Original: This result implies that the MLIPS in (3.1) is guaranteed to out perform the IPS estimator in (2.2) when the sample size n is much larger than the dimension d of the problem.
Revised: This result implies that the MLIPS in (3.1) is guaranteed to out perform the IPS estimator in (2.2) when the sample size n is much larger than the dimension d of the problem.

Original: Moreover, since the mean squared error term E [( (cid:101) V − V ) 2 ] itself is of the order 1 /n , the effect of reduction does not vanish when n → ∞ .
Revised: Moreover, since the mean squared error term E [( (cid:101) V − V ) 2 ] itself is of the order 1 /n , the effect of reduction does not vanish when n → ∞ .

Original: We interpret Theorem 3.9 in a more general way.
Revised: We interpret Theorem 3.9 in a more general way.

Original: The ξ ( n ) term is in general a decreasing function of the sample size n and is mostly of a higher order than 1 /n in common parametric statistical models.
Revised: The ξ ( n ) term is in general a decreasing function of the sample size n and is mostly of a higher order than 1 /n in common parametric statistical models.

Original: Correspondingly, for a sufﬁciently large n , we obtain (3.15), which implies that our method leads to a 1 / (2 n ) · Var(Π( r, x, a ; β ∗ )) reduction on the mean squared error, which has a non-vanishing effect when n → ∞ .  
Revised: Correspondingly, for a sufﬁciently large n , we obtain (3.15), which implies that our method leads to a 1 / (2 n ) · Var(Π( r, x, a ; β ∗ )) reduction on the mean squared error, which has a non-vanishing effect when n → ∞ . E C OMPLIMENTARY R ESULTS FOR S ECTION 4.

Revised: We provide the following extra results in Table 2 for the experiment in Section 4.3.

Revised: The additional results of standard deviation of the Hamming losses of the policies optimized by using IPS, POEM and their ML-variants are computed over ten runs.

Revised: From the Table 2, we can see that our ML-based approach induces much smaller standard deviations than the improvements made on performances and, thus, it consistently boosts the performance of IPS and POEM in policy optimization.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: 05e

Revised: [Table]

Revised: [Table]

Revised: 31e-2)

Revised: [Table]

Revised: Lin

Revised: [Table]

Revised: [Table]

Revised: (9.98e-2) 2.

Revised: (6.23e-3) 1.

Revised: (5.92e-2)

Revised: MLIPS

Revised: [Table]

Revised: NN 1.

Revised: (4.88e-6) 3.

Revised: (9.10e-2) 2.

Revised: (1.22e-3) 0.

Revised: (2.08e-2)

Revised: MLPOEM

Revised: [Table]

Revised: Lin 1.

Revised: (3.48e-6) 3.

Revised: (3.16e-1) 2.

Revised: (7.96e-3) 0.

Revised: (4.21e-2)

Revised: MLPOEM

Revised: [Table]

Revised: NN 1.

Revised: (5.81e-6) 3.

Revised: (2.03e-1) 2.

Revised: (1.22e-2) 0.

Revised: (2.46e-2)