Original: Incrementality Bidding via Reinforcement Learning under Mixed and Delayed Rewards
Revised: Incrementality Bidding via Reinforcement Learning under Mixed and Delayed Rewards

Original: Abstract
Revised: Abstract

Original: Incrementality, which measures the causal effect of showing an ad to a potential customer (e.g. a user in an internet platform) versus not, is a central object for advertisers in online advertising platforms.
Revised: Incrementality, which measures the causal effect of showing an ad to a potential customer (e.g. a user in an internet platform) versus not, is a central object for advertisers in online advertising platforms.

Original: This paper investigates the problem of how an advertiser can learn to optimize the bidding sequence in an online manner without knowing the incrementality parameters in advance.
Revised: This paper investigates the problem of how an advertiser can learn to optimize the bidding sequence in an online manner without knowing the incrementality parameters in advance.

Original: We formulate the ofﬂine version of this problem as a specially structured episodic Markov Decision Process (MDP) and then, for its online learning counterpart, propose a novel reinforcement learning (RL) algorithm with regret at most (cid:101)
Revised: We formulate the ofﬂine version of this problem as a specially structured episodic Markov Decision Process (MDP) and then, for its online learning counterpart, propose a novel reinforcement learning (RL) algorithm with regret at most (cid:101)

Original: O
Revised: O

Original: ( H 2 √ T ) , which depends on the number of rounds H and number of episodes T , but does not depend on the number of actions (i.e., possible bids).
Revised: ( H 2 √ T ) , which depends on the number of rounds H and number of episodes T , but does not depend on the number of actions (i.e., possible bids).

Original: A fundamental difference between our learning problem from standard RL problems is that the realized reward feedback from conversion incrementality is mixed and delayed .
Revised: A fundamental difference between our learning problem from standard RL problems is that the realized reward feedback from conversion incrementality is mixed and delayed .

Original: To handle this difﬁculty we propose and analyze a novel pairwise moment-matching algorithm to learn the conversion incrementality, which we believe is of independent interest.
Revised: To handle this difﬁculty we propose and analyze a novel pairwise moment-matching algorithm to learn the conversion incrementality, which we believe is of independent interest.

Original:  Introduction
Revised:  Introduction

Original: Nowadays, online advertising systems (e.g. Google, Facebook, and Amazon) have demonstrated their signiﬁcant power for connecting advertisers and potential customers.
Revised: Nowadays, online advertising systems (e.g. Google, Facebook, and Amazon) have demonstrated their signiﬁcant power for connecting advertisers and potential customers.

Original: Moreover, automated auctions are widely used in online ad platforms for matching advertisers and users, and for price discovery [23, 8].
Revised: Moreover, automated auctions are widely used in online ad platforms for matching advertisers and users, and for price discovery [23, 8].

Original: Many advertisers are bidding repeatedly online to show ads to users.
Revised: Many advertisers are bidding repeatedly online to show ads to users.

Original: Therefore, an important problem for advertisers is to design efﬁcient bidding algorithms to maximize their accumulated utility.
Revised: Therefore, an important problem for advertisers is to design efﬁcient bidding algorithms to maximize their accumulated utility.

Original: Moreover, this is also an important problem for online ad platforms, because auto-bidding — i.e., the advertisers authorize ad platforms to bid on behalf of them —has become prevalent in online advertising [9, 11].
Revised: Moreover, this is also an important problem for online ad platforms, because auto-bidding — i.e., the advertisers authorize ad platforms to bid on behalf of them —has become prevalent in online advertising [9, 11].

Original: A growing body of work is investigating the problem of learning to bid in repeated auctions [25, 10, 5, 13, 12, 4, 20, 19].
Revised: A growing body of work is investigating the problem of learning to bid in repeated auctions [25, 10, 5, 13, 12, 4, 20, 19].

Original: However, existing work assumes that the value of showing ads to a user (after winning the auction) comes from either an oblivious adversary or is sampled from an underlying ﬁxed distribution.
Revised: However, existing work assumes that the value of showing ads to a user (after winning the auction) comes from either an oblivious adversary or is sampled from an underlying ﬁxed distribution.

Original: Unfortunately, this ignored the crucial causal effect of showing an ad to a potential customer and cannot capture how much an ad will change the user’s conversion rate, which should have been what the advertiser is bidding for.
Revised: Unfortunately, this ignored the crucial causal effect of showing an ad to a potential customer and cannot capture how much an ad will change the user’s conversion rate, which should have been what the advertiser is bidding for.

Original: For example, if a user sees the same ad multiple times in a very short time period, then the conversion 1 of this user may not be much better — in fact, may even be worse — than that of showing ads to this user only once within this period.
Revised: For example, if a user sees the same ad multiple times in a very short time period, then the conversion 1 of this user may not be much better — in fact, may even be worse — than that of showing ads to this user only once within this period.

Original: Such multiple redundant ad placements are commonly known in reality to be cost-inefﬁcient for the advertisers, but cannot be captured by previous models of learning to bid with either random or completely adversarial ad values.
Revised: Such multiple redundant ad placements are commonly known in reality to be cost-inefﬁcient for the advertisers, but cannot be captured by previous models of learning to bid with either random or completely adversarial ad values.

Original: To overcome the drawbacks of previous approaches in measuring the value of an advertising opportunity, a recent visionary whitepaper by Lewis and Wong [18] introduced the important notion of incrementality , which properly quantiﬁes this causal effect of showing an ad to the user as validated by massive real data from leading advertising platforms.
Revised: To overcome the drawbacks of previous approaches in measuring the value of an advertising opportunity, a recent visionary whitepaper by Lewis and Wong [18] introduced the important notion of incrementality , which properly quantiﬁes this causal effect of showing an ad to the user as validated by massive real data from leading advertising platforms.

Original: However, [18] focuses on the ofﬂine learning problem of estimating incrementality from past data.
Revised: However, [18] focuses on the ofﬂine learning problem of estimating incrementality from past data.

Original: The online learning of incrementality parameters, as well as how to utilize it to optimize bidding sequences, has not been studied; this is what we embark on in this paper.
Revised: The online learning of incrementality parameters, as well as how to utilize it to optimize bidding sequences, has not been studied; this is what we embark on in this paper.

Original: In Figure 1, we visualize the conversion incrementality caused by three ad placements and used the colored areas to capture the incrementality caused by each ad placement.
Revised: In Figure 1, we visualize the conversion incrementality caused by three ad placements and used the colored areas to capture the incrementality caused by each ad placement.

Original: There are two main challenges in learning incrementality, i.e., delayed and mixed conversion feedback.
Revised: There are two main challenges in learning incrementality, i.e., delayed and mixed conversion feedback.

Original: First, the conversion may not happen immediately when the ad is shown to the user, even though the conversion rate may quickly peak after showing the ad (see Ad 2 and Ad 3 in Figure 1).
Revised: First, the conversion may not happen immediately when the ad is shown to the user, even though the conversation

Original: [Figure]
Revised: [Figure]

Revised: rate may quickly peak after showing the ad (see Ad 2 and

Revised: Ad 3 in Figure 1).

Original: Such delayed conversion feedback in online advertising is studied only recently in ofﬂine setups, e.g., [6, 21, 22, 3].
Revised: Such delayed conversion feedback in online advertising is studied only recently in ofﬂine setups, e.g., [6, 21, 22, 3].

Original: Second, when a conversion happens, it is a mixed effect from multiple previous ad placements before the conversion.
Revised: Second, when a conversion happens, it is a mixed effect from multiple previous ad placements before the conversion.

Original: In this work, we formulate incrementality bidding as an episodic Markov Decision Process (MDP), where each episode represents an interaction with a single user drawn from a population of the same characteristics (and thus assumed to have the same incrementality parameters).
Revised: In this work, we formulate incrementality bidding as an episodic Markov Decision Process (MDP), where each episode represents an interaction with a single user drawn from a population of the same characteristics (and thus assumed to have the same incrementality parameters).

Original: The advertiser wants to learn the incrementality and optimize bidding for this user population.
Revised: The advertiser wants to learn the incrementality and optimize bidding for this user population.

Original: This is hardly possible in general given the two difﬁculties above as well as the potentially complex dependence of incrementality on the entire history.
Revised: This is hardly possible in general given the two difﬁculties above as well as the potentially complex dependence of incrementality on the entire history.

Original: We thus make a Markovian assumption that the conversion incrementality of an ad placement at each round h only depends on its last ad impression shown before this round.
Revised: We thus make a Markovian assumption that the conversion incrementality of an ad placement at each round h only depends on its last ad impression shown before this round.

Original: In addition, we introduce the heterogeneous Poisson process to formally capture the mixed and delayed rewards feedback in the conversion process (Poisson process is also mentioned by Lewis and Wong [18], but only qualitatively).
Revised: In addition, we introduce the heterogeneous Poisson process to formally capture the mixed and delayed rewards feedback in the conversion process (Poisson process is also mentioned by Lewis and Wong [18], but only qualitatively).

Original: To optimize the sequential  biddingstrategies inthe online setup, there is an intrinsic exploration versus exploitation tradeoff in incrementality bidding.
Revised: To optimize the sequential strategy for incrementality bidding in online learning, there is an intrinsic exploration versus exploitation tradeoff.

Original: On one hand, the advertiser needs to bid high in order to win the ad placement opportunity and also to learn the conversion incrementality.
Revised: On one hand, the advertiser needs to bid high in order to win the ad placement opportunity and also to learn the conversion incrementality.

Original: On the other hand, she also doesn’t want to bid too high such that her conversion reward may be less than her cost, i.e., the payment due to winning the auction.
Revised: On the other hand, she also doesn’t want to bid too high such that her conversion reward may be less than her cost, i.e., the payment due to winning the auction.

Original: This observation naturally motivates our design of a model-based RL algorithm that adopts a UCB-style approach to balance exploration and exploitation.
Revised: This observation naturally motivates our design of a model-based RL algorithm that adopts a UCB-style approach to balance exploration and exploitation.

Original: Our Results and Contributions.
Revised: Our Results and Contributions.

Original: The main contributions of this paper are two folds: (1) we introduce an episodic MDP, coupled with a heterogeneous Poisson process, to capture the incrementality bidding problem; (2) we design a novel reinforcement learning algorithm for the advertiser to optimize her sequential bidding while learning incrementality simultaneously.
Revised: The main contributions of this paper are two folds: (1) we introduce an episodic MDP, coupled with a heterogeneous Poisson process, to capture the incrementality bidding problem; (2) we design a novel reinforcement learning algorithm for the advertiser to optimize her sequential bidding while learning incrementality simultaneously.

Original: Technically , our main result is the design of an RL algorithm for incrementality bidding that provably has regret at most (cid:101) O
Revised: Technically , our main result is the design of an RL algorithm for incrementality bidding that provably has regret at most (cid:101) O

Original: ( H 2 √ T ) and is near-optimal in terms of the dependence on T .
Revised: ( H 2 √ T ) and is near-optimal in terms of the dependence on T .

Original: Conceptually , our result demonstrates the possibility of designing highly-efﬁcient RL algorithms for incrementality bidding despite its various challenges.
Revised: Conceptually , our result demonstrates the possibility of designing highly-efﬁcient RL algorithms for incrementality bidding despite its various challenges.

Original: To the best of our knowledge, this is the ﬁrst work that analyzes incrementality bidding in an online setting theoretically.
Revised: To the best of our knowledge, this is the ﬁrst work that analyzes incrementality bidding in an online setting theoretically.

Original: Notably, our regret bound is independent of the number of possible bids (equiv.
Revised: Notably, our regret bound is independent of the number of possible bids (equiv.

Original: actions in standard
Revised: actions in standard

Original: RL).
Revised: RL).

Original: This is due to the special structures of the incrementality bidding problem, which has a small effective action space.
Revised: This is due to the special structures of the incrementality bidding problem, which has a small effective action space.

Original: Speciﬁcally, there are only two effective outcomes for an advertiser—winning or losing— that eventually affect the state transitions.
Revised: Speciﬁcally, there are only two effective outcomes for an advertiser—winning or losing— that eventually affect the state transitions.

Original: The conversion incrementality only happens if the learner wins.
Revised: The conversion incrementality only happens if the learner wins.

Original: Finally, we remark that our results are independent of auction formats and all our results hold for second price auctions, ﬁrst price auctions, and other standard single-item auction formats.
Revised: Finally, we remark that our results are independent of auction formats and all our results hold for second price auctions, ﬁrst price auctions, and other standard single-item auction formats.

Original: To achieve this regret bound, we provide a novel regret decomposition to incorporate with the convergence rate analysis of our novel parameter estimation method.
Revised: To achieve this regret bound, we provide a novel regret decomposition to incorporate with the convergence rate analysis of our novel parameter estimation method.

Original: One key technical novelty of this paper is a new parameter estimation method, Pairwise MomentMatching ( PAMM ) algorithm, that can estimate reward parameters under mixed and delayed conversion feedback.
Revised: One key technical novelty of this paper is a new parameter estimation method, Pairwise MomentMatching ( PAMM ) algorithm, that can estimate reward parameters under mixed and delayed conversion feedback.

Original: In this type of models with mixture effects, standard likelihood-based model ﬁtting procedure would lead to solving non-convex optimization problems resulting in difﬁculties for computation and analysis, as well as unpleasant gap between theory and algorithm [15].
Revised: In this type of models with mixture effects, standard likelihood-based model ﬁtting procedure would lead to solving non-convex optimization problems resulting in difﬁculties for computation and analysis, as well as unpleasant gap between theory and algorithm [15].

Original: However,
Revised: However,

Original: PAMM estimates incrementality parameters in an online and computationally efﬁcient manner without the need of solving nonconvex optimization problems.
Revised: PAMM estimates incrementality parameters in an online and computationally efﬁcient manner without the need of solving nonconvex optimization problems.

Original: The estimators are provably consistent with a nearly-optimal convergence rate.
Revised: The estimators are provably consistent with a nearly-optimal convergence rate.

Original: We believe this novel technique will be of independent interest for other online learning problems with delayed and mixed reward feedback.
Revised: We believe this novel technique will be of independent interest for other online learning problems with delayed and mixed reward feedback.

Original: Related Work.
Revised: Related Work.

Original: As mentioned in the introduction, our work is highly inspired by the visionary work of Lewis and Wong [18], who ﬁrstly proposed incrementality bidding (and attribution).
Revised: As mentioned in the introduction, our work is highly inspired by the visionary work of Lewis and Wong [18], who ﬁrstly proposed incrementality bidding (and attribution).

Original: However, the method proposed in [18] lacks a theoretical guarantee and cannot be adapted to online learning settings.
Revised: However, the method proposed in [18] lacks a theoretical guarantee and cannot be adapted to online learning settings.

Original: Our work is generally related to the learning to bid literature [25, 10, 5, 13, 12, 4].
Revised: Our work is generally related to the learning to bid literature [25, 10, 5, 13, 12, 4].

Original: However, all these existing papers usually model this problem as (contextual) bandits and don’t consider the causal effect of showing an ad to the user captured by incrementality (i.e., the additional value or conversion given the ad placements in previous rounds), which causes the main difference from this paper.
Revised: However, all these existing papers usually model this problem as (contextual) bandits and don’t consider the causal effect of showing an ad to the user captured by incrementality (i.e., the additional value or conversion given the ad placements in previous rounds), which causes the main difference from this paper.

Original: Speciﬁcally, in all these previous works [25, 10, 5, 13, 12, 4], the reward for an ad impression is realized immediately at the current round whereas, in our model, any ad impression only increases the rate of conversions, which however may be realized in any later time following a Poisson process and, more importantly, is mixed with the incrementality of other ad impressions.
Revised: Speciﬁcally, in all these previous works [25, 10, 5, 13, 12, 4], the reward for an ad impression is realized immediately at the current round whereas, in our model, any ad impression only increases the rate of conversions, which however may be realized in any later time following a Poisson process and, more importantly, is mixed with the incrementality of other ad impressions.

Original: We believe our model captures the effect of advertising more realistically.
Revised: We believe our model captures the effect of advertising more realistically.

Original: Some other less related works include delayed feedback in other different RL problems, which have only been studied recently [17, 14].
Revised: Some other less related works include delayed feedback in other different RL problems, which have only been studied recently [17, 14].

Original: We handle delayed conversion feedback through a parametric inhomogeneous Poisson process, which is different from these works.
Revised: We handle delayed conversion feedback through a parametric inhomogeneous Poisson process, which is different from these works.

Original:  The Sequential Optimization Problem of Incrementality Bidding ( IB )
Revised:  The Sequential Optimization Problem of Incrementality Bidding ( IB )

Original: This paper adopts the perspective of an online advertiser, and studies the advertiser’s learning problem of optimizing her sequential bidding policy.
Revised: This paper adopts the perspective of an online advertiser, and studies the advertiser’s learning problem of optimizing her sequential bidding policy.

Original: Next, we will ﬁrst describe the background and motivation of this problem and then introduce the formal model.
Revised: Next, we will ﬁrst describe the background and motivation of this problem and then introduce the formal model.

Original: We consider the sequential bidding problem for a single advertiser — i.e., the decision maker or learner — in online advertising systems.
Revised: We consider the sequential bidding problem for a single advertiser — i.e., the decision maker or learner — in online advertising systems.

Original: The advertiser participates online advertising auctions in order to win opportunities for showing ads to Internet users and, ultimately, gain conversions (i.e., purchases of products or services) from Internet users.
Revised: The advertiser participates online advertising auctions in order to win opportunities for showing ads to Internet users and, ultimately, gain conversions (i.e., purchases of products or services) from Internet users.

Original: The interaction between the advertiser and Internet users happens in an episodic manner; each episode corresponds to one Internet user.
Revised: The interaction between the advertiser and Internet users happens in an episodic manner; each episode corresponds to one Internet user.

Original: Speciﬁcally, each episode has H rounds, during which the advertiser (learner) will interact with the user of that episode for at most H times.
Revised: Speciﬁcally, each episode has H rounds, during which the advertiser (learner) will interact with the user of that episode for at most H times.

Original: Notably, H is a ﬁnite number since (1) an interaction happens only when a user visits a particular webpage and (2) if an advertiser already bids for a user for sufﬁciently many times and still was not able to get conversion, the advertiser typically will cease to advertise to this user (at least for some time).
Revised: Notably, H is a ﬁnite number since (1) an interaction happens only when a user visits a particular webpage and (2) if an advertiser already bids for a user for sufﬁciently many times and still was not able to get conversion, the advertiser typically will cease to advertise to this user (at least for some time).

Original: We use T to denote the total number of episodes.
Revised: We use T to denote the total number of episodes.

Original: To focus on the fundamentals of the problem, this work studies the basic setup with T i.i.d.
Revised: To focus on the fundamentals of the problem, this work studies the basic setup with T i.i.d.

Original: episodes (i.e., Internet users) that occur sequentially and evenly distributed H rounds at integer time point 1 , 2 , · · · , H within each episode.
Revised: episodes (i.e., Internet users) that occur sequentially and evenly distributed H rounds at integer time point 1 , 2 , · · · , H within each episode.

Original: Interesting future directions include examining the more general settings in which each Internet user in the corresponding episode may depend on a context feature, the time points of the H rounds are different for different episodes, and the episodes may happen in batch (i.e., multiple similar users arrive at the same time).
Revised: Interesting future directions include examining the more general settings in which each Internet user in the corresponding episode may depend on a context feature, the time points of the H rounds are different for different episodes, and the episodes may happen in batch (i.e., multiple similar users arrive at the same time).

Original: Conversion incrementality.
Revised: Conversion incrementality.

Original: Central to our bidding optimization problem is the notion of “conversion incrementality” [18].
Revised: Central to our bidding optimization problem is the notion of “conversion incrementality” [18].

Original: Intuitively, incrementality captures how much increase the advertiser’s ads can boost the user’s conversion rates.
Revised: Intuitively, incrementality captures how much increase the advertiser’s ads can boost the user’s conversion rates.

Original: Notably, the reward of showing an ad is the incrementality while not the conversion rate itself.
Revised: Notably, the reward of showing an ad is the incrementality while not the conversion rate itself.

Original: This is very different from the previous learning to bid literature since they assume the conversion (or value) comes independently with the previous ad placements (i.e., from oblivious adversary or an underlying distribution).
Revised: This is very different from the previous learning to bid literature since they assume the conversion (or value) comes independently with the previous ad placements (i.e., from oblivious adversary or an underlying distribution).

Original: This is why previous studies almost all use the (contextual) bandits setup, while not MDP.
Revised: This is why previous studies almost all use the (contextual) bandits setup, while not MDP.

Original: We now formally describe the model of incrementality bidding, following [18].
Revised: We now formally describe the model of incrementality bidding, following [18].

Original: Within any episode t ∈ [ T ] , each round h ∈
Revised: Within any episode t ∈ [ T ] , each round h ∈

Original: [ H ] has an incrementality rate function d h ( τ ; A h , θ h ) for any continuous time τ ∈ [ h, ∞ ) which is the “density” of the conservation incrementality at any time τ ∈
Revised: [ H ] has an incrementality rate function d h ( τ ; A h , θ h ) for any continuous time τ ∈ [ h, ∞ ) which is the “density” of the conservation incrementality at any time τ ∈

Original: [ h, ∞ ) .
Revised: [ h, ∞ ) .

Original: Intuitively, d h ( τ ; A h , θ h ) models how much an ad shown at round h boosts the conversion rate density at any future time τ > h .
Revised: Intuitively, d h ( τ ; A h , θ h ) models how much an ad shown at round h boosts the conversion rate density at any future time τ > h .

Original: Notably, the integral of this rate function d h ( τ ; A h , θ h ) does not need to be 1 as there may be many conversions per user impression.
Revised: Notably, the integral of this rate function d h ( τ ; A h , θ h ) does not need to be 1 as there may be many conversions per user impression.

Original: We assume that d h ( τ ; A h , θ h ) depends on an to-be-learnt unknown parameter θ h and
Revised: We assume that d h ( τ ; A h , θ h ) depends on an to-be-learnt unknown parameter θ h and

Original: additionally a subset
Revised: additionally a subset

Original: A h ⊂ [ h ] containing all previous rounds, at which our learner’s ads were shown to the corresponding user of this episode.
Revised: A h ⊂ [ h ] containing all previous rounds, at which our learner’s ads were shown to the corresponding user of this episode.

Original: Let θ =
Revised: Let θ =

Original: ( θ 1 , θ 2 , · · · , θ H ) denote all the to-be-learnt parameters.
Revised: ( θ 1 , θ 2 , · · · , θ H ) denote all the to-be-learnt parameters.

Original: It is generally intractable to learn an arbitrary rate function d h ( τ ; A h , θ h ) .
Revised: It is generally intractable to learn an arbitrary rate function d h ( τ ; A h , θ h ) .

Original: Next, we introduce a natural parameterized family of functions for d h ( τ ; A h , θ h ) , with to-be-learnt parameters, in order to capture advertising applications.
Revised: Next, we introduce a natural parameterized family of functions for d h ( τ ; A h , θ h ) , with to-be-learnt parameters, in order to capture advertising applications.

Original: First, we have the following assumption for d h ,
Revised: First, we have the following assumption for d h ,

Original: Assumption 1 (Markovian Incrementality) .
Revised: Assumption 1 (Markovian Incrementality) .

Original: d h is only affected by the learner’s last ad impression before h , rather than the entire history
Revised: d h is only affected by the learner’s last ad impression before h , rather than the entire history

Original: A h of the learner’s advertising performance.
Revised: A h of the learner’s advertising performance.

Original: Formally, at current round h , suppose the last round at which the learner’s ad was shown is h − l , then the incrementality function d h ( τ ; A h , θ h ) =
Revised: Formally, at current round h , suppose the last round at which the learner’s ad was shown is h − l , then the incrementality function d h ( τ ; A h , θ h ) =

Original: d h ( τ ; (cid:96), θh ) .
Revised: d h ( τ ; (cid:96), θ

Revised: [Equation]

Original: The above assumption is built upon the sense that the users normally weighs more on the recently visited ads and are generally memory-less.
Revised: The above assumption is built upon the sense that the users normally weighs more on the recently visited ads and are generally memory-less.

Original: Moreover, we assume each incrementality rate function d h ( τ ; (cid:96), θ h ) for any h ∈
Revised: Moreover, we assume each incrementality rate function d h ( τ ; (cid:96), θ h ) for any h ∈

Original: [ H ] has the following parametric form:
Revised: [ H ] has the following parametric form:

Original: [Equation]
Revised: [Equation]

Original: in which θ h = { β h ( (cid:96) ) } h − 1 (cid:96) =1 ∪ { λ h } contain all the parameters at round h of the given IB instance.
Revised: in which θ h = { β h ( (cid:96) ) } h − 1 (cid:96) =1 ∪ { λ h } contain all the parameters at round h of the given IB instance.

Original: In other words, the incrementality rate is a re-scaled exponential distribution and the re-scaling is due to fact that each user impression may lead to more (or less) than one user conversions.
Revised: In other words, the incrementality rate is a re-scaled exponential distribution and the re-scaling is due to fact that each user impression may lead to more (or less) than one user conversions.

Original: In our
Revised: In our

Original: RL formulation of the problem, these are all the to-be-learnt parameters.
Revised: RL formulation of the problem, these are all the to-be-learnt parameters.

Original: The Markovian property is reﬂected in β h ( (cid:96) ) .
Revised: The Markovian property is reﬂected in β h ( (cid:96) ) .

Original: Note that (cid:82) ∞ τ = h d h ( τ ; (cid:96), θ h ) dτ
Revised: Note that (cid:82) ∞ τ = h d h ( τ ; (cid:96), θ h ) dτ

Original: = β h ( (cid:96) ) .
Revised: = β h ( (cid:96) ) .

Original: So β h ( (cid:96) ) can be viewed as the expected number of conversions triggered by the ad impression at round h , given the last time for showing the learner’s ad is (cid:96) rounds before.
Revised: So β h ( (cid:96) ) can be viewed as the expected number of conversions triggered by the ad impression at round h , given the last time for showing the learner’s ad is (cid:96) rounds before.

Original: In this paper, we assume ∀ h, (cid:96), c β ≤ β h ( (cid:96) )
Revised: In this paper, we assume ∀ h, (cid:96), c β ≤ β h ( (cid:96) )

Original: ≤ C β for some positive constants c β and C β .
Revised: ≤ C β for some positive constants c β and C β .

Original: Without loss of generality, we assume C β ≤ 1 .
Revised: Without loss of generality, we assume C β ≤ 1 .

Original: Remark 1.
Revised: Remark 1.

Original: The assumption of exponential decaying rate in Equation (1) is not necessary for our approach to work, and is primarily for the exposition.
Revised: The assumption of exponential decaying rate in Equation (1) is not necessary for our approach to work, and is primarily for the exposition.

Original: Our techniques can be applied to most parametric family of rate functions such as truncated exponential, logistic, Beta, and Gamma distributions [18], as long as the CDF function is invertible.
Revised: Our techniques can be applied to most parametric family of rate functions such as truncated exponential, logistic, Beta, and Gamma distributions [18], as long as the CDF function is invertible.

Original: From conversion incrementality to realized conversions.
Revised: From conversion incrementality to realized conversions.

Original: While incrementality captures how the rate of conversions increases, it did not model how conversions are realized in reality.
Revised: While incrementality captures how the rate of conversions increases, it did not model how conversions are realized in reality.

Original: This would be important when we introduce our reinforcement learning problem — after all, in real applications, the only observations our learner (an advertiser) can see are the realized conversions at different time, rather than the continuous conversion rate function.
Revised: This would be important when we introduce our reinforcement learning problem — after all, in real applications, the only observations our learner (an advertiser) can see are the realized conversions at different time, rather than the continuous conversion rate function.

Original: To capture this, we adopt the standard assumption that conversions are realized based on a continuous-time inhomogeneous Poisson process [18].
Revised: To capture this, we adopt the standard assumption that conversions are realized based on a continuous-time inhomogeneous Poisson process [18].

Original: More speciﬁcally, let the ordered subset W = { w 1 , w 2 , · · · , w n } ⊆
Revised: More speciﬁcally, let the ordered subset W = { w 1 , w 2 , · · · , w n } ⊆

Original: [ H ] denote the rounds at which the learning agent wins within the episode (for convenience, let w 0 = 0 ).
Revised: [ H ] denote the rounds at which the learning agent wins within the episode (for convenience, let w 0 = 0 ).

Original: Then the conversions arrive according to a Poisson process with a time-varying rate r ( τ ) , deﬁned as follows 
Revised: Then the conversions arrive according to a Poisson process with a time-varying rate r ( τ ) , deﬁned as follows r ( τ ; w, θ )

Original: [Equation]
Revised: = n (cid:88) i = d w i ( τ ; w i − w i − 1 , θ w i ) , ∀ τ ∈ [0 , H ] .

Original: More details of inhomogeneous Poisson process is deferred to Appendix A.
Revised: More details of inhomogeneous Poisson process is deferred to Appendix A.

Original: The optimization problem of bidding for incrementality.
Revised: The optimization problem of bidding for incrementality.

Original: While the above incrementality modeling of user conversions originates from the inﬂuential idea of [18], our following formulation of the optimal bidding problem under their user modeling is new to the best of our knowledge, and so is its reinforcement learning solutions presented in this paper for addressing the situations with unknown environment parameters.
Revised: While the above incrementality modeling of user conversions originates from the inﬂuential idea of [18], our following formulation of the optimal bidding problem under their user modeling is new to the best of our knowledge, and so is its reinforcement learning solutions presented in this paper for addressing the situations with unknown environment parameters.

Original: Following current practice, we assume online ad opportunities are sold to advertisers by auctions, and our learner is one of these advertisers.
Revised: Following current practice, we assume online ad opportunities are sold to advertisers by auctions, and our learner is one of these advertisers.

Original: For the ease of presentation, we restrict our descriptions to the widely adopted second price auctions ; however, all our results can be easily adapted to any single-item auction format without entry fees (e.g., the ﬁrst price auction).
Revised: For the ease of presentation, we restrict our descriptions to the widely adopted second price auctions ; however, all our results can be easily adapted to any single-item auction format without entry fees (e.g., the ﬁrst price auction).

Original: Let B denote the bidding space, which may be continuous.
Revised: Let B denote the bidding space, which may be continuous.

Original: Notably, 0 ∈ B , meaning the bidder/learner can choose to not participate this round’s auction.
Revised: Notably, 0 ∈ B , meaning the bidder/learner can choose to not participate this round’s auction.

Original: Moreover, denote the distribution of the highest bid among other bidders (HOB) at round h as F h (the cumulative distribution function or CDF).
Revised: Moreover, denote the distribution of the highest bid among other bidders (HOB) at round h as F h (the cumulative distribution function or CDF).

Original: For any episode t (corresponding to an Internet user), at each round h , the learner submits a bid b ∈ B and wins the opportunity to show its ad to the user if her bid is the highest.
Revised: For any episode t (corresponding to an Internet user), at each round h , the learner submits a bid b ∈ B and wins the opportunity to show its ad to the user if her bid is the highest.

Original: Let p h :
Revised: Let p h :

Original: B
Revised: B

Original: → R ≥ 0 represents the average (conditional) payment function of the learner if she wins.
Revised: → R ≥ 0 represents the average (conditional) payment function of the learner if she wins.

Original: Let v denote the learner’s average value per conversion, which is known to the learner.
Revised: Let v denote the learner’s average value per conversion, which is known to the learner.

Original: We assume v is bounded, w.l.o.g., within [0 , 1] .
Revised: We assume v is bounded, w.l.o.g., within [0 , 1] .

Revised: Moreover, we can assume B ⊆ [0 , 1] .

Original: Then for any bid b ∈ B , given the current state (cid:96) , the immediate expected utility of the learner at round h (for any episode) is,
Revised: Then for any bid b ∈ B , given the current state (cid:96) , the immediate expected utility of the learner at round h (for any episode) is,

Original: [Equation]
Revised: [Equation]

Original: The astute reader may already ﬁnd the expected utility function of the learner doesn’t depend on parameter λ h .
Revised: The astute reader may already ﬁnd the expected utility function of the learner doesn’t depend on parameter λ h .

Original: This raises the question why we still need to know the function class of d h (i.e., exponential function) and why we still need to learn parameter λ .
Revised: This raises the question why we still need to know the function class of d h (i.e., exponential function) and why we still need to learn parameter λ .

Original: As we will see later in Section 4, due to the complexity of inhomogeneous Poisson process, it is crucial to know the function class of d h and we have to learn parameters λ ﬁrst in order to get a good estimator to β afterwards.
Revised: As we will see later in Section 4, due to the complexity of inhomogeneous Poisson process, it is crucial to know the function class of d h and we have to learn parameters λ ﬁrst in order to get a good estimator to β afterwards.

Original: In second price auctions, p h ( b ) is the expected second highest bid conditioning on winning, which can be computed as follows
Revised: In second price auctions, p h ( b ) is the expected second highest bid conditioning on winning, which can be computed as follows

Original: [Equation]
Revised: [Equation]

Original: As mentioned above, our techniques apply to other auction formats as well.
Revised: As mentioned above, our techniques apply to other auction formats as well.

Original: For example, in ﬁrst price auctions, p h ( b ) = b has an even simpler format.
Revised: For example, in ﬁrst price auctions, p h ( b ) = b has an even simpler format.

Original:  The Optimal Policy of IB and its Reinforcement Learning
Revised:  The Optimal Policy of IB and its Reinforcement Learning

Original: While it may not appear obvious at the ﬁrst glance, we explain how IB can be reduced to a ﬁnitehorizon Markov Decision Process (MDP) in this section and then introduce the corresponding reinforcement learning problem for IB when the environment parameters — namely, the incrementality parameters θ h
Revised: While it may not appear obvious at the ﬁrst glance, we explain how IB can be reduced to a ﬁnitehorizon Markov Decision Process (MDP) in this section and then introduce the corresponding reinforcement learning problem for IB when the environment parameters — namely, the incrementality parameters θ h

Original: = { β h ( l ) } h − 1 l =1 ∪ { λ h } and HOB distribution F h — are unknown.
Revised: = { β h ( l ) } h − 1 l =1 ∪ { λ h } and HOB distribution F h — are unknown.

Original: 3.1 The Ofﬂine Optimal Policy of IB
Revised: 3.1 The Ofﬂine Optimal Policy of IB

Original: We now formulate the optimal planning problem for the IB problem in its ofﬂine version, i.e., when all environment parameters are known.
Revised: We now formulate the optimal planning problem for the IB problem in its ofﬂine version, i.e., when all environment parameters are known.

Original: We start by deﬁning what is a policy for the IB problem.
Revised: We start by deﬁning what is a policy for the IB problem.

Original: Deﬁnition 1 (Policy) .
Revised: Deﬁnition 1 (Policy) .

Original: A deterministic (dynamic) policy is a sequence of mappings
Revised: A deterministic (dynamic) policy is a sequence of mappings

Original: π
Revised: π

Original: =
Revised: =

Original: ( π 1 , · · · , π H ) , in which mapping π h : [ h ] → B maps any (cid:96) ∈
Revised: ( π 1 , · · · , π H ) , in which mapping π h : [ h ] → B maps any (cid:96) ∈

Original: [ h ] to a bid.
Revised: [ h ] to a bid.

Original: Here, (cid:96) is chosen such that the last winning of the learner happens at round h − (cid:96) .
Revised: Here, (cid:96) is chosen such that the last winning of the learner happens at round h − (cid:96) .

Original: Note that it sufﬁce to consider policies that depend only on the number of consecutive loses (cid:96) due to the Markovian property of incrementality as mentioned above.
Revised: Note that it sufﬁce to consider policies that depend only on the number of consecutive loses (cid:96) due to the Markovian property of incrementality as mentioned above.

Original: The MDP re-formulation of IB .
Revised: The MDP re-formulation of IB .

Original: The MDP will have H states , with state (cid:96) ∈
Revised: The MDP will have H states , with state (cid:96) ∈

Original: [ H ] denoting that the last winning round is (cid:96) rounds ago from current round.
Revised: [ H ] denoting that the last winning round is (cid:96) rounds ago from current round.

Original: The action set is B , containing all possible bids.
Revised: The action set is B , containing all possible bids.

Original: Both the state and action space are the same for each round h , however the transition and rewards are different for different rounds.
Revised: Both the state and action space are the same for each round h , however the transition and rewards are different for different rounds.

Original: Speciﬁcally, at round h , the transition probability from state l and action b is to enter state 1 at round h
Revised: Speciﬁcally, at round h , the transition probability from state l and action b is to enter state 1 at round h

Original: + 1 with probability F h ( b ) , i.e., the probability of winning the auction, and to enter state (cid:96)
Revised: + 1 with probability F h ( b ) , i.e., the probability of winning the auction, and to enter state (cid:96)

Original: + 1 with probability 1 − F h ( b ) .
Revised: + 1 with probability 1 − F h ( b ) .

Original: The immediate reward from transitioning is r h ( (cid:96), b, 1)
Revised: The immediate reward from transitioning is r h ( (cid:96), b, 1)

Original: = (cid:82) ∞
Revised: = (cid:82) ∞

Original: h v ( τ ) d h ( τ ; (cid:96), θ h ) dτ − p h ( b ) for tuple ( (cid:96), b, 1) (the winning case) and r h ( (cid:96), b, (cid:96) + 1)
Revised: h v ( τ ) d h ( τ ; (cid:96), θ h ) dτ − p h ( b ) for tuple ( (cid:96), b, 1) (the winning case) and r h ( (cid:96), b, (cid:96) + 1)

Original: = 0 for tuple ( (cid:96), b, (cid:96) + 1) (the losing case).
Revised: = 0 for tuple ( (cid:96), b, (cid:96) + 1) (the losing case).

Original: This last statement depends crucially on the independence of the incrementality across different winning rounds.
Revised: This last statement depends crucially on the independence of the incrementality across different winning rounds.

Original: The initial state of this
Revised: The initial state of this

Original: MDP is always the lowest state, i.e., (cid:96) 1 = 1 .
Revised: MDP is always the lowest state, i.e., (cid:96) 1 = 1 .

Original: Consequently, the optimal policy in IB can be computed via standard dynamic programming.
Revised: Consequently, the optimal policy in IB can be computed via standard dynamic programming.

Original: Speciﬁcally, when the action set B has ﬁnite support (i.e., the advertiser has ﬁnite bids to choose from), the following proposition follows from the fact that MDPs with ﬁnite states, actions and horizons can be solved efﬁciently by dynamic programming [1].
Revised: Speciﬁcally, when the action set B has ﬁnite support (i.e., the advertiser has ﬁnite bids to choose from), the following proposition follows from the fact that MDPs with ﬁnite states, actions and horizons can be solved efﬁciently by dynamic programming [1].

Original: Proposition 1.
Revised: Proposition 1.

Original: The ofﬂine optimal policy π for any IB instance can be computed by dynamic programming in poly ( |B| , H ) time.
Revised: The ofﬂine optimal policy π for any IB instance can be computed by dynamic programming in poly ( |B| , H ) time.

Original: We remark that even when B ⊆ R is continuous with upper bound B , an (cid:15) -optimal policy can be computed in poly ( B(cid:15) , H ) time since the reward function is continuous in b ∈ B and an (cid:15) -optimal action can be found at each round during the backward induction by discretizing the entire line of bid into (cid:15) segments.
Revised: We remark that even when B ⊆ R is continuous with upper bound B , an (cid:15) -optimal policy can be computed in poly ( B(cid:15) , H ) time since the reward function is continuous in b ∈ B and an (cid:15) -optimal action can be found at each round during the backward induction by discretizing the entire line of bid into (cid:15) segments.

Original: This (cid:15) -optimal action choices lead to an approximately optimal policy [1].
Revised: This (cid:15) -optimal action choices lead to an approximately optimal policy [1].

Original: 3.2 Reinforcement Learning of IB with Unknown Parameters
Revised: 3.2 Reinforcement Learning of IB with Unknown Parameters

Original: We now turn to the much more interesting and realistic situation of the online learning, particularly reinforcement learning, of the optimal IB solution.
Revised: We now turn to the much more interesting and realistic situation of the online learning, particularly reinforcement learning, of the optimal IB solution.

Original: While we have shown that the ofﬂine version of
Revised: While we have shown that the ofﬂine version of

Original: IB problem can be viewed as an MDP, its online counterpart cannot be similarly tackled by standard reinforcement learning techniques due to the following novel challenges speciﬁc to the IB problem:
Revised: IB problem can be viewed as an MDP, its online counterpart cannot be similarly tackled by standard reinforcement learning techniques due to the following novel challenges speciﬁc to the IB problem:

Original: • Challenge 1: mixed reward feedback.
Revised: • Challenge 1: mixed reward feedback.

Original: Any realized learner reward, arising from a user conversion, is naturally a mixture effect of all the previous winning rounds, as modeled by the accumulated incrementality rate in Equation (2).
Revised: Any realized learner reward, arising from a user conversion, is naturally a mixture effect of all the previous winning rounds, as modeled by the accumulated incrementality rate in Equation (2).

Original: This is in contrast to standard MDP, in which the reward of this round is realized independently, conditioned on its state.
Revised: This is in contrast to standard MDP, in which the reward of this round is realized independently, conditioned on its state.

Original: It is not difﬁcult to see that the loglikelihood of our mixed rewards model is non-convex, which renders the standard RL techniques based on maximum likelihood estimation for reward parameter estimation not applicable.
Revised: It is not difﬁcult to see that the loglikelihood of our mixed rewards model is non-convex, which renders the standard RL techniques based on maximum likelihood estimation for reward parameter estimation not applicable.

Original: • Challenge 2: delayed reward realizations.
Revised: • Challenge 2: delayed reward realizations.

Original: Moreover, the conversions in the IB problem follow a intricate Poisson process with heterogeneous rate and thus has delays.
Revised: Moreover, the conversions in the IB problem follow a intricate Poisson process with heterogeneous rate and thus has delays.

Original: That is, showing an ad may not immediately lead to conversions.
Revised: That is, showing an ad may not immediately lead to conversions.

Original: Such lack of immediate reward feedback renders standard Q-learning style of algorithms not easily applicable to our setup since we cannot use immediate reward feedback to update Q-functions any more.
Revised: Such lack of immediate reward feedback renders standard Q-learning style of algorithms not easily applicable to our setup since we cannot use immediate reward feedback to update Q-functions any more.

Original: To overcome the ﬁrst challenges above, we design a novel parameter estimation methods based on moment matching to estimate θ h ’s from mixed reward feedback, and prove its convergence rate in
Revised: To overcome the ﬁrst challenges above, we design a novel parameter estimation methods based on moment matching to estimate θ h ’s from mixed reward feedback, and prove its convergence rate in

Original: Section 4.
Revised: Section 4.

Original: To address the second challenge above, we design a model-based no-regret RL algorithm that integrate the parameter estimation above, and analyze its regret in Section 5.
Revised: To address the second challenge above, we design a model-based no-regret RL algorithm that integrate the parameter estimation above, and analyze its regret in Section 5.

Original: Learner’s Observations.
Revised: Learner’s Observations.

Original: We now formalize our learning setup.
Revised: We now formalize our learning setup.

Original: We consider the episodic online learning setting, where the learner does not know the IB parameters θ h = { β h ( (cid:96) ) } h − 1 (cid:96) =1 ∪ { λ h } for each h ∈
Revised: We consider the episodic online learning setting, where the learner does not know the IB parameters θ h = { β h ( (cid:96) ) } h − 1 (cid:96) =1 ∪ { λ h } for each h ∈

Original: [ H ] neither the distribution of the highest other bid (HOB) F h .
Revised: [ H ] neither the distribution of the highest other bid (HOB) F h .

Original: Following the typical practice, we assume that the learner can always observe the realized HOB at any round h in each episode, denoted as m th , regardless whether the learner wins or loses.
Revised: Following the typical practice, we assume that the learner can always observe the realized HOB at any round h in each episode, denoted as m th , regardless whether the learner wins or loses.

Original: This is usually termed as “ full information feedback ” in the literature of learning to bid .
Revised: This is usually termed as “ full information feedback ” in the literature of learning to bid .

Original: 6 Moreover, the learner also knows the set of its winning rounds denoted by an ordered set W t ⊂
Revised: 6 Moreover, the learner also knows the set of its winning rounds denoted by an ordered set W t ⊂

Original: [ H ] as well as the set of the time stamps of all conversions C t (a subset of R ) from each episode.
Revised: [ H ] as well as the set of the time stamps of all conversions C t (a subset of R ) from each episode.

Original: In the online learning setting, the learner aims to design a bidding algorithm A to minimize the expected regret deﬁned in the following,
Revised: In the online learning setting, the learner aims to design a bidding algorithm A to minimize the expected regret deﬁned in the following,

Original: [Equation]
Revised: [Equation]

Original: where OPT( θ, F ) represents the optimal expected utility achieved by the learner at each episode if the ground-truth parameters θ and F are given (recall that the MDP always starts from state (cid:96) = 1 ), E represents the expectation over the randomness of algorithm A , and π t is the policy in the t th episode generated by the algorithm A .
Revised: where OPT( θ, F ) represents the optimal expected utility achieved by the learner at each episode if the ground-truth parameters θ and F are given (recall that the MDP always starts from state (cid:96) = 1 ), E represents the expectation over the randomness of algorithm A , and π t is the policy in the t th episode generated by the algorithm A .

Original: Here, we slightly abuse the notation to denote R ( π ; (cid:98) θ, (cid:98) F ) as the expected utility achieved by policy π for estimated parameters (cid:98) θ and (cid:98) F .
Revised: Here, we slightly abuse the notation to denote R ( π ; (cid:98) θ, (cid:98) F ) as the expected utility achieved by policy π for estimated parameters (cid:98) θ and (cid:98) F .

Original: [Equation]
Revised: [Equation]

Original: where (cid:98) P π represents the joint distribution of states ( (cid:96) 1 , · · · , (cid:96) H ) induced by a bidding policy π in the MDP formulated in Section 3.1 with estimated parameters (cid:98) θ and (cid:98) F .
Revised: where (cid:98) P π represents the joint distribution of states ( (cid:96) 1 , · · · , (cid:96) H ) induced by a bidding policy π in the MDP formulated in Section 3.1 with estimated parameters (cid:98) θ and (cid:98) F .

Original: Note, for second price auctions, (cid:98) p h( b ) =
Revised: Note, for second price auctions, (cid:98) p h

Revised: [Equation]

Original: b− 1 (cid:98) F h ( b ) (cid:82) b 0 (cid:98) F h ( v ) dv (replacing F by(cid:98) F in Eq.
Revised: b ) = b − [Equation] replacing F by

Original:  In other words, R ( π ; (cid:98) θ, (cid:98) F ) captures the expected total reward in the MDP when the reward function and transition probability are parameterized by (cid:98) θ and (cid:98) F .
Revised: [Equation] F in Eq. In other words, R ( π ; (cid:98) θ, (cid:98) F ) captures the expected total reward in the MDP when the reward function and transition probability are parameterized by (cid:98) θ and (cid:98) F .

Revised: Remark on a non-degeneracy assumption on the learner.

Revised: We assume that the learner’s bidding space B is upper bounded away from extreme bids.

Revised: More formally, there exists a small constant csuch that F ( b ) ≤ 1 − c 0 for any b ∈ B (i.e., no feasible bids guarantee sure winning).

Original: Incrementality Parameter Estimation via Pairwise Moment-Matching
Revised: We call such learner c 0 -bounded . There are two reasons for making this assumption. The ﬁrst is a technical reason: it makes sure that the learner always has at least c 0 probability to lose, which turns out to be crucial to estimate the incrementality parameters, as we do in the next section. This is because incrementality captures the difference between winning and losing.

Revised: If the probability of losing is extremely small, there is generally no way to accurately estimate their differences (though it turns out that we can afford small winning probabilities).

Revised: The second reason for assuming c 0 -bounded learners is realistic motivations since in auto bidding practice, bidders typically explicitly specify upper bounds of their bids, under which the bidder’s winning probability is almost always capped strictly below 1 .

Revised:  Incrementality Parameter Estimation via Pairwise Moment-Matching

Original: Though maximum likelihood estimation (MLE) is a natural and straightforward parameter estimation strategy, it comes with many drawbacks we want to avoid.
Revised: Though maximum likelihood estimation (MLE) is a natural and straightforward parameter estimation strategy, it comes with many drawbacks we want to avoid.

Original: First, the MLE criterion under our model of mixed rewards is not a convex problem and optimizing the likelihood would suffer from multiple local optimal or saddle point issues.
Revised: First, the MLE criterion under our model of mixed rewards is not a convex problem and optimizing the likelihood would suffer from multiple local optimal or saddle point issues.

Original: The theoretical claims may not match the true algorithm performance.
Revised: The theoretical claims may not match the true algorithm performance.

Original: Secondly, the computational efﬁciency of the MLE problem is worse than what we will introduce next, which makes it less attractive in large-scale problems.
Revised: Secondly, the computational efﬁciency of the MLE problem is worse than what we will introduce next, which makes it less attractive in large-scale problems.

Original: Our algorithm is carefully tailored for estimating parameters of our model, given the observed trajectories of ﬁrst t rounds.
Revised: Our algorithm is carefully tailored for estimating parameters of our model, given the observed trajectories of ﬁrst t rounds.

Original: Speciﬁcally, the trajectory of each episode t ∈ [ T ] can be described by two sequences: (1) the time stamps C t of realized conversions within time [0 , ∞ ] ; and (2) the ordered subset W t = { w t 1 , · · · , w tn t } ⊆
Revised: Speciﬁcally, the trajectory of each episode t ∈ [ T ] can be described by two sequences: (1) the time stamps C t of realized conversions within time [0 , ∞ ] ; and (2) the ordered subset W t = { w t 1 , · · · , w tn t } ⊆

Original: [ H ] which contains all the winning steps within this episode (for convenience, we extend the above notations and by letting w t 0 = 0 , w tn t +

Revised: [ H ] which contains all the winning steps within this episode (for convenience, we extend the above notations and by letting w t 0 = 0 , w tn t +

Original: = H + 1 ), where n t denotes the number of winning events (showing the ad) in the episode t .
Revised: = H + 1 ), where n t denotes the number of winning events (showing the ad) in the episode t .

Original: {+Equivalently, we can also represent W t by a H -length binary sequence ζ t ∈ { 0 , 1 } H such that ζ th = 
+}
Revised: {+Equivalently, we can also represent W t by a H -length binary sequence ζ t ∈ { 0 , 1 } H such that ζ th = 
+}

Revised: if and only if episode t wins at time h .

Original: Notably, both C t and W t may be empty if there is no conversion nor winning.
Revised: Notably, both C t and W t may be empty if there is no conversion nor winning.

Original: Our estimation is based on an online algorithm: from a high-level perspective, it progresses by time intervals [ h, h +1) sequentially to produce estimates of parameters θ h , by pairwisely coupling episodes and matching statistical moments.
Revised: Our estimation is based on an online algorithm: from a high-level perspective, it progresses by time intervals [ h, h + 1) sequentially to produce estimates of parameters θ h , by pairwisely coupling episodes and matching statistical moments.

Original: Therefore, we call it the pairwise moment-matching ( PAMM ) algorithm, described in Algorithm 1.
Revised: Therefore, we call it the pairwise moment-matching ( PAMM ) algorithm, described in Algorithm 1.

Original: The PAMM algorithm is speciﬁcally designed for estimating parameters from mixed reward signals, each drawn from a Poisson process.
Revised: The PAMM algorithm is speciﬁcally designed for estimating parameters from mixed reward signals, each drawn from a Poisson process.

Original: Its primary advantage lies in its online nature, guaranteed convergence and efﬁcient computation.
Revised: Its primary advantage lies in its online nature, guaranteed convergence and efﬁcient computation.

Original: This is achieved by the following observations.
Revised: This is achieved by the following observations.

Original: First, all information about reward scale parameter β h and reward variance parameter
Revised: First, all information about reward scale parameter β h and reward variance parameter

Original: λ
Revised: λ

Original: h is intrinsically reﬂected only in the “differential behaviors” for those episodes which had conversions vs. those episodes which did not have conversions at time step h  , conditioning their matched history before time has deﬁned in the sample matching step.
Revised: h is intrinsically do 5: Deﬁne n h to winning h . Deﬁne Ψ = { 0 , 1 } h −be set of all possible winning sequences before time h

Original: This is due to the Markovian property for the Poisson process, and our moment matching is hinged into this property. Second, for the Poisson distribution (as a member of the exponential family), the method of matching the canonical parameter (mean) gives an equivalent estimator as the MLE in a single Poisson case (but not necessary in our more complicated model). These two ideas motivate our procedure, by matching the episodes, we precisely locate the needed Poisson signals and solve the estimators by the mean value structure.
Revised: For each winning sequence φ ∈ Ψ h , deﬁne

Original: Because of this, the PAMM can recover the parameters with essentially optimal rate (in the online sense), even though we do not use the full likelihood that would result in non-convexity and an unpleasant gap between the theory and the practical algorithm.
Revised: [Equation]

Original: 4: for t ∈ (cid:101) Ω h where w ts = h for some s ∈ [ H ] ,apply the reﬁning and matching procedure do 5: Find another episode φ ( t ) ∈ Λ h such that
Revised: Moreover, deﬁne n φ = | Ω φ | , n (cid:48) φ = | Λ φ | , and (cid:101) n φ = (cid:0) n [Equation]

Revised: +(

Original: [Equation]
Revised: [Equation]

Revised: (cid:48)

Revised: [Equation]

Revised: [Equation]

Revised: 1(cid:1) − 1 as the harmonic mean of n φ and n (cid:48) φ .

Revised: For φ

Revised: [Equation]

Original: If such a φ ( t ) is successfully found, remove φ ( t ) from Λ h , add t to Ω h .
Revised: t [Equation] (cid:88) [Equation]

Original: 6: end for 7: For any t , deﬁne N th and N th to be the number of conversions of episode t within [ h, h + 12 ) and [ h + 12 , h + 1) , respectively.
Revised: N th and Y [Equation]

Original:  For each t ∈ Ω h  , deﬁne the two statistics that model the difference of the number of the conversions between two episodes
Revised: [Equation] For each φ ∈ { 0 , 1 } h − 1 , deﬁne [Equation]

Original: [Equation]
Revised: [Equation]

Original: 8: Deﬁne (cid:98)µ h = 1 n h (cid:80) t ∈ Ω h
Revised: and set (cid:98)

Original: [Equation]
Revised: [Equation]

Original: th and
Revised: α φ X

Original: [Equation]
Revised: [Equation]

Original:  = 
Revised: and (cid:98) η h = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: th . Estimate λ h by
Revised: Estimate λ h by

Original: [Equation]
Revised: [Equation]

Original: 9: for 0 ≤ (cid:96) ≤ h do 10:
Revised: 10) 8: for 1 ≤ (cid:96) ≤ h do 9:

Original: Deﬁne the (cid:96) th slice of Ω h as Ω h,(cid:96)
Revised: Deﬁne the (cid:96) th slice of Ψ h as Ψ h,(cid:96)

Original: = { t : t ∈ Ω h, w ts t −  1  =  h − (cid:96) } . Set n h ( (cid:96) ) =
Revised: = { φ ∈ Ψ h : φ h − (cid:96) = 1 , φ k = 0 , h − (cid:96) <

Original: | Ω h,(cid:96) | .
Revised: k ≤ h − 1 } .

Original: For each t ∈ Ω h,(cid:96) ,  deﬁne (cid:98) µ h(cid:96) = 1 n h ( (cid:96)
Revised: For each winning sequence φ ∈ Ψ h,(cid:96) , still deﬁne X φ and Y φ following (7). Deﬁne

Original: [Equation]
Revised: [Equation]

Original: (cid:80) t ∈ Ω h,(cid:96) X th and (cid:98) η h(cid:96)
Revised: α φ(cid:96) X φ and (cid:98) η h(cid:96)

Original: =1 n h ( (cid:96) )
Revised: =

Original: (cid:80) t ∈ Ω h,(cid:96)
Revised: (cid:88) φ ∈ Ψ h,(cid:96)

Original: Y th .
Revised: [Equation]

Original: Estimate β h ( (cid:96) ) by
Revised: φ(cid:96)

Original: [Equation]
Revised: [Equation]

Revised: (11) where α φ(cid:96) is deﬁned following (8) after replacing Ψ h by Ψ h,(cid:96) .

Revised: Estimate β h ( (cid:96) ) by

Original: [Equation]
Revised: [Equation]

Original: 11: end for 
Revised: 12) 12: end for 13: end for reﬂected only in the “differential behaviors” for those episodes which had conversions vs. those episodes which did not have conversions at time step h , conditioning their matched history before time h as deﬁned in the sample matching step.

Original: The detailed derivation of  the algorithm are given in Appendix B.
Revised: This is due to the Markovian property for the Poisson process, and our moment matching is hinged into this property. Second, for the Poisson distribution (as a member of the exponential family), the method of matching the canonical parameter (mean) gives an equivalent estimator as the MLE in a single Poisson case (but not necessary in our more complicated model). These two ideas motivate our procedure, by matching the episodes, we precisely locate the needed Poisson signals and solve the estimators by the mean value structure. Because of this, the PAMM can recover the parameters with essentially optimal rate (in the online sense), even though we do not use the full likelihood that would result in non-convexity and an unpleasant gap between the theory and the practical algorithm.

Original: In practice, terms in (8) and (9) may result in ±∞ , but such an event only happens with probability tending to zero (which is why Theorem 1 has high probability guarantees).
Revised: In practice, terms in (10) and (12) may result in ±∞ , but such an event only happens with probability tending to zero (which is why Theorem 1 has high probability guarantees).

Original: Standard numerical protection can be introduced in practice which will not change our theoretical claims.
Revised: Standard numerical protection can be introduced in practice which will not change our theoretical claims.

Original: Moreover, from the implementation perspective, one start from the ﬁrst interval [0 , 1) and move forward by running PAMM for each [ h, h + 1) .
Revised: Moreover, from the implementation perspective, one start from the ﬁrst interval [0 , 1) and move forward by running PAMM for each [ h, h + 1) .

Original: In each new interval, the matching step can be directly carried over by further reﬁning the matching of the previous interval.
Revised: In each new interval, the matching step can be directly carried over by further reﬁning the matching of the previous interval.

Original: As a result, the whole estimation procedure only involves one sweep of the data without the commuting back and forth between different time intervals.
Revised: As a result, the whole estimation procedure only involves one sweep of the data without the commuting back and forth between different time intervals.

Original: We have introduced the PAMM algorithm in the current form as a self-contained algorithm for estimating the inhomogeneous Poisson process, which can be of independent interest.
Revised: We have introduced the PAMM algorithm in the current form as a self-contained algorithm for estimating the inhomogeneous Poisson process, which can be of independent interest.

Original: However, the steps of PAMM can be easily embedded in our online learning algorithm presented in Section 5 without losing its validity.
Revised: However, the steps of PAMM can be easily embedded in our online learning algorithm presented in Sectionwithout losing its validity.

Original: Theorem 1 (Estimation Errors with Finite Samples) .
Revised: Theorem 1 (Estimation Errors with Finite Samples) .

Original: Suppose there exist positive constants C λ and c β , such that c λ ≤ λ h ≤ 1 /c λ and β h ( (cid:96) ) ≥ c β for any h and (cid:96) .
Revised: Suppose there exist positive constants C λ and c β , such that c λ ≤ λ h ≤ 1 /c λ and β h ( (cid:96) ) ≥ c β for any h and (cid:96) , and the learner is c 0 -bounded.

Original: Also assume that the expected number of total conversions for each episode within each interval [ h, h + 1) is always bounded above by a constant C T .
Revised: Also assume that the expected number of total conversions for each episode within each interval [ h, h + 1) is always bounded above by a constant C T .

Revised: Let n h be the total number of winning episodes at time h and n h ( (cid:96) ) be the total number of winning episodes at time h with state (cid:96) .

Original: Let (cid:98) λ
Revised: Let (cid:98) λ

Original: h and (cid:98) β h be the estimators obtained from Algorithm 1.
Revised: h and (cid:98) β h be the estimators obtained from Algorithm 1.

Original: Then, for any 0 ≤ h <
Revised: Then, for any 0 ≤ h <

Original: H and any 0 ≤ (cid:96) <
Revised: H and any 0 ≤ (cid:96) <

Original: h , we have the following guarantees for PAMM :
Revised: h , we have the following guarantees for PAMM :

Original: For any 4 e − C 0 n h < δ < 1 / 2 , with probability at least 1 − δ , | (cid:98) λ
Revised: For any 4 e − C 0 n h < δ < 1 / 2 , with probability at least 1 − δ , | (cid:98) λ

Original: h − λ h | ≤ C(cid:115) log 1  /δ n h 2.
Revised: h − λ h | ≤ C 1 (cid:113) log1 /δ n h .

Original: For any 4 e − C 0 n h ( (cid:96) ) <
Revised: For any 4 e − C 0 n h ( (cid:96) ) <

Original: δ < 1 / 2 , with probability at least 1 − δ , | (cid:98) β h ( (cid:96) ) 
Revised: δ < 1 / 2 , with probability at least 1 − δ , | (cid:98) β h ( (cid:96) ) − β h ( (cid:96) )

Original: − β h ( (cid:96) ) | ≤ C(cid:115) log 1 /δ n h(cid:96) .
Revised: | ≤ C 2 (cid:113) log1 /δ n h ( (cid:96) ) .

Original: In the above results, C 0  , C 1 , C 2 are positive constants that only depend on c  β , c λ and C  T are explicitly given in the proof.
Revised: In the above results, C 0 = c (cid:48) c 0 2 , C 1 = 4 (cid:114) C 2 T c 2 β (1 − e − cλ/ 2 ) 2 c (cid:48) c 0 and C 2 =(cid:113) 2 C 2 T c (cid:48) c 0 , where c (cid:48) is the Bernstein constant (see Lemma 1).

Original:  The Full RL Algorithm and its Regret Analysis
Revised:  The Full RL Algorithm and its Regret Analysis

Original: We propose a reinforcement learning algorithm by incorporating our parameter estimation method
Revised: We propose a reinforcement learning algorithm by incorporating our parameter estimation method

Original: PAMM and the optimal ofﬂine planning for the MDP mentioned in Section 3 to learn parameters θ, F as well as the conversion incrementality and decide the bid at each round.
Revised: PAMM and the optimal ofﬂine planning for the MDP mentioned in Section 3 to learn parameters θ, F as well as the conversion incrementality and decide the bid at each round.

Original: Our algorithm adopts a UCB-style procedure to handle the exploration-exploitation tradeoff in incrementality bidding, which is summarized in Algorithm 2.
Revised: Our algorithm adopts a UCB-style procedure to handle the exploration-exploitation tradeoff in incrementality bidding, which is summarized in Algorithm 2.

Original: The algorithm adopts pure explorations in several beginning episodes so that we can get enough ad impressions for each state in every round.
Revised: The algorithm adopts pure explorations in several beginning episodes so that we can get enough ad impressions for each state in every round.

Original: These pure explorations are achieved by setting bids appropriately such that we can enforce to explore one ( h, (cid:96) ) pair in each exploration (Line 4 in Algorithm 2) 7 .
Revised: These pure explorations are achieved by setting bids appropriately such that we can enforce to explore one ( h, (cid:96) ) pair in each exploration (Line 4 in Algorithm 2) 7 .

Original: In particular, we need
Revised: In particular, we need

Original: O ( H 2 log( T/δ )) explorations to make sure n h ( (cid:96) ) ≥ log(4 T/δ )
Revised: O ( H 2 log( T/δ )) explorations to make sure n h ( (cid:96) ) ≥ log(4 T/δ )

Original: Cso that we can provide a regret bound with high probability, where C 0 is a positive constant explicitly deﬁned in Theorem 1.
Revised: Cso that we can provide a regret bound with high probability, where C 0 is a positive constant explicitly deﬁned in Theorem 1.

Original: Recall n h ( (cid:96) ) is the total number of winnings at round h given the state is (cid:96) .
Revised: Recall n h ( (cid:96) ) is the total number of winnings at round h given the state is (cid:96) .

Original: After pure explorations, at each episode t , we use the PAMM algorithm to get an estimator (cid:98) θ t (resp.
Revised: After pure explorations, at each episode t , we use the PAMM algorithm to get an estimator (cid:98) θ t (resp.

Original: (cid:98) β t )
Revised: (cid:98) β t )

Original: given the observed ad impression time W t and the conversion time C t up to episode t .
Revised: given the observed ad impression time W t and the conversion time C t up to episode t .

Original: In addition, we can estimate F easily using empirical distribution function and this implies we can get a uniform convergence for the transition probabilities for any action b ∈ B given any state (cid:96) .
Revised: In addition, we can estimate F easily using empirical distribution function and this implies we can get a uniform convergence for the transition probabilities for any action b ∈ B given any state (cid:96) .

Original: Then we apply UCB-style RL algorithm with estimated parameters (cid:98) θ t to decide the policy in next episode.
Revised: Then we apply UCB-style RL algorithm with estimated parameters (cid:98) θ t to decide the policy in next episode.

Original: Speciﬁcally, given the estimated (cid:98) β t , ∀ h ∈
Revised: Speciﬁcally, given the estimated (cid:98) β t , ∀ h ∈

Original: [ H ] , we construct a conﬁdence region centered at (cid:98) β t and we do ofﬂine planning for the optimism in the conﬁdence region to get the policy in next episode.
Revised: [ H ] , we construct a conﬁdence region centered at (cid:98) β t and we do ofﬂine planning for the optimism in the conﬁdence region to get the policy in next episode.

Original: Remark 2 (Computational Efﬁciency of Alg. 2) .
Revised: Remark 2 (Computational Efﬁciency of Alg. 2) .

Original: An important observation is that R ( π ; (cid:98) θ, (cid:98) F ) is non-decreasing in (cid:98) β h ( (cid:96) )
Revised: An important observation is that R ( π ; (cid:98) θ, (cid:98) F ) is non-decreasing in (cid:98) β h ( (cid:96) )

Original: (see Eq.
Revised: (see Eq.

Original: (6)) for any ﬁxed π and (cid:98) F .
Revised: (6)) for any ﬁxed π and (cid:98) F .

Original: Therefore, the optimal choice of (cid:98) β in
Revised: Therefore, the optimal choice of (cid:98) β in

Original: Eq.

Original: (10) in Algorithm 2 is to set (cid:98) β h ( (cid:96) )

Original: = (cid:98)

Original: β th ( (cid:96) )

Original: + C 2 (cid:113) log(1 /δ )

Original: n th ( (cid:96) ) , and π t +1 will be the optimal policy for the MDP parameterized by (cid:98) F t and (cid:98) β , and thus can be computed efﬁciently (Proposition 1).

Original: Regret Analysis.

Original: Combining the convergence rate analysis from Theorem 1 and a novel regret decomposition technique (necessary for our particular setup), we are able to prove the regret guarantee of our online bidding algorithm.

Original: Our regret bound is summarized in Theorem 2, the proof of which is deferred to Appendix C.

Original: Theorem 2.

Original: Under Assumption 1, for any ﬁxed δ , we have with probability at least 1 − δ , the regret achieved by our Algorithm 2 is bounded by

Original: [Equation]

Original: Discussions and Future Work

Original: Our RL algorithm lies in the model-based RL algorithm literature.

Original: The best known regret bound for model-based RL is (cid:101) O ( √ H 2 SAT ) [2], where S is the number of states and A is the number of

Original: 1: Input : Given δ , H , T , and C 0 , C 2 deﬁned in Theorem 1.
Revised: 1: Input : Given δ , H , T , and C 0 , C 2 deﬁned in Theorem 1.

Original: Set n h ( (cid:96) )
Revised: Set n h ( (cid:96) )

Original: = 0 , ∀ h ∈
Revised: = 0 , ∀ h ∈

Original: [ H ] , (cid:96) ∈
Revised: [ H ] , (cid:96) ∈

Original: [ h ] .
Revised: [ h ] .

Original: 2: for Episode t = 1 , · · · , T do 3: if ∃ h, (cid:96), n h ( (cid:96) ) < log(4 T/δ )
Revised: 2: for Episode t = 1 , · · · , T do 3: if ∃ h, (cid:96), n h ( (cid:96) ) < log(4 T/δ )

Original: Cthen 4:
Revised: Cthen 4:

Original: Set bids, s.t.
Revised: Set bids, s.t.

Original: b th = b th − (cid:96) = 1 , and b th (cid:48)
Revised: b th = b th − (cid:96) = 1 , and b th (cid:48)

Original: = 0 , ∀ h (cid:48)
Revised: = 0 , ∀ h (cid:48)

Original: (cid:54)
Revised: (cid:54)

Original: =
Revised: =

Original: h, h − (cid:96) .
Revised: h, h − (cid:96) .

Original: [Pure Exploration] 5: else 6:
Revised: [Pure Exploration] 5: else 6:

Original: For h ∈ [ H ] , observe state (cid:96) th and set the bid b th = π th ( (cid:96) th ) .
Revised: For h ∈ [ H ] , observe state (cid:96) th and set the bid b th = π th ( (cid:96) th ) .

Original: 7: end if 8:
Revised: 7: end if 8:

Original: Observe the set of the ad impression time W t , the set of the conversion time C t , and the vector of HOBs m t = { m th , ∀ h ∈
Revised: Observe the set of the ad impression time W t , the set of the conversion time C t , and the vector of HOBs m t = { m th , ∀ h ∈

Original: [ H ] } .
Revised: [ H ] } .

Original: Update n h ( (cid:96) ) given W t .
Revised: Update n h ( (cid:96) ) given W t .

Original: Update (cid:98) θ t = { (cid:98)
Revised: Update (cid:98) θ t = { (cid:98)

Original: β th ( (cid:96) ) , ∀ (cid:96) ∈
Revised: β th ( (cid:96) ) , ∀ (cid:96) ∈

Original: [ h ]; (cid:98)
Revised: [ h ]; (cid:98)

Original: λ th } through PAMM method (Algorithm 1) given ( W t , C t ) .
Revised: λ th } through PAMM method (Algorithm 1) given ( W t , C t ) .

Original: Update (cid:98) F th ( b ) = 1 t (cid:80) ts =1 I { m sh ≤ b } , ∀ b .
Revised: Update (cid:98) F th ( b ) = 1 t (cid:80) ts =1 I { m sh ≤ b } , ∀ b .

Original: 12: if ∀ h ∈ [ H ] , (cid:96) ∈
Revised: 12: if ∀ h ∈ [ H ] , (cid:96) ∈

Original: [ h ] , n h ( (cid:96) ) ≥ 1 then 13:
Revised: [ h ] , n h ( (cid:96) ) ≥ 1 then 13:

Original: Construct conﬁdence region (cid:99) CR t for parameters θ , s.t.
Revised: Construct conﬁdence region (cid:99) CR t for parameters θ , s.t.

Original: (cid:99) CR t = (cid:40) ( (cid:98) β, (cid:98) λ ) : ∀ h, | (cid:98) β h ( (cid:96) ) − (cid:98)
Revised: (cid:99) CR t = (cid:40) ( (cid:98) β, (cid:98) λ ) : ∀ h, | (cid:98) β h ( (cid:96) ) − (cid:98)

Original: β th ( (cid:96)
Revised: β th ( (cid:96)

Original: [Equation]
Revised: [Equation]

Original: Compute the bidding policy π t +1 for ( t + 1) th episode, s.t.,
Revised: Compute the bidding policy π t +1 for ( t + 1) th episode, s.t.,

Original: [Equation]
Revised: [Equation]

Original: 15: end if 16: end for actions.
Revised: 15: end if 16: end for Eq.

Original: Since S = H in our case, our regret bound is slightly worse from the best possible by an factor of √ H but is independent of the number of actions.
Revised: (13) in Algorithm 2 is to set (cid:98) β h ( (cid:96) ) = (cid:98) β th ( (cid:96) ) + C 2 (cid:113) log(1 /δ )

Revised: n th ( (cid:96) ) , and π t +1 will be the optimal policy for the MDP parameterized by (cid:98) F t and (cid:98) β , and thus can be computed efﬁciently (Proposition 1).

Revised: Regret Analysis.

Revised: Combining the convergence rate analysis from Theorem 1 and a novel regret decomposition technique (necessary for our particular setup), we are able to prove the regret guarantee of our online bidding algorithm.

Revised: Our regret bound is summarized in Theorem 2, the proof of which is deferred to Appendix C.

Revised: Theorem 2.

Revised: Under Assumption 1, for any ﬁxed δ , we have with probability at least 1 − δ , the regret achieved by our Algorithm 2 is bounded by

Revised: [Equation]

Revised:  Discussions and Future Work

Revised: Our RL algorithm lies in the model-based RL algorithm literature.

Revised: The best known regret bound for model-based RL is (cid:101) O ( √ H 2 SAT ) [2], where S is the number of states and A is the number of actions.

Revised: Since S = H in our case, our regret bound is slightly worse from the best possible by an factor of √ H , but is independent of the number of actions.

Original: This may be due to the complexity of our model.
Revised: This may be due to the complexity of our model.

Original: Whether we can close this gap is an interesting open question.
Revised: Whether we can close this gap is an interesting open question.

Original: As we mentioned earlier, our algorithm can be extended to handle continuous bidding space and our regret analysis still works.
Revised: As we mentioned earlier, our algorithm can be extended to handle continuous bidding space and our regret analysis still works.

Original: In this paper, we assume the conversion rate function d h doesn’t depend on the shown ads of other bidders but only on the learner’s last ad impression.
Revised: In this paper, we assume the conversion rate function d h doesn’t depend on the shown ads of other bidders but only on the learner’s last ad impression.

Original: Relaxing these assumptions may result in very interesting future work.
Revised: Relaxing these assumptions may result in very interesting future work.

Original: Acknowledgements
Revised: Acknowledgements

Original: T. Li is supported in part by the NSF grant DMS-2015298 and 3Caverliers award from the University of Virginia.A Formal Deﬁnition of Inhomogeneous Poisson Process
Revised: T. Li is supported in part by the NSF grant DMS-2015298 and 3Caverliers award from the University of Virginia.

Original: The inhomogeneous Poisson (point) process is a Poisson point process with a Poisson parameter set as some time-dependent function r ( τ ) .

Original: In particular, the expected number of points observed in a time interval [ a, b ] is Λ( a, b ) =

Original: (cid:82) b a r ( τ ) dτ .

Original: Let N ( a, b ) represent the number of points of inhomogeneous

Original: Poisson process with intensity function r ( t ) occurring in the interval [ a, b ] , then the probability of n points existing in the interval [ a, b ] is given by,

Original: [Equation]

Original: In this paper, the points mean the conversions and the time-dependent intensity function r ( · ) is deﬁned in Eq.

Original: and it depends on the realization of the conversions and parameter θ .

Original: B Poisson Process Estimation

Original: We ﬁrst introduce the main idea of the the PAMM algorithm.

Original: [-Since conversions follow the Poisson process, we have N th ∼ Poisson ( (cid:82) h +1 / 
-]
Revised: [-Since conversions follow the Poisson process, we have N th ∼ Poisson ( (cid:82) h +1 / 
-]

Original: h r ( τ ; w t , θ h ) dτ ) and N th ∼ Poisson ( (cid:82) h

Original: +1 h +1 / 2 r

Original: ( τ ; w t , θ h ) dτ ) independently.

Original: Suppose s t = max { w ti : w ti ≤ h − 1 , 0 ≤ i ≤ H } .

Original: Since each d h is from the re-scaled exponential rate family, it is not difﬁcult to see that

Original: [Equation]

Original: Similarly, we have

Original: [Equation]

Original: Therefore, E ( X th ) and E ( Y th ) precisely locate the parameter of interest, θ h .

Original: By aggregating the information over proper segments in Ω h , we can thus estimate { β h

Original: ( (cid:96) ) , 0 ≤ (cid:96) ≤ h } and λ h based on the method of moments.

Original: Notice that λ h is shared for all episodes t ∈ Ω h while β h

Original: (cid:96) ) is shared only within a subset of Ω h , for a given (cid:96) .

Original: Therefore, we will tailor the moment match operations according to different aggregation levels, as indicated in (8) and (9).

Original: Next, we proceed to study the theoretical property of the algorithm.

Original: Lemma 1 (Bernstein’s Inequality [24]) .

Original: Suppose X 1 , · · · X n are independent, mean-zero, subexponential random variables.

Original: Then, for every (cid:15) > 0 , we have

Original: [Equation]

Original: where (cid:107) X

Original: i (cid:107) ψ is the sub-exponential norm (Orlicz norm) of X i and c (cid:48) is an absolute constant (Bernstein constant).

Original: Proof of Theorem 1.

Original: Deﬁne

Original: [Equation]

Original: and

Original: [Equation]

Original: Taking the logarithm of the ratio between the two quantities, we get

Original: [Equation]

Original: Our estimator (cid:98) λ h is motivated by the above identity with the observation that

Original: [Equation]

Original: Since X th is the difference between two independent Poisson random variables (with a slight abuse of notations)

Original: [Equation]

Original: where µ 1 h,w t = (cid:82)

Original: h +1 / 2 h r ( τ ; w t , θ h ) dτ ≤ C T , µ 2 h,w t

Original: =

Original: (cid:82)

Original: h +1 / 2 h r ( τ ; w φ ( t ) , θ h ) dτ ≤ C T . Therefore, X th is a sub-exponential random variable. The sub-exponential norm of X th is deﬁned as
Revised: H. Xu is supported in part by an ARO award W911NF-23-1-0030.

Original: [Equation]

Original: Optimizing the above problem by Poisson properties, it can be shown that (cid:107)

Original: X th (cid:107) ψ ≤ 2 C T .

Original: Moreover, by the property of sub-exponential norm, we have

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: By Lemma 1, we have

Original: Pr [Equation] | [Equation]
Revised: Checklist

Original: for some absolute constant c (cid:48) > 0 .

Original: Assuming δ > exp( − 4 c (cid:48) n h ) , and setting (cid:15)

Original: = 2 C T √ c (cid:48) (cid:113) log1 /δ

Original: n h , it gives

Original: [Equation]

Original: For notational convenience, we will denote this relation by [Equation] where O P (cid:0)(cid:113) log1 /δ n h (cid:1) refers to any generic term in the order of (cid:113) log1 /δ n h with probability at least 1 − 2 δ .
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: Furthermore, by Taylor expansion, we have

Original: [Equation]

Original: for some (cid:101) µ h lying between (cid:98) µ h and µ h .

Original: Since λ h ≤ C λ and β h ( (cid:96) ) ≥ c β , we have

Original: [Equation]

Original: in which the right hand side is a constant.

Original: Therefore, for (cid:98) µ h − µ h being small enough such that [Equation] we have [Equation]
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: Consequently, under the event

Original: [Equation]

Original: for 2 C T √

Original: [Equation]

Original: <

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: we have

Original: [Equation]

Original: This happens with probability at least 1 − 2 δ , as long as

Original: [Equation]

Original: Similarly, because

Original: [Equation]

Original: we can show the same property for log (cid:98)

Original: η h that with probability at least 1 − 2 δ

Original: [Equation]

Original: Combining the results for log (cid:98) µ h and log (cid:98) η h , we know that

Original: [Equation]

Original: with probability at least 1 − 4 δ .

Original: Note that we can relax the bound constraint on β h and allow negative values.

Original: This is because the only places where we use the lower bound c β so far are (12) and (13).

Original: In general, as long as we can have a lower bound for | µ h | and | η h | , the proof still works.

Original: For the estimation of β h ( (cid:96) ) , a similar approach can be taken, but the stratum of focus will be on the slide Ω h,(cid:96) .

Original: First, notice that all X th share the same expectation within Ω h,(cid:96) with

Original: [Equation]

Original: Therefore, it reads that

Original: [Equation]

Original: Similarly, we have

Original: [Equation]

Original: Now the moment matching equation becomes

Original: [Equation] In this case, note that X th , t ∈ Ω h,(cid:96) are i.i.d observations. By Lemma 1 again, we have [Equation]
Revised: [Yes] (c) Did you discuss any potential negative societal impacts of your work?

Original: with probability at least 1 − 2 δ and similarly,

Original: [Equation]

Original: with probability at least 1 − 2 δ .

Original: Denote the joint of these two events as E 2 , which happens with probability at least 1 − 4 δ .

Original: To control the concentration of the estimating equations

Original: [Equation]

Original: let

Original: [Equation]

Original: Under E

Original: [Equation]

Original: we have

Original: [Equation]

Original: We have, by Taylor expansion,

Original: [Equation] 14) where (cid:101) µ h(cid:96) − (cid:101) η h(cid:96) is between (cid:98) µ h(cid:96) − (cid:98) η h(cid:96) and µ h(cid:96) − η h(cid:96) such that (cid:101)
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Equation]

Original: (cid:101) η h(cid:96)

Original: [Equation]

Original: µ

Original: [Equation]

Original: − η h(cid:96)

Original: + O (∆ 2 ) .

Original: By the assumptions of the theorem, |

Original: [Equation]

Original: [Equation]

Original: and the right hand side is a constant the only depends on c β and c λ .

Original: Now, for a small (cid:113) log1 /δ

Original: n h ( (cid:96) )

Original: such that

Original: [Equation]

Original: we have

Original: [Equation]

Original: Under the event E

Original: [Equation]
Revised: [Yes] 2.

Original: inserting this relation into (14)

Original: [Equation]

Original: we have

Original: [Equation]

Original: We reparameterize 4 δ by δ for δ < 1 / 2 , leading to

Original: [Equation]

Original: with probability at least 1 − δ with

Original: [Equation]

Original: and

Original: [Equation]

Original: For simplicity, we deﬁne

Original: [Equation]

Original: in the theorem statement.

Original: Again, we can relax the constraint on β h .

Original: The proof still holds, as long as (15) holds. This is true if we assume | β h | > c β . C Proof of Theorem To prove Theorem 2, we need several auxiliary lemmas.
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: First, we provide a lemma to bound

Original: R ( π ; (cid:98) θ, (cid:98) F ) − R ( π ; θ, (cid:98) F ) .

Original: The importance of this lemma is that we provide a reformulation of function R ( π ; (cid:98) θ, (cid:98) F ) by utilizing the fact that the conversion incrementality can only happen if the learner wins the opportunity to show the ad to the user.

Original: This observation makes our regret bound independent of the number of actions.

Original: Lemma 2.

Original: Given estimated parameters (cid:98) θ and (cid:98) F , for any bidding policy π , we have

Original: R ( π ; (cid:98) θ, (cid:98) F

Original: [Equation]

Original: , where (cid:98) P π ( L ) is the probability of sequence L

Original: = ( (cid:96) 1 , (cid:96) 2 , · · · , (cid:96) H ) induced by the MDP parameterized by (cid:98) θ and (cid:98) F by adopting policy π and o h ( b ) models whether the learner wins ( o h ( b ) = 1 ) or not ( o h ( b ) = 0 ) at round h with bid b (given HOB follows distribution (cid:98) F h at each round h ).

Original: Proof.

Original: Recall, by deﬁnition of R ( π ; (cid:98) θ, (cid:98) F ) in Eq.

Original: (6), for any (cid:98) θ and (cid:98) F ,

Original: [Equation] Let o h ( b ) denote the realized outcome of the learner at round h , s.t., [Equation] [Equation]
Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Original: Given the above reformulation of R ( π ; (cid:98) θ, (cid:98) F ) , we can derive the difference between R ( π ; (cid:98) θ, (cid:98) F ) and

Original: R ( π ; θ, (cid:98) F ) for any bidding policy π and any distribution of HOB (cid:98) F h at each round h ,

Original: R ( π ; (cid:98) θ, (cid:98) F ) − R ( π ; θ, (cid:98) F ) =

Original: E

Original: L∼ (cid:98) P π ,o h ( · ) ∼ Ber ( (cid:98) F h ( · )) (cid:34) H (cid:88) h

Original: = (cid:104)(cid:0)(cid:98) β h

Original: ( (cid:96) h )

Original: − β h ( (cid:96) h ) (cid:1) · v (cid:105) · I { o h ( π ( (cid:96) h ))

Original: = 1 } (cid:35) ≤

Original: E

Original: L∼ (cid:98) P π ,o h ( · ) ∼ Ber ( (cid:98) F h ( · )) (cid:34) H (cid:88) h

Original: = (cid:12)(cid:12)(cid:98) β h

Original: ( (cid:96) h )

Original: − β h ( (cid:96) h ) (cid:12)(cid:12) · I { o h ( π h ( (cid:96) h ))

Original: = 1 } (cid:35) (22)

Original: Second, we extend the well-known simulation lemma in standard RL [16] to our case.

Original: Please note both transition probability and expected utility function of the learner (Eq. (3)) depend on F .

Original: Lemma 3.

Original: For any ﬁxed bidding strategy π , we have

Original: [Equation]

Original: holds with probability at least 1 − δ .

Original: In addition, we have with probability at least 1 − δ ,

Original: [Equation]

Original: Proof.

Original: Denote (cid:99) M t as the MDP induced by a bidding policy π and parameters θ, (cid:98) F t .

Original: Similarly, let

Original: M represent the MDP for a bidding policy π with parameters θ and F .

Original: Let (cid:98) Γ th ( ·| (cid:96), b ) and Γ h ( ·| (cid:96), b ) be the transition at round h for MDP (cid:99) M t and M , respectively.

Original: Given the MDP re-formulation in Subsection 3.1, we have for any (cid:96), b and h = 2 , 3 , · · · , H , (cid:88)

Original: [Equation]

Original: In addition, we denote r h as the expected reward function for the MDP M at round h , similarly, we can also deﬁne (cid:98) r th for the MDP (cid:99) M t .

Original: For any ( (cid:96), b ) , we can bound | r h ( (cid:96), b ) − (cid:98) r th

Original: ( (cid:96), b ) | as below:

Original: [Equation]

Original: |

Original: [Equation]

Original: where the ﬁrst equality is based on the deﬁnition of r h ( · , · ) and (cid:98) r th ( · , · ) , the second inequality is based on triangle inequality and the last inequality holds because of the fact that β h ( (cid:96) ) v h ∈ [0 , 1] and b ∈ [0 , 1] .

Original: By DKW inequality (Lemma 4) and union bound, with probability at least 1 − δ , we have for all h ∈

Original: [ H ]

Original: [Equation]

Original: Given the above bound of transition and reward between two MDPs M and (cid:99) M t and the Simulation

Original: Lemma (Lemma 6), with probability at least 1 − δ , we have

Original: [Equation]

Original: For notation simplicity, we denote π ∗ as the optimal bidding policy corresponding to parameters θ and F , then we have

Original: [Equation]
Revised: [Yes] 3.

Original: where the ﬁrst inequality holds because of the deﬁnition of OPT( θ, (cid:98) F t ) and the second inequality is based on Eq.

Original: (23) for the ﬁxed bidding policy π ∗ .

Original: Using the same argument as inEq. (24), we complete the proof. Given the above two auxiliary lemmas, we are ready to prove Theorem 2. Proof of Theorem 2.
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: For notation simplicity, let

Original: [Equation]

Original: In addition, we denote τ as the number of episodes for pure explorations.

Original: Note, in each pure exploration t ≤ τ , we exactly increase n h ( (cid:96) ) by 1 for one ( h, (cid:96) ) pair.

Original: To make sure n h ( (cid:96) ) ≥ log(4 T/δ ) C, we only need τ ≤ H 2 log(4 T/δ ) C.

Original: We ﬁrst propose a new decomposition of the expected regret Regret( T ) in the following way,

Original: Regret( T ) =

Original: [Equation]

Original: E (cid:34)

Original: [Equation]

Original: (cid:88) t =

Original: [Equation]

Original: F ) (cid:35) ≤

Original: Hτ + ( T − τ ) · OPT( θ

Original: [Equation]

Original: F ) − E (cid:34) T (

Original: cid:88) t = τ +

Original: R ( π t

Original: [Equation]

Original: θ

Original: [Equation]

Original: F ) (cid:35) =

Original: Hτ + ( T − τ ) · OPT( θ, F ) −

Original: T (cid:88) t = τ +

Original: OPT(

Original: [Equation]

Original: (cid:98) F

Original: [Equation]

Original: − 1 ) (cid:124) (cid:123)(cid:122) (cid:125)

Original: (i)

Original: +

Original: [Equation]

Original: (cid:88)

Original: [Equation]

Original: OPT

Original: [Equation]

Original: F

Original: [Equation]

Original: ) −

Original: [Equation]

Original: (cid:88) t

Original: [Equation]

Original: +

Original: OPT [Equation] (cid:98) F t − 1 ) (cid:124) (cid:123)(cid:122) (cid:125) (ii) + T (cid:88) t = τ +
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: OPT

Original: [Equation]

Original: F

Original: [Equation]

Original: ) − E (cid:34)

Original: [Equation]

Original: (cid:88) t = τ +

Original: R (

Original: [Equation]

Original: t

Original: [Equation]

Original: The ﬁrst inequality above is based on the fact the reward (expected utility of the learner) at each round is bounded by [0 , 1] .

Original: Then we bound the above terms separately in the following.

Original: Term (i).

Original: By Lemma 3, for any ﬁxed t ∈ [ T ] , we have | OPT

Original: [Equation]

Original: θ,

Original: [Equation]

Original: ) 2 t holds with probability at least 1 − δ .

Original: Then by union bound over t = τ, · · · , T − 1 , we have

Original: [Equation] holds with probability at least 1 − δ . Term (ii). After pure explorations, for any t ≥ τ + 1 , we have 4 e − C 0 n h ( (cid:96) ) <
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: δ/T . Then by Theorem 1, we have with probability at least 1 − δ/T , the true parameter θ ∈ (cid:99) CR t . Then by the deﬁnition of OPT( (cid:99) CR t , (cid:98) F t ) , for any t ∈ [ T ] , we have OPT( θ, (cid:98) F t ) − OPT( (cid:99) CR t , (cid:98) F t ) ≤ 0 holds with probability 1 − δ/T .
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: Taking union bound over t = τ, · · · , T − 1 , then with probability at least 1 − δ , we have

Original: [Equation]

Original: Term (iii).

Original: For notation simplicity, let (cid:101) θ t ∈ (cid:99) CR t be the parameter corresponding to π t +1 .

Original: Actually, it is the Then we can bound Term (iii) as below,

Original: [Equation]
Revised: [N/A] 4.

Original: where the inequality is based on Lemma 2 in Appendix. Here we slightly abuse the notation, let o th := o h (cid:0) π t ( (cid:96) th ) (cid:1) . An important observation is that o th can be viewed as the action the learner takes at round h in the t th episode and it only contains two different options, i.e., 0 (lose) or 1 (win).
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: This rewriting can help us reduce the dependency of regret bound with number of bids, and we can easily extend to handle continuous bid space.

Original: Let n th ( (cid:96), o ) represent the total number observations ofpair ( (cid:96), o ) up to round h and t th episode. Then we have, [Equation] where the ﬁrst inequality is based on the fact that (cid:101) θ t ∈ (cid:99) CR t , ∀ t ∈ [ T ] and n th ( (cid:96), 1) = n h ( (cid:96) ) in episode t given the deﬁnition in Algorithm 2.
Revised: [N/A] (b) Did you mention the license of the assets?

Original: The third inequality is based on Lemma 5.

Original: Term (iv). Again, applying Lemma 3 and union bound over t = 1 , · · · , T − 1 , we can bound term (iv) in the following way, [Equation] Putting it all together, we can bound the regret as below,
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Equation] holds with probability at least 1 − δ . D Auxiliary Technical Lemmas In this section, we enumerate several useful technical lemmas used in this paper.
Revised: [N/A] (d) Did you discuss whether

Original: First, we introduce the well-known Dvoretzky–Kiefer–Wolfowitz (DKW) inequality which is used to bound the gap between (cid:98) F t ( · ) and F ( · ) . Lemma 4 (DKW Inequality) . For any episode t , at each round h ∈ [ F ] , [Equation]
Revised: and how consent was obtained from people whose data you’re using/curating?

Original: log

Original: [Equation] [Equation] holds with probability at least 1 − δ . Second, we describe the celebrated simulation lemma, which was introduced and named in [16].
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: For completeness of the presentation, we provide the proof for this lemma.

Original: Lemma 5 (Lemma 7.5 in [1]) .

Original: Consider arbitrary T sequence of trajectories, { (cid:96) th , o th } Hh =1 for t = 1 , · · · , T , we have

Original: [Equation] i
Revised: [N/A] 5.

Original: Finally, we describe the well-known simulation lemma.

Original: For completeness of exposition, we provide a simple proof for this lemma. Lemma 6 (Simulation Lemma [16]) . Consider two different MDPs M and M (cid:48) with the same state and action spaces, S and A . If the transition ( P and P (cid:48) resp.) and reward functions ( r and r (cid:48) resp., bounded by [0 , 1] ) of these two MDPs satisfy ∀ s ∈ S , a ∈ A , (cid:88)
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Equation]

Original: Then for every non-stationary policy π and ﬁxed initial state s 1 , the two MDPs satisfy (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) (cid:12)

Original: E { s h ,a h } H h =1 ∼ M,π   (cid:88) h

Original: [Equation]

Original: [Equation]

Original: h ) 

Original:  −

Original: E { s

Original: [Equation]

Original: ,a h } H h =1 ∼ M (cid:48) ,π   (cid:88)

Original: h ∈

Original: [ H ] r (cid:48) h ( s h , a h )   (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) (cid:12) ≤ H ( H − 1) ε 1 + Hε(29)
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: Proof. First, we denote V πh ( s, M ) as the total reward from round h to H when the state at round h is s . Similarly we can deﬁne V πh ( s, M (cid:48) ) . Then Eq.
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: (29) is equivalent to show | V π 1 ( s 1 , M ) − V π 1 ( s 1 , M (cid:48) )

Original: [-| ≤ H ( H − 1)ε 1 + Hε 
-]
Revised: [-| ≤ H ( H − 1)ε 1 + Hε 
-]

Original: and we will prove it using the inductive hypothesis that for any state s , policy π and round h

Original: = 1 , 2 , · · · , H ,

Original: [Equation]

Original: For h = H , the hypothesis is clearly true since reward of two MDPs at round H only differs at most ε 2 and this establishes the base case.

Original: For the inductive step, we utilize the recursive formula of V πh ( s, M ) and V πh ( s, M (cid:48) ) in the following way,

Original: [Equation]

Original: [Table]

Original: V πh

Original: [Table]
Revised: [N/A]

Original: M

Original: [Table]

Original: ) V πh

Original: [Table]

Original: P (cid:48)

Original: ( s (cid:48) | s, a ) (cid:12)(cid:12)

Original: [-V πh +
-]
Revised: [-V πh +
-]

Original: ( s (cid:48) , M ) − V πh +1 ( s (cid:48) , M (cid:48) )

Original: (cid:12)(cid:12) ≤ ε

Original: [Equation]

Original: where the ﬁrst inequality holds since | r ( s, a ) − r (cid:48)

Original: ( s, a ) | ≤ ε 2 ; the second inequality is based on the triangle inequality; the third inequality is because of the fact that V πh +1 ( s, M ) ≤ H − h (recall the reward at each round is bounded by [0 , 1] ); the fourth inequality holds due to the inductive hypothesis.

Original: Then we complete the proof.