Sustainable Online Reinforcement Learning for Auto-bidding
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recently, auto-bidding technique has become an essential tool to increase thereturn on investment (ROI) for advertisers.
Facing the complex and ever-changingbidding environments in the real-world advertising system (RAS), state-of-the-artauto-bidding policies usually leverage reinforcement learning (RL) algorithms togenerate real-time bids on behalf of the advertisers.
Due to safety concerns, it was believed that the RL process can only be carried out in a virtual advertisingsystem (VAS) which is usually built based on the historical data generated in the
RAS.
In this paper, we argue that there exists signiﬁcant gaps between VAS and
RAS, making the RL process in the VAS suffer from the problem of inconsistency between online and ofﬂine (IBOO).
Firstly, we formally deﬁne the IBOO and systematically analyze its causes and inﬂuences.
Then, we design a safety function and prove its Lipschitz smooth property in theory, which provides theoretical foundations for solving the IBOO problem.
Moreover, based on this property, we propose a sustainable online RL (SORL) framework to continuously improve the auto-bidding polices through direct interactions with the RAS, which can completely resolve the IBOO problem.
Speciﬁcally, we devise a safe and efﬁcientonline exploration policy that can continuously collect data from the RAS.
We also develop a variance-suppressed conservative Q-learning method to improvethe auto-bidding policies with the collected data.
Extensive experiments on bothsimulated and real-world advertising systems validate the effectiveness of ourapproach.
Introduction
In the era of Internet, online advertising business has become one ofthe main proﬁt models for many companies, such as Google [7] and
Alibaba [4], which, at the same time, beneﬁts millions of advertiserswho are willing to bid for impression opportunities.
Contemporaryonline advertising systems, as auctioneers, usually have large amount ofcandidate advertisers contesting for numerous impression opportunitiesat every moment.
Making auction decisions based on accurate evaluation of each impression opportunity for all advertisers within severalmilli-seconds is computationally infeasible.
Therefore, a real-worldadvertising system (RAS) adopts a cascade architecture.
In this architecture, the auction of each impression opportunity is completed throughseveral stages.
Without loss of generality, we here simply view the RASas a system with two stages: stage 1 and stage 2, as shown in Fig.
Theauction process of each impression opportunity is described as follows: in stage 1, rough evaluations are conducted for all the candidate advertisers ( ⇠ 10 6 ), and a group ofthe most promising advertisers ( ⇠ 10 2 ) are fed to the stage 2; and in stage 2, accurate valuations andauction are carried out to determine the winning advertiser of the impression opportunity.
The winnerwill gain the value of the impression and pay the market price.
See Appendix A.1.1 for detailedexplanations on the structures of the RAS.
Faced with huge amount of impression opportunitiesat every moment, advertisers cannot bid in the granularity of individual opportunities.
Recently,many auto-bidding policies have emerged to realize automatic real-time biddings for each impressionopportunity on behalf of advertisers [6, 5, 3, 2], which signiﬁcantly increase their return on investment(ROI).
State-of-the-art auto-bidding policies usually leverage reinforcement learning (RL) algorithms[6].
It was believed that the auto-bidding policies being updated cannot directly access to the RASduring the training process for safety concerns [2].
A common solution in most existing works[2, 3, 5, 6] is to learn the auto-bidding policy in a virtual advertising system (VAS) which is builtbased on the historical data of advertisers in the RAS.
It is worth noting that the historical data isoften collected merely from stage 2 of the RAS due to computational limitations.
See Appendix
A.1.2 for the details of the VAS structures.
IBOO Challenges However, we argue that this common scheme suffers from the problem ofinconsistencies between online and ofﬂine (IBOO).
Here, "online" refers to the RAS, while "ofﬂine"refers to the VAS.
In brief, the IBOO indicates the gaps between the RAS and VAS.
Formally, the
IBOO can be deﬁned as follows.
Deﬁnition 1 (Inconsistencies Between Online and Ofﬂine) .
The IBOO are deﬁned as the structuraldifferences between the RAS and the VAS, which can degrade the real performance of auto-biddingpolicies trained with the VAS using RL when applied in the RAS.
We next systematically analyze the causes and inﬂuences of the IBOO.
Causes: There are twodominated causes of the IBOO.
One is that the VAS cannot accurately reﬂect the cascade auctionarchitecture of the RAS. Speciﬁcally, the VAS is built only based on the historical data generated instage 2 of the RAS 1 , which makes the VAS lack the mechanism of stage 1 in the RAS.
For example,an advertiser can bid for every impression opportunity in the VAS no matter how its policy changes.
However, in the RAS, an advertiser can lose the impression opportunities that appear in the VAS if itfails to pass the stage 1, or can gain extra impression opportunities that are not in VAS if it passes thestage 1 due to policy improvements, as shown in Fig. 2(a).
In addition, the VAS does not involvethe exact inﬂuences of other advertisers, which makes up the second cause.
For example, in the RAS, the market prices are determined by the bids of all advertisers which can change during thetraining process (even in the situation where all other advertisers’ policies are ﬁxed).
However, the VAS always provides constant market prices for advertisers, as shown in Fig. 2(b).
Inﬂuences of IBOO:
Essentially, the IBOO makes the VAS provides the advertiser with unreal rewards and statetransition functions during RL training process.
Fig. 2(c) displays the returns of the auto-biddingpolicies evaluated in the RAS during training with the VAS and RAS 2 , respectively.
It can be seenthat, due to the IBOO, we can only obtain sub-optimal auto-bidding policies with the VAS.
Related Work To avoid the IBOO, one may consider training auto-bidding policies with traditional
RL algorithms based on the data collected by some behavior policies directly from the RAS.
However,this approach will suffer from extrapolation errors that can seriously degrade the policy’s performancein RAS [12, 32].
As shown in Fig. 2(c), the expected cumulative rewards of traditional RL method[18] training with a ﬁxed data buffer are lower than those of traditional RL method training with RASand VAS. Though we can leverage ofﬂine RL techniques [12, 14, 15] to mitigate this challenge, wecannot guarantee that the collected data contains sufﬁcient transitions from high-reward regions [32]. This will strain the capacity of the ofﬂine RL algorithms to train near-optimal auto-bidding policies. Thus, extra data collections with different behavior policies (presumably better behavior policies)from the RAS are required.
The policies trained by the ofﬂine RL methods cannot be directly usedas behavior policies for data collections in the RAS, since the performance variance of the trainedpolicies can be large [29]. Off policy evaluations (OPEs) have been recently studied for selecting
[Figure] policies with good performance without applying them to real-world environments [13, 38]. However,existing OPEs in auto-bidding are usually conducted in the VAS and can be both time-consuming andinaccurate (See Appendix G.2 for details).
Therefore, we may still have little conﬁdence to widelyapply the policies trained by ofﬂine RL methods to advertisers for further online explorations in the RAS, even though they are proven to perform well by OPE.
Note that there exist some safe online RLmethods for safely exploring in the environments [21, 24, 26, 25, 23, 22].
However, they are eitherdeveloped based on the constraints that are not suitable for the auto-bidding problem or designedfor systems with speciﬁc assumptions.
Recently, with the development of ofﬂine RL methods, manyalgorithms for efﬁcient online explorations on the premise of having an ofﬂine dataset [16, 27] haveemerged.
However, they often focus on the efﬁciency of RL training process rather than the safety ofexplorations.
Detailed related works are presented in Appendix B.
Hence, new schemes need to be devised so as to avoid the IBOO and improve the performance ofauto-bidding policies.
Besides, it is worth noting that the IBOO resembles the sim2real problemstudied in other realms, such as computer visions [9] and robotics [10].
Nonetheless, to the best ofour knowledge, this is the ﬁrst work that systematically analyzes and resolves the IBOO in the ﬁeld
[Figure] of auto-bidding.
Main Contributions
The contribution of this work is fourfold.
Firstly, this is the ﬁrst work toformally deﬁne the IBOO in the auto-bidding ﬁeld and systematically analyze its causes and inﬂuences,to the best of knowledge.
Secondly, we design a safety function and prove its Lipschitz smoothproperty in theory, which provides theoretical foundations for solving the IBOO problem.
Thirdly,based on this property, we propose a sustainable online RL (SORL) framework to continuouslyimprove the auto-bidding polices through direct interactions with the RAS, which can completelyresolve the IBOO problem.
Speciﬁcally, we develop a safe and efﬁcient exploration (SER) policythat can continuously collect data from the RAS.
We also design a variance-suppressed conservative Q-learning (V-CQL) method to improve the auto-bidding policies based on the collected data withlow performance variance.
Furthermore, extensive experiments on both simulated and real-worldadvertising systems validate the effectiveness of our approach.
Problem Formulation
In this paper, we study the auto-bidding problem from the perspective of a single advertiser, whichcan be viewed as an episodic task of T 2 N + time steps.
Between time step t and t + 1 , there are
N t 2 N + impression opportunities, each of which has a positive value v j,t  v M and is sold to the 117 winning advertiser at a market price p j,t  p M , where v M > 0 and p M > 0 are the upper boundsfor values and market prices, respectively, j 2 { 1 , 2 , ..., N t } , t 2 { 1 , 2 , ..., T } .
Let the total budgetof the advertiser be B > 0 .
The auto-bidding problem can be formulated as a Constraint Markov Decision Process (CMDP) [17] < S , A , R, P ,  , C > .
The state s t , [ b t , T  t, B  b t ] 2 S is composed of three elements, including the budget left s t (1) =
b t , the time left s t (2) =
T  t andthe budget consumed s t (3) =
B  b t .
The action a t 2 A , [ A min , A max ] is the bidding price at timestep t , where A max > A min > 0 are the upper and lower bounds, respectively.
The reward r t ( s t , a t )and constraint function c t ( s t , a t ) are the total value of impressions won between time step t andt + 1 and the corresponding costs, while P denotes the state transition rule.
Note that R , C and Pare all directly determined by the RAS.
Moreover,  2 [0 , 1] is the discounted factor.
We denotethe policy as µ : S !
A , and let ⇧ be the policy space.
The goal of the auto-bidding problem isto maximize the total value of impression opportunities earned by the advertiser under the budgetconstraint, which can be expressed as [Equation] subject to
[Equation]
One can leverage standard RL algorithms to learn a state-of-the-art auto-bidding policy [6] with the
VAS, where the constraint in (1) is met by terminating the training episode early once the budgetruns out.
However, as stated before, this way of learning suffers from the challenge of IBOO.
Hence,we ﬁrst theoretically analyze the safety zones during online explorations in the RAS in Section 3and then, in Section 4, propose the SORL framework to solve the auto-bidding problem in an onlinefashion, which can completely avoid the IBOO.
Theory: Lipschitz Safety Function
Unlike the safety functions in other realms such as robotics, it is not appropriate to construct animmediate-evaluated safety function to assess the safety degree of each state-action pair merelybased on their values in auto-bidding.
Actually, any action in any state would be safe as long as thesummation of all subsequent rewards maintains at a high level.
This implies that the safety degree ofa state-action pair is equivalent to its Q value which is highly related to the behaviors in the followingstates.
Hence, we here design the safety function directly as the Q function Q µ ( s t , a t ) .
Proposition 1 (Safety function) .
A proper safety function in auto-bidding should be a function ofstates, actions and policies, i.e., S ⇥
A ⇥ ⇧ !
R , which can directly be the Q function Q µ ( s t , a t ) .
The detailed proofs can be found in Appendix D.2.
We next analyze the characteristics of the safetyfunction.
According to the Bellman equation, we have:
[Equation]
For theoretical analysis, we formulate the analytical expressions of the reward function R , theconstraint function C and the transition rule P .
The proof can be found in Appendix D.1.
Proposition 2 (Analytical expressions of R , C and P ) .
Based on the characteristic of the two-stagecascaded auction in the RAS, we can formulate the reward function R as r t ( s t , a t ) =
P j { a t v 1 j,t p 1 j,t , a t v 2 j,t  p 2 j,t } v j,t , the constraint function C as c t ( s t , a t ) =
P j { a t v 1 j,t  p 1 j,t , a t v 2 j,t p 2 j,t } p j,t , and the state transition rule P as s t +1 =
s t
+ [ 4 s t (1) , 4 s t (2) , 4 s t (3)] , where 4 s t (1) = [Equation]
s
[Equation] } p j,t and 4 s t (2) =  1 .
Note that p 1 j,t and v 1 j,tdenote the market price and rough value of impression j in stage 1, and p 2 j,t and v 2 j,t denote themarket price and accurate value of impression j in stage 2.
We make the following assumption on the impression opportunities, whose rationalities can be foundin Appendix C.1.
Assumption 1 (Bounded Impression Distributions) .
Between time step t and t + 1 , we assume thenumbers of winning impressions with action a t in the ﬁrst stage n t, 1 and the second stage n t, 2 canboth be bounded by linear functions, i.e., n t, 1  k 1 a t , n t, 2  k 2 a t , where k 1 , k 2 > 0 are constants.
We claim that the reward function r t ( s t , a t ) is Lipschitz smooth with respect to actions a t at anygiven state s t .
See Appendix E.1 for the proof.
Theorem 1 (Lipschitz Smooth of r t ( s t , a t ) ) .
Under Assumption 1, the reward function r t ( s t , a t ) is
L r -Lipschitz smooth with respect to actions a t at any given state s t , where L r =
( k 1 + k 2 ) v M .
We make the following mild assumptions, whose rationalities can be found in Appendix C.2.
Assumption 2 (Bounded Partial Derivations of Q µ ( s t , a t ) ) .
We assume that the partial derivation of
Q µ ( s t , a t ) with respect to s t (1) and s t (3) is bounded, i.e.,  @ Q µ ( s t ,a t ) @
s t (1)   k 3 and  @
Q µ ( s t ,a t ) @
s t (3)   k 4 ,where k 3 , k 4 > 0 are constants.
Based on Theorem 1, we can prove that the Q function Q µ ( s t , a t ) is also Lipschitz smooth.
See
Appendix E.2 for proof.
Theorem 2 (Lipschitz Smooth of Q µ ( s t , a t ) ) .
Under Assumption 1 and 2, the Q function Q µ ( s t , a t )is an L Q -Lipschitz smooth function with respect to the actions a t at any given state s t , where
L Q =
[ v M +  ( k 3 + k 4 ) p M ]( k 1 + k 2 ) .
This gives us a way to ﬁnd the safety zone of actions a t at eachstate s t under policy µ with Q µ ( s t , a t ) .
Suppose we have a safepolicy µ s that can direct interact with the RAS (e.g., a state-ofthe-art auto-bidding policy [6] will do).
Then, as shown in Fig.3, a safety zone of actions a t in state s t under policy µ s can bedetermined as [ µ s ( s t )  ⇠ , µ s ( s t ) + ⇠ ] , where ⇠ > 0 is a smallvalue.
Speciﬁcally, if we wish the decrease of safety degrees (i.e.,the Q value) within the safety zone to be no more than a threshold✏ s > 0 , then we can design ⇠ as ⇠  ✏ s L Q .
In addition, we ﬁnd thatthe safety zone of a state is not only effective when the subsequentactions strictly follow the safe policy µ s .
Actually, we can relaxthis constraint in practice for the sake of exploration efﬁciency.
Apractical online exploration policy ⇡ can be: sampling within thesafety zone [ µ s ( s t )  ⇠ , µ s ( s t )+ ⇠ ] in certain  T  1 consecutivetime steps, and sticking to µ s in the rest of T   T time steps. Note that  T 2 N + and  T  T .
We below give the upper bound of the difference between the totalvalue of impression opportunities earned by the advertiser when using ⇡ and µ s , i.e., | V ( ⇡ )  V ( µ s ) | . Theorem 3 (Upper Bound of | V ( ⇡ )  V ( µ s ) | ) . If the exploration policy ⇡ is⇡ ( s t ) = ⇢ sampling from [ µ s ( s t )  ⇠ , µ s ( s t ) + ⇠ ] , t 1  t  t 2 ; µ s ( s t ) , otherwise . 8 t 2 { 0 , 1 , ..., T  1 } , (3) where 0  t 1 < t 2  T  1 and t 1 , t 2 2 N + , then the expected accumulated rewards V ( ⇡ ) satisﬁes
[Equation]
[Equation] (4) where  T = t 2  t 1 + 1 . See Appendix E.3 for the detailed proofs.
As ⇠  ✏ s L Q , we can guarantee that | V ( ⇡ )  V ( µ s ) | ✏ s  t 1  T . Experiment results in Section 5.1 validate the safety of ⇡ .
Algorithm: SORL Framework
We propose a sustainable online RL (SORL) framework for auto-bidding, where the policy is learneddirectly with the RAS.
In the SORL framework, we design a safe and efﬁcient exploration (SER)policy based on the Lipschitz property of the safety function to continuously collect data from the
RAS; and we also develop a variance-suppressed conservative Q-learning (V-CQL) method to learnthe target policies with the collected data.
Fig.
4 shows the whole structure of the SORL framework.
Warm Booting To start with, we need a safe policy µ s to boot the explorations in the RAS.
Previousworks on the auto-bidding problem provide us with a number of such safe policies, many of whichhave already been actually applied to the RAS.
Here, we exploit the state-of-art auto-bidding policy[6], an RL method trained with the VAS, to boot the online explorations.
Denote the data collectedby µ s as D s = { ( s k , a k , r k , s 0 k ) } k .
4. Variance-suppressed Conservative Q-learning
Training the target policy with a ﬁxed dataset D s naturally falls into the realm of ofﬂine RL, wheremany advanced algorithms have been proposed, such as CQL [14], BCQ [12], etc.
As shown laterin Fig.
6, the performance of ofﬂine RL methods initialized with different parameters or hyperparameters can have high variance in practice.
Thus, extra parameter selection work needs to bedone [29].
However, existing OPE methods for auto-bidding problem are not very accurate, andtarget policies with poor performance can affect the efﬁciency of subsequent online explorations (see
Appendix F.2.1 for explanations).
Hence, to decrease the performance variance of target policies, wedevelop the V-CQL method which is a variant of the CQL algorithm.
Speciﬁcally, we implement theregularizer R ( µ ) in the CQL with a novel expression:
[Equation]
distribution of
[Equation]
new
[Equation]
distribution of
[Equation]
[Equation]
where b
Q new is the Q function for the target policy 3 , b
Q old represents the old Q function (here meansthe Q function of µ s ), D KL ( · , · ) represents the KL-divergence, and  > 0 is a constant controlling theweight of R ( µ ) .
To interpret, the regularizer R ( µ ) tries to limit the distance between the distributionof target policy’s Q function and the distribution of the old Q function.
This can limit the derivationtarget policy from old policy µ s .
Due to the Lipschitz smooth property of the safety function, theperformance of target policies can have smaller variance.
Comprehensive explanations of the V-CQLare shown in Appendix F.1.
Denote the obtained target policy using the V-CQL based on D s as µ 0 .
4. Safe and Efﬁcient Online Exploration Policy
To obtain the exploration policy ⇡ , we only need to design the distribution to sample from withinthe safety zone [ µ s ( s t )  ⇠ , µ s ( s t ) + ⇠ ] .
In the following, we use ˜ ⇡ x and ⇡ x to denote the samplingdistribution and the corresponding exploration policy, respectively, where x represents the version.
Note that despite we control the variance of target policy’s performance by the V-CQL, µ 0 is stillnot a good choice for sampling.
On the one hand, µ 0 is a deterministic policy that often lacks ofexploratory characteristics.
On the other hand, its deviation degree from the safe policy µ s cannot beregulated during explorations and cannot be guaranteed to be within ⇠ .
Hence, we need to design anew stochastic exploration policy whose randomness and deviations from µ s can be easily adjusted.
Vanilla Exploration Policy A vanilla sampling method of the safety zone can directly be samplingfrom a (truncated) Gaussian distribution ˜ ⇡ e, N = N ( µ s ( s t ) ,  2 ) with mean µ s ( s t ) and variance  2 .
SER Exploration Policy
However, ˜ ⇡ e, N resembles the policy µ s to a large extent, which makesthe explorations conservative.
To lift up the efﬁciency of explorations, we shift the distribution ˜ ⇡ e, N towards the state-actions pairs that are potentially promising, while constraining its deviations withina threshold ✏ e > 0 .
This can be formulated as a functional optimization problem:
[Equation]
where the optimization variable ˜ ⇡ e is the shifted distribution,  ( s t , a t ) is the potential functionevaluating the potential degrees of state-action pairs.
Note that a vanilla choice for  ( s t , a t ) candirectly be the Q function b
Q new trained with D s using the V-CQL.
Further designs of  ( s t , a t ) areshown in Appendix F.2.1.
Based on Euler equations, we can derive the form of ˜ ⇡ e from (7) as
[Equation]
where C ( s t ) =
[Equation]
acts asthe normalization factor.
The complete deductions of ˜ ⇡ e are givenin Appendix F.2.2.
We can interpret ˜ ⇡ e as a deviation from ˜ ⇡ e, Ntowards the potential function exp { 1   ( s t , a t ) } , where ˜ ⇡ e, N guarantees the safety and exp { 1   ( s t , a t ) } ensures the efﬁciency.
Notethat  and  in (7) are both hyper-parameters that can control therandomness and deviations from µ s of the SER policy, respectively.
As shown in Fig.
5, the smaller the value of  , the largerthe deviations of ˜ ⇡ e from µ s ; besides, the bigger the value of  ,the greater the randomness of ˜ ⇡ e .
Hence, one can control therandomness and deviations from the safe policy µ s of explorationpolicy easily by adjusting the value of  and  4 .
4. Iteratively Updated Structure
The SORL framework is designed to work in an iterative manner, as shown in Fig.
Denotethe SER policy and the target policy in the ⌧ -th iteration as ⇡ e, ⌧ and µ ⌧ , and the data collectedin the ⌧ -th iteration as D on, ⌧ .
A straight design for the sampling distribution ˜ ⇡ e, ⌧ can be ˜ ⇡ e, ⌧ =1 C ⌧ ( s t ) ˜ ⇡
e, N exp {  ⌧ ( s t , a t ) /  ⌧ } , where C ⌧ ( s t ) acts as the normalization factor and  ⌧ is the hyperparameter used in iteration ⌧ .
We note that the proposed exploration policies, including ⇡ e, N , ⇡ e and⇡ e, ⌧ , are all speciﬁc implementations of the exploration policy ⇡ , and the differences between themare only the ways to sample within the safety zone.
Hence, when ⇠  ✏ s L Q , they all satisfy the [-SER policy is modiﬁedto ⇡ e, ⌧ +1 = 1 C 
-]safetyproperty, i.e.,
[Equation]
We apply the V-CQL to iteratively improve the target policy.
Speciﬁcally, at iteration ⌧ , we substituteb
Q old in (5) with b
Q ⌧  1 which is the Q function learned in iteration ⌧  1 , and train a new Q functionb
Q new b
Q ⌧ for the next iteration.
The ﬁnal target policy would be Q ⇤ =
lim ⌧ !
b
[Equation] ⌧ .
A summaryof the SORL framework is present in Appendix F.3.
Experiments
We conduct experiments on both the simulated advertising system and the RAS.
The following threequestions are mainly studied in the experiments: (1) Does the V-CQL help to reduce the performancevariance of the trained policy?
(2) Are the proposed vanilla exploration policy and SER policy safewhen exploring in the RAS (with a safety threshold of ✏ s = 5% V ( µ s ) )?
(3) Can the proposed SORLframework continuously improve the target policy?
Experiment Setup RAS:
We conduct our experiments on one of the world’s largest E-commerceplatforms, TaoBao.
See Appendix G.1 for details.
Evaluation Metrics: We utilize four metrics toevaluate the performance of our approach.
The ﬁrst metric the total consumed budget (ConBdg)of the advertiser.
Note that the ConBdg will not exceed the total budget due to the mechanismsin the advertising system.
Hence, the larger the ConBdg, the more revenue the advertiser canobtain.
The second metric is the ROI which is deﬁned as the ratio between the total revenue and the
ConBdg of the advertiser.
The third metric is the BuyCnt which is the total number of the purchasedimpressions.
The fourth metric is the cost per acquisition (CPA) which is deﬁned as the averagecost for each successfully converted impression.
Clearly, large values of BuyCnt and ROI with asmall CPA indicates a good performance of the auto-bidding policy.
Baselines: Uniﬁed Solutionto Constrained Bidding (USCB) [6] is the state-of-the-art auto-bidding policy trained with RL inthe VAS.
Conservative Q-learning (CQL) [14] is a popular ofﬂine RL method that can mitigateextrapolation errors by learning conservative Q functions.
5. Main Results
[Figure]
To Answer Question (1): We leverage the V-CQL and CQLalgorithm to train a set of the target policies based on the collecteddata with 100 different initial parameters.
To ensure fairness, wealso apply the USCB to train a set of target policies in the RASwith exactly the same 100 initial parameters.
We carry out the
OPE for these three sets of target policies using the VAS wherethe initial safety policy µ s is trained, as shown in Fig.
Notethat the R / R ⇤ is a main metric used in the OPE for auto-bidding(see Appendix G.2).
We can see that the maximum R / R ⇤ ofthe V-CQL is larger than that of the USCB as well as that of theµ s , which indicates that the V-CQL is capable of training a betterpolicy.
Although the maximum R / R ⇤ of the V-CQL and the CQLare about in the same level, the variance of R / R ⇤ of the V-CQL is smaller than that of the CQL.
Thisindicates that the V-CQL can reduce the performance variance compared with CQL when initializingwith different parameters.
For real-world experiments, we apply the three sets of policies trainedby the V-CQL, CQL and USCB to 1 , 600 advertisers for seven days in the RAS, and the averagevalues of the metrics are shown in Table 1.
We can see that the V-CQL outperforms USCB and CQLmethods in almost all metrics.
[Table]
[Table]
To Answer Question (2): As the performance variance of the
V-CQL is still larger than that of the USCB, we need to fullyexamine the safety of the SER policy using different  ( s, a ) corresponding to target policies of different performance levels.
Inthe simulated advertising system, we train seven different versions of target policies with V-CQL method which have variousperformance levels.
We construct seven SER policies with the potential functions {  i ( s, a ) } 7 i =1 being the Q functions of the targetpolicies.
We also construct a vanilla exploration policy ⇡ e, N forcomparison.
The hyper-parameters for all exploration policies are = 1 ,  = 0 .
We apply the ⇡ e, N and seven SER policies tothe simulated advertising system, and the ROIs of explorations policies are shown in Fig.
We can see that the ROI of the SER policy rises with the performancelevel of the target policy trained by the V-CQL.
The worst ROI of the SER policy drops about 3 .
52%compared with the initial safety policy, which is smaller than the safety threshold ✏ s = 5% , as wellas the decline rate in ROI of the corresponding target policy 6 .
This indicates the safety ofthe SER policy.
Moreover, we can control the safety degree of the SER policy by regulating thehyper-parameters  and  , and the results are shown in Appendix G.3.2.
[Table]
For real-world experiments, we apply ⇡ e and ⇡ e, N to 325 1 , 300 advertisers in the RAS and compare the performance with the initial safety policy µ s , as shown in
Table.
We can see that the varieties of the ConBdgand BuyCnt of ⇡ e are all with in the safety threshold5% , and are better than those of ⇡ e, N , which indicatesthe safety of ⇡ e .
[Table]
.13% 0.33% 1.60% 3.57% -0.98% 2.55% 3 -th
µ3.20% 0.95% -1.01% 0.06% 2.48% 3.75% -3.91% -0.15%
[Figure]
To Answer Question (3):
Weﬁrst apply the SORL algorithmin the simulated advertising system, and the results during iterations are shown in Fig. 8.
Speciﬁcally, from Fig.
8(a),we can see that the decline ratein ROI of both SER policiesand the vanilla exploration policies are smaller than ✏ s = 5% ,which implies the safety of theproposed exploration policies.
Moreover, the ROI of the SERpolicy ⇡ e, ⌧ rises with iterations and is alway higher than that of ⇡ e, N , which indicates that the SERpolicy is safer.
From Fig.
8(b), we can see that the ROI of the target policy µ ⌧ trained with datacollected by ⇡ e, ⌧ rises with the number of iterations and converges to the optimal ROI (i.e., the ROI ofthe optimal policy) at the 5 -th iteration.
This indicates the efﬁciency of the SER policy.
For real-worldexperiments, we utilize 10 , 000 advertisers to collect data from the RAS with ⇡ e and compare thetarget policies in 4 iterations on 1 , 500 advertisers using A/B tests, and the results are shown in Table.3.
We can see that the performances of target policies are getting better with iterations and exceeds thestate-of-the-art auto-bidding policy USCB, which validates the effectiveness of the SORL framework.
Conclusions
In this paper, we deﬁne the IBOO in auto-bidding problems and systematically analyze its manifestations, causes and inﬂuences.
Then, we design a safety function to identify the safety zonesin auto-bidding and prove its Lipschitz smooth property, which provides theoretical guarantees forsolving the IBOO problem.
Based on this property, we propose a SORL framework to directly learnauto-bidding policies with the RAS.
The SORL works in an iterative way, where the SER policy isdesigned to safely and efﬁciently collect data from the RAS, and the V-CQL is designed to continuously improve the target auto-bidding policies.
Finally, extensive experiments on both simulated andreal-world advertising systems validate the effectiveness of the proposed SORL framework.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For authors.
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We believe there is no negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 3and Appendix C.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix D andAppendix E.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] The codes are
proprietary. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes](c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
Our work uses own
data, and the data used in the simulation experiments and real-world experiments isreleased in supplementarymaterial.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
N/A] (
d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]