Sustainable Online Reinforcement Learning for Auto-bidding
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Recently, auto-bidding technique has become an essential tool to increase thereturn on investment (ROI) for advertisers.
Facing the complex and ever-changingbidding environments in the real-world advertising system (RAS), state-of-the-artauto-bidding policies usually leverage reinforcement learning (RL) algorithms togenerate real-time bids on behalf of the advertisers.
Due to safety concerns, itwas believed that the RL process can only be carried out in a virtual advertisingsystem (VAS) which is usually built based on the historical data generated in the
RAS.
In this paper, we argue that there exists signiﬁcant gaps between VAS and
RAS, making the RL process in the VAS suffer from the problem of inconsistencybetween online and ofﬂine (IBOO).
Firstly, we formally deﬁne the IBOO and systematically analyze its causes and inﬂuences.
Then, we design a safety function and prove its Lipschitz smooth property in theory, which provides theoreticalfoundations for solving the IBOO problem.
Moreover, based on this property, we propose a sustainable online RL (SORL) framework to continuously improvethe auto-bidding polices through direct interactions with the RAS, which cancompletely resolve the IBOO problem.
Speciﬁcally, we devise a safe and efﬁcientonline exploration policy that can continuously collect data from the RAS.
Wealso develop a variance-suppressed conservative Q-learning method to improvethe auto-bidding policies with the collected data.
Extensive experiments on bothsimulated and real-world advertising systems validate the effectiveness of ourapproach.
Introduction
In the era of Internet, online advertising business has become one ofthe main proﬁt models for many companies, such as Google [7] and
Alibaba [4], which, at the same time, beneﬁts millions of advertiserswho are willing to bid for impression opportunities.
Contemporaryonline advertising systems, as auctioneers, usually have large amount ofcandidate advertisers contesting for numerous impression opportunitiesat every moment.
Making auction decisions based on accurate evaluation of each impression opportunity for all advertisers within severalmilli-seconds is computationally infeasible.
Therefore, a real-worldadvertising system (RAS) adopts a cascade architecture.
In this architecture, the auction of each impression opportunity is completed throughseveral stages.
Without loss of generality, we here simply view the RASas a system with two stages: stage 1 and stage 2, as shown in Fig.
Theauction process of each impression opportunity is described as follows: in stage 1, rough evaluations are conducted for all the candidate advertisers ( ⇠ 10 6 ), and a group ofthe most promising advertisers ( ⇠ 10 2 ) are fed to the stage 2; and in stage 2, accurate valuations andauction are carried out to determine the winning advertiser of the impression opportunity.
The winnerwill gain the value of the impression and pay the market price.
Faced with huge amount of impressionopportunities at every moment, advertisers cannot bid in the granularity of individual opportunities.
Recently, many auto-bidding policies have emerged to realize automatic real-time biddings for eachimpression opportunity on behalf of advertisers [6, 5, 3, 2], which have signiﬁcantly increase theirreturn on investment (ROI).
State-of-the-art auto-bidding policies usually leverage reinforcementlearning (RL) algorithms [6].
It was believed that the auto-bidding policies being updated cannotdirectly access to the RAS during the training process for safety concerns [2].
A common solutionin most existing works [2, 3, 5, 6] is to learn the auto-bidding policy in a virtual advertising system(VAS) which is built based on the historical data of advertisers in the RAS.
It is worth noting that thehistorical data is often collected merely from stage 2 of the RAS due to computational limitations.
IBOO Challenges However, we argue that this common scheme suffers from the problem ofinconsistencies between online and ofﬂine (IBOO).
Here, "online" refers to the RAS, while "ofﬂine"refers to the VAS.
In brief, the IBOO indicates the gaps between the RAS and VAS.
Formally, the
IBOO can be deﬁned as follows.
Deﬁnition 1 (Inconsistencies Between Online and Ofﬂine) .
The IBOO are deﬁned as the structuraldifferences between the RAS and the VAS, which can degrade the real performance of auto-biddingpolicies trained with the VAS using RL when applied in the RAS.
We next systematically analyze the causes and inﬂuences of the IBOO.
Causes: There are twodominated causes of the IBOO.
One is that the VAS is built based on the historical data of stage 2that cannot accurately reﬂect the cascade auction architecture of the RAS.
For example, an advertisercan bid for every impression opportunity in the VAS no matter how its policy changes during training.
However, in the RAS, an advertiser can lose the impression opportunities that appear in the VAS if itfails to pass the stage 1, or can gain extra impression opportunities that are not in VAS if it passesthe stage 1 due to policy improvements, as shown in Fig. 2(a).
In addition, the bidding involving alladvertisers naturally falls into the realm of multi-agent RL, where each single advertiser is affectedby the policies of all other advertisers.
However, the VAS does not involve the exact inﬂuences ofother advertisers during RL training, which makes up the second cause.
For example, in the RAS,the market prices are determined by the bids of all advertisers which can change during the trainingprocess (even in the situation where all other advertisers’ policies are ﬁxed).
However, the VASalways provides constant market prices for advertisers, as shown in Fig. 2(b).
Inﬂuences of IBOO:
Essentially, the IBOO makes the VAS provides the advertiser with unreal rewards and state transitionfunctions during RL training process.
Fig. 2(c) displays the returns of the auto-bidding policiesevaluated in the RAS during training with the VAS and RAS 1 , respectively.
It can be seen that, due tothe IBOO, we can only obtain sub-optimal auto-bidding policies when training with the VAS.
Related Work To avoid the IBOO, one may consider training auto-bidding policies with traditional
RL algorithms based on the data collected by a certain behavior policy directly from the RAS.
However, this method will face extrapolation errors that can seriously degrade the policy performance[12].
The returns of training with a ﬁxed data buffer shown in Fig. 2(c) testify this problem.
Thoughwe can leverage ofﬂine RL techniques [12, 14, 15] to mitigate it, the performance variance of thetrained policy can be large [29] and further off policy evaluations (OPE) are required [13].
However,the OPE in auto-bidding needs to be manually designed based on the characteristics of the RAS, whichcan be time consuming and inaccurate.
Besides, we may still have little conﬁdence to widely applythe learned policies to advertisers for further online explorations in the RAS, even though they areproven to perform well by OPE.
Note that there exist some safe online RL methods to explore in theenvironments with safety guarantees [21, 24, 26, 25, 23, 22].
Some popular algorithms [21, 23, 22]are built based on Constraint Markov Decision Process (CMDP) [17] where the constraint functionnaturally exists under each state-action pair.
However, there are no such off-the-shelf constraintfunctions in the environment of the RAS.
Other approaches are often designed based on systems with [Figure] speciﬁc models and extra assumptions, which are not suitable for the auto-bidding problem.
Recently,with the development of the ofﬂine RL ﬁeld, many algorithms for efﬁcient online explorations onthe premise of having an ofﬂine dataset [16, 27] have emerged.
However, they often focus on theefﬁciency of RL training process rather than the safety of it.
Hence, new schemes need to be devisedso as to avoid the IBOO and improve the performance of auto-bidding policies.
Besides, it is worthnoting that the IBOO resembles the sim2real problems studied in other realms, such as computervisions [9] and robotics [10].
Nonetheless, to the best of our knowledge, this is the ﬁrst work thatsystematically analyzes and resolves the IBOO in the ﬁeld of auto-bidding.
Comprehensive relatedworks are presented in Appendix B.
Main Contributions
The contributions of this work are in four-fold.
Firstly, we formally deﬁnethe IBOO in auto-bidding and systematically analyze its causes and inﬂuences.
Secondly, we designa safety function and prove its Lipschitz smooth property in theory, which provides theoreticalfoundations for solving the IBOO problem.
Thirdly, based on this property, we propose a sustainableonline RL (SORL) framework to continuously improve the auto-bidding polices through directinteractions with the RAS, which can completely resolve the IBOO problem.
Speciﬁcally, we developa safe and efﬁcient exploration (SER) policy that can continuously collect data from the RAS.
Wealso design a variance-suppressed conservative Q-learning (V-CQL) method to improve the autobidding policies based on the collected data with low performance variance.
Furthermore, extensiveexperiments on both simulated and real-world advertising systems validate the effectiveness of ourapproach.
Problem Formulation
In this paper, we study the auto-bidding problem from the perspective of a single advertiser, whichcan be viewed as an episodic task of T 2 N + time steps.
Between time step t and t + 1 , there are
N t 2 N + impression opportunities, each of which has a positive value v j,t  v M and is sold to the 111 winning advertiser at a market price p j,t  p M , where v M > 0 and p M > 0 are the upper bounds forvalues and market prices, respectively, j 2 { 1 , 2 , ..., N t } , t 2 { 1 , 2 , ..., T } .
Let the total budget of theadvertiser be B > 0 .
The auto-bidding problem can be formulated as a CMDP < S , A , R, P ,  , C > .
The state s t , [ b t , T  t, B  b t ] 2 S is composed of three elements, including the budget left 115 s t (1)
= b t , the time left s t (2) =
T  t and the budget consumed s t (3) =
B  b t .
The actiona t 2 A , [ A min , A max ] is the bidding price at time step t , where A max > A min > 0 are the upperand lower bounds, respectively.
The reward r t ( s t , a t ) and constraint function c t ( s t , a t ) are the totalvalue of impressions won between time step t and t + 1 and the corresponding costs, while P denotesthe state transition rule.
Note that R , C and P are all directly determined by the RAS.
Moreover, 2 [0 , 1] is the discounted factor.
We denote the policy as µ : S !
A , and let ⇧ be the policy space.
The goal of the auto-bidding problem is to maximize the total value of impression opportunitiesearned by the advertiser under the budget constraint, which can be expressed as
[Equation]
One can leverage standard RL algorithms to learn a state-of-the-art auto-bidding policy [6] with the
VAS, where the constraint in (1) is met by terminating the training episode early once the budgetruns out.
However, as stated before, this way of learning suffers from the challenge of IBOO.
Hence,we ﬁrst theoretically analyze the safety zones during online explorations in the RAS in Section 3and then, in Section 4, propose the SORL framework to solve the auto-bidding problem in an onlinefashion, which can completely avoid the IBOO.
Theory: Lipschitz Safety Function
Unlike the safety functions in other realms such as robotics, it is not appropriate to construct animmediate-evaluated safety function to assess the safety degree of each state-action pair merelybased on their values in auto-bidding.
Actually, any action in any state would be safe as long as thesummation of all subsequent rewards maintains at a high level.
This implies that the safety degree ofa state-action pair is equivalent to its Q value which is highly related to the behaviors in the followingstates.
Hence, we here design the safety function directly as the Q function Q µ ( s t , a t ) .
Proposition 1 (Safety function) .
A proper safety function in auto-bidding should be a function ofstates, actions and policies, i.e., S ⇥
A ⇥ ⇧ !
R , which can directly be the Q function Q µ ( s t , a t ) .
The detailed proofs can be found in Appendix D.2.
Remind that the safety function can help determinewhich action is safer in the same state under the same policy, while comparing the value of the safetyfunction between actions in distinct states or policies would be meaningless.
We next analyze thecharacteristics of the safety function.
According to the Bellman equation, we have:
[Equation]
For theoretical analysis, we formulate the analytical expressions of the reward function R , theconstraint function C and the transition rule P .
The proof can be found in Appendix D.1.
Proposition 2 (Analytical expressions of R , C and P ) .
Based on the characteristic of the two-stagecascaded auction in the RAS, we can formulate the reward function R as r t ( s t , a t ) =
P j { a t v 1 j,t p 1 j,t , a t v 2 j,t  p 2 j,t } v j,t , the constraint function C as c t ( s t , a t ) =
P j { a t v 1 j,t  p 1 j,t , a t v 2 j,t p 2 j,t } p j,t , and the state transition rule P as s t +1 =
s t
+ [ 4 s t (1) , 4 s t (2) , 4 s t (3)] , where 4 s t (1) =4 s t (3) =
P j { a t v 1 j,t  p 1 j,t , a t v 2 j,t  p 2 j,t } p j,t and 4 s t (2) =  1 .
Note that p 1 j,t and v 1 j,tdenote the market price and rough value of impression j in stage 1, and p 2 j,t and v 2 j,t denote themarket price and accurate value of impression j in stage 2.
We make the following assumption on the impression opportunities, whose rationalities can be foundin Appendix C.1.
Assumption 1 (Bounded Impression Distributions) .
Between time step t and t + 1 , we assume thenumbers of winning impressions with action a t in the ﬁrst stage n t, 1 and the second stage n t, 2 canboth be bounded by linear functions, i.e., n t, 1  k 1 a t , n t, 2  k 2 a t , where k 1 , k 2 > 0 are constants.
We claim that the reward function r t ( s t , a t ) is Lipschitz smooth with respect to actions a t at anygiven state s t .
See Appendix E.1 for the proof.
Theorem 1 (Lipschitz Smooth of r t ( s t , a t ) ) .
Under Assumption 1, the reward function r t ( s t , a t ) is
L r -Lipschitz smooth with respect to actions a t at any given state s t , where L r =
( k 1 + k 2 ) v M .
We make the following mild assumptions, whose rationalities can be found in Appendix C.2.
Assumption 2 (Bounded Partial Derivations of Q µ ( s t , a t ) ) .
We assume that the partial derivation of
Q µ ( s t , a t ) with respect to s t (1) and s t (3) is bounded, i.e.,  @ Q µ ( s t ,a t ) @
s t (1)   k 3 and  @
Q µ ( s t ,a t ) @
s t (3)   k 4 ,where k 3 , k 4 > 0 are constants.
Based on Theorem 1, we can prove that the Q function Q µ ( s t , a t ) is also Lipschitz smooth.
See
Appendix E.2 for proof.
Theorem 2 (Lipschitz Smooth of Q µ ( s t , a t ) ) .
Under Assumption 1 and 2, the Q function Q µ ( s t , a t )is an L Q -Lipschitz smooth function with respect to the actions a t at any given state s t , where
L Q =
[ v M + ( k 3 + k 4 ) p M ]( k 1 + k 2 ) .
can be determined as [
[Equation] where [Equation] This gives us a way to ﬁnd the safety zone of actions a t at eachstate s t under policy µ with Q µ ( s, a ) .
Suppose we have a safepolicy µ s that can direct interact with the RAS (e.g., a state-ofthe-art auto-bidding policy [6] will do).
Then, as shown in Fig.3, a safety zone of actions a t in state s t under policy µ s can bedetermined as [ µ s ( s )  ⇠ , µ s ( s )+ ⇠ ] , where ⇠ > 0 is a small value.
Speciﬁcally, if we wish the decrease of safety degrees (i.e., the Q value) within the safety zone to be no more than a threshold✏ s > 0 , then we can design ⇠ as [Equation]
In addition, we remind that, theoretically, the safety zone of astate is effective only when the subsequent actions strictly followthe safe policy µ s until the end of the episode.
Nonetheless, wecan relax this constraint in practice for the sake of explorationefﬁciency.
For example, we can generate actions in several (even all) states by not followingthe safe policy µ s , but by sampling within the safety zones.
Experiment results in Section 5.1demonstrate the feasibility of such operations.
See Appendix F.2.1 for further illustrations onpractical implementations.
We highlight that the online explorations in the RAS would be possible ifwe always stay within the safety zones.
Algorithm: SORL Framework
We propose a sustainable online RL (SORL) framework for auto-bidding, where the policy is learneddirectly with the RAS.
In the SORL framework, we design a safe and efﬁcient exploration (SER)policy based on the Lipschitz property of the safety function to continuously collect data from the
RAS; and we also develop a variance-suppressed conservative Q-learning (V-CQL) method to learnthe target policies with the collected data.
Fig.
4 shows the whole structure of the SORL framework.
Warm Booting To start with, we need a safe policy µ s to boot the explorations in the RAS.
Previousworks on the auto-bidding problem provide us with a number of such safe policies, many of whichhave already been actually applied to the RAS.
Here, we exploit the state-of-art auto-bidding policy[6], an RL method trained with the VAS, to boot the online explorations.
Denote the data collectedby µ s as D s = { ( s k , a k , r k , s 0 k ) } k .
4. Variance-suppressed Conservative Q-learning
Training the target policy with a ﬁxed dataset D s naturally falls into the realm of ofﬂine RL, wheremany advanced algorithms have been proposed, such as CQL [14], BCQ [12], etc.
As shown laterin Fig.
6, the performance of ofﬂine RL methods initialized with different parameters or hyperparameters can have high variance in practice.
Thus, extra parameter selection work needs to bedone [29].
However, existing OPE methods for auto-bidding problem are not very accurate, andtarget policies with poor performance can affect the efﬁciency of subsequent online explorations (see
Appendix F.2.2 for explanations).
Hence, to decrease the performance variance of target policies, wedevelop the V-CQL method which is a variant of the CQL algorithm.
Speciﬁcally, we implement theregularizer R ( µ ) in the CQL with a novel expression:
[Equation]
distribution of
[Equation]
distribution of
[Equation]
where b
Q new is the Q function for the target policy 2 , b
Q old represents the old Q function (here meansthe Q function of µ s ), D KL ( · , · ) represents the KL-divergence, and  > 0 is a constant controlling theweight of R ( µ ) .
To interpret, the regularizer R ( µ ) tries to limit the distance between the distributionof target policy’s Q function and the distribution of the old Q function.
This can limit the derivationtarget policy from old policy µ s .
Due to the Lipschitz smooth property of the safety function, theperformance of target policies can have smaller variance.
See Appendix F.1 for detailed illustrations.
We denote the obtained target policy using the V-CQL based on D s as µ 0 .
4. Safe and Efﬁcient Online Exploration Policy
Despite we control the variance of target policy’s performance by the V-CQL, µ 0 is still not a goodchoice for further online explorations.
On the one hand, µ 0 is a deterministic policy that is oftenlack of exploratory characteristics.
On the other hand, its derivation degree from the safe policy µ scannot be regulated during explorations.
Hence, we need to design a new stochastic explorationpolicy whose randomness and derivations from µ s can be easily adjusted.
Vanilla Exploration Policy Due to the Lipschitz smooth property of the safety function, a vanillaonline exploration policy can directly be a Gaussian stochastic policy ⇡ e, N with mean of safety actionµ s ( s t ) and variance of  2 , i.e., [Equation] Based on the three-sigma rule, we can let  = 13 ⇠  ✏ s 3 L Q to guarantee the safety of explorations.
SER Exploration Policy
However, ⇡ e, N resembles the policy µ s to a large extent, which makesthe explorations conservative.
To lift up the efﬁciency of explorations, we shift the distribution of⇡ e, N towards the state-actions pairs that are potentially promising, while constraining its derivationswithin a threshold ✏ e > 0 .
This can be formulated as a functional optimization problem:
[Equation]
(6) where the optimization variable ⇡ e is the SER policy,  ( s t , a t ) is the potential function evaluatingthe potential degrees of state-action pairs.
Note that a vanilla choice for  ( s t , a t ) can directly be the Q function b
Q new trained on D s with the V-CQL.
Further designs of  ( s t , a t ) are shown in Appendix F.2.2.
Based on Euler equations, we can derive the form of ⇡ e from (7) as
[Equation]
where C ( s t ) =
[Equation]
{
[Equation]
[Equation]
s [Equation] [Equation] ) } d a tacts as the normalization factor.
The complete deductions of ⇡ eare given in Appendix F.2.3.
We can interpret ⇡ e as a deviationfrom ⇡ e, N towards the potential function exp { 1   ( s t , a t ) } , where⇡ e, N guarantees the safety and exp { 1   ( s t , a t ) } ensures the efﬁ- 238 ciency.
Note that  and  in (7) are both hyper-parameters thatcan control the randomness and derivations from µ s of the SERpolicy, respectively.
As shown in Fig.
5, the smaller the valueof  , the larger the derivations of ⇡ e from µ s ; besides, the biggerthe value of  , the greater the randomness of ⇡ e .
Hence, one cancontrol the randomness and derivations from the safe policy µ s ofexploration policy easily by adjusting the value of  and  .
Wenote that for safety concerns, we should keep   ✏ s 3 L Q .
A way toimplement the SER policy ⇡ e in practice is described in Appendix F.2.4.
4. Iteratively Updated Structure
The SORL framework is designed to work in an iterative manner, as shown in Fig.
Speciﬁcally, ateach iteration ⌧ 2 N , the SER policy collects data from the RAS, and both the target policy and the
SER policy are updated based on the collected data for next iteration.
Denote the SER policy and thetarget policy in the ⌧ -th iteration as ⇡ e, ⌧ and µ ⌧ , and the data collected in the ⌧ -th iteration as D on, ⌧ .
A straight design for ⇡ e, ⌧ +1 can be the policy in (7) where ⇡ e, N is substitute by ⇡ e, ⌧ , i.e.,⇡ e, ⌧ + ⇡ e
[Equation] where C ⌧ +1 ( s t ) acts as the normalization factor,  ⌧ is the hyper-parameter used in iteration ⌧ , and⇡ e, 0 = ⇡ e, N .
However, (8) requires us to store all the previous potential functions to calculatethe current SER policy, which is costly. Hence, we omit the history potential functions {  i } ⌧ i =1and gradually decrease  ⌧ to zero, i.e.,  1 >  2 > · · ·  ⌧ > 0 .
Thus, the {+SER policy is modiﬁedto ⇡ e, ⌧ +1 = 1 C 
+}
⌧ +1 ( s t ) ⇡ e, N exp {  ⌧ +1 ( s t , a t ) /  ⌧ +1 } .
We apply the V-CQL to iteratively improvethe target policy.
Speciﬁcally, at iteration ⌧ , we substitute b
Q old in (4) with b
Q ⌧  1 which is the Qfunction learned in iteration ⌧  1 , and train a new Q function b
Q new b
Q ⌧ for the next iteration.
Theﬁnal target policy would be Q ⇤ =
lim ⌧ !
b
Q ⌧ .
A summary of the SORL framework is present in Appendix F.3.
Experiments
We conduct experiments on both the simulated advertising system and the RAS.
The following threequestions are mainly studied in the experiments: (1) Does the V-CQL help to reduce the performancevariance of the trained policy?
(2) Are the proposed vanilla exploration policy and SER policy safewhen exploring in the RAS (with a safety threshold of ✏ s = 5% )?
(3) Can the proposed SORLframework continuously improve the target policy?
Experiment Setup RAS:
We conduct our experiments on one of the world’s largest E-commerceplatforms, TaoBao.
See Appendix G.1 for details.
Evaluation Metrics: We utilize four metrics toevaluate the performance of our approach.
The ﬁrst metric the total consumed budget (ConBdg)of the advertiser.
Note that the ConBdg will not exceed the total budget due to the mechanismsin the advertising system.
Hence, the larger the ConBdg, the more revenue the advertiser canobtain.
The second metric is the ROI which is deﬁned as the ratio between the total revenue and the
ConBdg of the advertiser.
The third metric is the BuyCnt which is the total number of the purchasedimpressions.
The fourth metric is the cost per acquisition (CPA) which is deﬁned as the averagecost for each successfully converted impression.
Clearly, large values of BuyCnt and ROI with a small CPA indicates a good performance of the auto-bidding policy.
Baselines: Uniﬁed Solutionto Constrained Bidding (USCB) [6] is the state-of-the-art auto-bidding policy which outperformsprevious algorithms in terms of both effectiveness and stability. Speciﬁcally, USCB is a RL methodtrained in the VAS.
Conservative Q-learning (CQL) [14] is a popular ofﬂine RL method that canmitigate extrapolation errors by learning conservative Q functions.
The initial safety policy µ s forbooting the explorations in the RAS is trained by the USCB.
5. Main Results
[Figure]
To Answer Question (1): We leverage the V-CQL and CQLalgorithm to train a set of the target policies based on the collecteddata with 100 different initial parameters.
To ensure fairness, wealso apply the USCB to train a set of target policies in the RASwith exactly the same 100 initial parameters.
We carry out the
OPE for these three sets of target policies using the VAS wherethe initial safety policy µ s is trained, as shown in Fig.
Notethat the R / R ⇤ is a main metric used in the OPE for auto-bidding(see Appendix G.2).
We can see that the maximum R / R ⇤ ofthe V-CQL is larger than that of the USCB as well as that of theµ s , which indicates that the V-CQL is capable of training a betterpolicy.
Although the maximum R / R ⇤ of the V-CQL and the CQL are about in the same level, thevariance of R / R ⇤ of the V-CQL is smaller than that of the CQL.
This indicates that the V-CQL canreduce the performance variance compared with CQL when initializing with different parameters.
Forreal-world experiments, we apply the three sets of policies trained by the V-CQL, CQL and USCBto 1 , 600 advertisers for seven days in the RAS, and the average values of the metrics are shown in Table 1.
We can see that the V-CQL outperforms USCB and CQL methods in almost all metrics.
[Table]
policies by CQL
To Answer Question (2): As the performance variance of the
V-CQL is still larger than that of the USCB, we need to fullyexamine the safety of the SER policy using different  ( s, a ) corresponding to target policies of different performance levels.
Inthe simulated advertising system, we train seven different versions of target policies with V-CQL method which have variousperformance levels.
We construct seven SER policies with the potential functions {  i ( s, a ) } 7 i =1 being the Q functions of the targetpolicies.
We also construct a vanilla exploration policy ⇡ e, N forcomparison.
The hyper-parameters for all exploration policies are = 1 ,  = 0 .
We apply the ⇡ e, N and seven SER policies tothe simulated advertising system, and the ROIs of explorationspolicies are shown in Fig.
We can see that the ROI of the SERpolicy rises with the performance level of the target policy trainedby the V-CQL.
The worst ROI of the SER policy drops about 3 .
52% compared with the initial safetypolicy, which is smaller than the safety threshold ✏ s = 5% , as well as the decline rate in ROI of thecorresponding target policy 6 .
This indicates the safety of the SER policy.
Moreover, we cancontrol the safety degree of the SER policy by regulating the hyper-parameters  and  , and theresults are shown in Appendix G.3.2.
In addition, we can see that the ROI of SER policy is a higherthan the vanilla policy when the performance of target policies is better than the initial safety policy(this is why we propose the V-CQL method to reduce the performance variance of CQL).
[Table]
For real-world experiments, we apply ⇡ e and ⇡ e, N to 323 1 , 300 advertisers in the RAS and compare the performance with the initial safety policy µ s , as shown in
Table.
We can see that the varieties of the ConBdg and BuyCnt of ⇡ e are all with in the safety threshold5% , and are better than those of ⇡ e, N , which indicatesthe safety of ⇡ e .
[Table]
[Figure]
To Answer Question (3):
Weﬁrst apply the SORL algorithmin the simulated advertising system, and the results during iterations are shown in Fig. 8.
Speciﬁcally, from Fig.
8(a),we can see that the decline ratein ROI of both SER policiesand the vanilla exploration policies are smaller than ✏ s = 5% ,which implies the safety of theproposed exploration policies.
Moreover, the ROI of the SERpolicy ⇡ e, ⌧ rises with iterationsand is alway higher than that of ⇡ e, N , which indicates that the SER policy is safer.
From Fig.
8(b),we can see that the ROI of the target policy µ ⌧ trained with data collected by ⇡ e, ⌧ rises with thenumber of iterations and converges to the optimal ROI (i.e., the ROI of the policy trained directly inthe simulated advertising system) at the 5 -th iteration.
In addition, the ROI of target policy trainedwith data collected by ⇡ e, ⌧ is higher than the ROI of the target policy trained with data collectedby ⇡ e, N , as well as the ROI of the policy trained with the VAS.
This indicates the efﬁciency of the SER policy.
For real-world experiments, we utilize 10 , 000 advertisers to collect data from the RASwith ⇡ e and compare the target policies in 4 iterations on 1 , 500 advertisers using A/B tests, andthe results are shown in Table.
We can see that the performances of target policies are gettingbetter with iterations and exceeds the state-of-the-art auto-bidding policy USCB, which validates theeffectiveness of the SORL framework.
Conclusions
In this paper, we deﬁne the IBOO in auto-bidding problems and systematically analyze its manifestations, causes and inﬂuences.
Then, we design a safety function to identify the safety zonesin auto-bidding and prove its Lipschitz smooth property, which provides theoretical guarantees forsolving the IBOO problem.
Based on this property, we propose a SORL framework to directlylearn auto-bidding policies with the RAS.
The SORL works in an iterative way, where the SERpolicy is designed to safely and efﬁciently collect data from the RAS, and the V-CQL is designedto continuously improve the target auto-bidding policies.
Finally, extensive experiments on boththe simulated advertising system and the RAS validates the effectiveness of the proposed SORLframework. A promising future work can be extending the SORL framework to all online advertisers.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] We believethere is no negative societal impacts.(d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section and Appendix C.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix D andAppendix E.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
Our work uses own
data, and the data used in the simulation experiments and real-world experiments is released in supplementarymaterial.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A]
[Table]