Sustainable Online Reinforcement Learning for Auto-bidding
Abstract
Recently, auto-bidding technique has become an essential tool to increase the revenue of advertisers.
Facing the complex and ever-changing bidding environments in the real-world advertising system (RAS), state-of-the-art auto-bidding policies usually leverage reinforcement learning (RL) algorithms to generate real-time bids on behalf of the advertisers.
Due to safety concerns, it was believed that the RL training process can only be carried out in an ofﬂine virtual advertising system (VAS) that is built based on the historical data generated in the RAS.
In this paper, we argue that there exists signiﬁcant gaps between the VAS and RAS, making the
RL training process suffer from the problem of inconsistency between online and ofﬂine (IBOO).
Firstly, we formally deﬁne the IBOO and systematically analyze its causes and inﬂuences.
Then, to avoid the IBOO, we propose a sustainable online
RL (SORL) framework that trains the auto-bidding policy by directly interacting with the RAS, instead of learning in the VAS.
Speciﬁcally, based on our proof of the Lipschitz smooth property of the Q function, we design a safe and efﬁcient online exploration (SER) policy for continuously collecting data from the RAS.
Meanwhile, we derive the theoretical lower bound on the safety of the SER policy.
We also develop a variance-suppressed conservative Q-learning (V-CQL) method to effectively and stably learn the auto-bidding policy with the collected data.
Finally, extensive simulated and real-world experiments validate the superiority of our approach over the state-of-the-art auto-bidding algorithm.
Introduction
In the era of Internet, online advertising business has become one of the main proﬁt models for many companies, such as Google [6] and Alibaba [3], which, at the same time, beneﬁts millions of advertisers who are willing to bid for impression opportunities.
Contemporary online advertising systems, as auctioneers, usually have large amount of candidate advertisers contesting for numerous impression opportunities at every moment.
Making auction decisions based on accurate evaluation of each impression opportunity for all advertisers within several milli-seconds is computationally infeasible.
Therefore, a real-world advertising system (RAS) adopts a cascade architecture [7, 10].
In this architecture, the auction of each impression opportunity is completed through several stages.
Without loss of generality, we here simply view the RAS as a system with two stages: stage 1 and stage 2, as shown in Fig.
The auction process of each impression opportunity is as follows: in stage 1, rough evaluations are conducted for all the candidate advertisers ( ⇠ 10 6 ), and a group of the most promising advertisers ( ⇠ 10 2 ) are fed to the stage 2; and in stage 2, accurate valuations and auction are carried out to determine the winning advertiser of the impression opportunity.
The winner will gain the value of the impression opportunity and pay the market price.
See
Appendix A.1.1 for detailed explanations on the RAS structures.
Faced with huge amount of impression opportunities at every moment, advertisers cannot bid in the granularity of individual opportunities.
Recently, many auto-bidding policies have emerged to realize automatic real-time biddings for each impression opportunity on behalf of advertisers [1, 2, 4, 5], which signiﬁcantly increase their revenues.
State-of-the-art auto-bidding policies are usually learned with reinforcement learning (RL) algorithms [5].
It was believed that the auto-bidding policy being trained cannot directly access to the RAS during the
RL training process for safety concerns [1].
A common solution in most existing works [1, 2, 4, 5] is to train the auto-bidding policy in a virtual advertising system (VAS) — an ofﬂine simulated environment that is built based on the advertisers’ historical data generated in the RAS.
See Appendix
A.1.2 for the details of the VAS structures.
IBOO Challenges.
However, we argue that there exists gaps between the RAS and the VAS, which makes this common solution suffer from the problem of inconsistencies between online and ofﬂine (IBOO).
Here, online refers to the RAS, while ofﬂine refers to the VAS, and inconsistencies refer to the gaps.
Formally, we can deﬁne the IBOO as follows.
Deﬁnition 1 (Inconsistencies Between Online and Ofﬂine, IBOO) .
The IBOO refers to the gaps between the RAS and the VAS that prevent the VAS from accurately simulating the RAS.
Speciﬁcally, there are two dominated gaps between the RAS and the VAS.
One is that the VAS cannot accurately simulate the cascade auction architecture of the RAS.
For example, due to the constraint on computing power, the VAS is built only based on the historical data generated in stage 2 of the RAS 3 , which makes the VAS lack the mechanism of stage 1 in the RAS.
In addition, the
VAS does not incorporate the exact inﬂuences of other advertisers as RAS does, which makes up the second dominated gap.
For example, in the RAS, the market prices are determined by the bids of all advertisers which can change during the training process.
However, the VAS always provides constant market prices for the auto-bidding policy being trained.
See ﬁgures in Appendix A.3 for illustrations.
Essentially, the IBOO makes the VAS provide the auto-bidding policy with false rewards and state transitions during RL training process.
As presented in Table.
1, the auto-bidding policy with better performance in the VAS (i.e., higher R/R ⇤ ) may yield poorer performance in the RAS (i.e., lower
A/B test value).
Hence, the IBOO can seriously degrade the performance of the auto-bidding policy in the RAS.
One way to address the IBOO challenge is to improve the VAS and reduce the gaps as much as possible.
However, due to the complex and ever-changing nature of the RAS, the reduction in
IBOO is usually very limited, resulting in little improvement in the auto-bidding policy performance for illustrations.
Hence, new schemes need to be devised so as to avoid the IBOO and improve the performance of auto-bidding policies.
Besides, it is worth noting that the IBOO resembles the sim2real problem studied in other realms, such as computer visions [8] and robotics [9].
Nonetheless, to the best of our knowledge, this is the ﬁrst work that formally put forward the concept of IBOO in the ﬁeld of auto-bidding and systematically analyzes and resolves it.
the opposite.
means that the RAS and VAS evaluate the same.
[Table]
In this paper, we propose a novel sustainable online reinforcement learning (SORL) framework to address the IBOO challenge.
For the ﬁrst time, the SORL abandons the way of learning with the VAS and trains the auto-bidding policy by directly interacting with the RAS.
Notably, the SORL can obtain true rewards and state transitions from the RAS and thereby does not suffer from the IBOO problem.
The SORL contains two main algorithms.
The ﬁrst one is a safe and efﬁcient online exploration policy for collecting data from the RAS, named as the SER policy.
Speciﬁcally, to guarantee the safety of explorations, we design a safety zone to sample actions from based on the Lipschitz smooth property of the Q function we theoretically proved.
We also derive the lower bound on the safety degree.
To increase the efﬁciency of explorations, we develop a sampling distribution that can make the collected data give more feedbacks to the auto-bidding policy being trained.
The second main algorithm is an effective and stable ofﬂine RL method to train the auto-bidding policy based on the collected data, named as the variance-suppressed conservative Q-learning (V-CQL).
Speciﬁcally, motivated by the observation that the optimal 4 Q function is in the quadratic form, we design a regularization term in the V-CQL to optimize the shape of the Q function.
The V-CQL can train the auto-bidding policy with high average, hence effective, and low variance, hence stable, in performance under different random seeds.
The whole SORL works in an iterative manner, alternating between collecting data and training the auto-bidding policy.
Extensive simulated and real-world experiments validate the effectiveness of our approach.
Related Work
VAS-based RL Auto-bidding Methods.
Impressed by powerful contextual learning and sequential decision-making capabilities of RL, modern auto-bidding policies, such as DRLB [1], RLB [4],
MSBCB [2] and USCB (state-of-the-art) [5], are usually learned by RL algorithms in a manually built VAS.
However, as stated before, they all suffer from the IBOO problem.
The SORL avoids using the VAS and thereby completely address the IBOO challenge.
Ofﬂine RL.
Ofﬂine RL (also known as batch RL) [11, 13, 26, 27, 30, 31, 38, 39, 40] aims to learn better policies based on a ﬁxed ofﬂine dataset collected by some behavior policies.
The main challenge ofﬂine RL addressed is the extrapolation error caused by missing data [27].
Speciﬁcally, ofﬂine RL algorithms usually address this challenge in three ways, including policy constraint methods such as
BCQ [11], BEAR [30], and conservative regularization methods such as CQL [13], BRAC [40], as well as modiﬁcations of imitation learning [37] such as ABM [31], CRR [38], BAIL[39].
However, there is another important challenge that ofﬂine RL cannot solve: the ﬁxed ofﬂine dataset cannot be guaranteed to contain sufﬁcient state-action pairs from high-reward regions [27].
This challenge exists in many practical applications, including the auto-bidding problem, and can inherently prevent ofﬂine RL algorithms from learning excellent policies.
Hence, a great advantage of the SORL is its ability to continuously collect data from OOD high-reward regions that can give new feedbacks to the auto-bidding policy being trained.
Off-Policy Evaluation (OPE).
OPE is an algorithm to evaluate the performance of the policy with ofﬂine data collected by other policies [12].
The state-of-the-art OPE algorithm in auto-bidding is to calculate the R/R ⇤ of the evaluated policy in the VAS (see Appendix B for details).
However, as shown in Table. 1, the OPE in auto-bidding is not accurate.
This indicates that we cannot rely on the
OPE to select auto-bidding policies with good performance for directing further online explorations.
Notably, the proposed ofﬂine training algorithm in the SORL, V-CQL, outperforms existing ofﬂine
RL algorithms in training stabilities and helps to reduce the OPE process during iterations in SORL.
Online RL.
Safety is of vital importance in the online exploration of many real-world industrial systems, including the RAS.
Many safe online RL algorithms have been proposed to guarantee the safety of explorations [17, 18, 19, 20, 21, 22, 33, 34, 35].
However, many of them are either developed based on the constraints that are not suitable for the auto-bidding problem or designed for systems with speciﬁc assumptions (see Appendix B for details).
Besides, many existing works [33, 34, 35, 21] assume that there exists a speciﬁc set of safe state-action pairs that can be explored.
Some work [33] even requires to know the safe states in advance.
However, in the auto-bidding problem, no speciﬁc actions at any state cannot be explored as long as the expected accumulative reward maintains at a high level.
This requires the exploration policy to maintain high performance throughout the iterations, which is more challenging.
Notably, the SER policy can meet this requirement with theoretical guarantees.
Recently, with the development of the ofﬂine RL ﬁeld, many algorithms for efﬁcient online explorations on the premise of having an ofﬂine dataset [14, 23] have emerged.
However, they often only focus on the efﬁciency of data collections but ignore the safety of it.
Notably, the SER policy in the SORL strikes a good balance between the efﬁciency and safety.
Supplementary related works are described in Appendix B.
Problem Settings
In this paper, we study the auto-bidding problem from the perspective of a single advertiser, which can be viewed as an episodic task of T 2 N + time steps.
Between time step t and t + 1 , there are
N t 2 N + impression opportunities, each of which has a positive value v j,t  v M and is sold to the winning advertiser at a market price p j,t  p M , where v M > 0 and p M > 0 are the upper bounds for values and market prices, respectively, j 2 { 1 , 2 , ..., N t } , t 2 { 1 , 2 , ..., T } .
Denote p 1 j,t  p M and v 1 j,t  v M as the market price and rough value of impression opportunity j in stage 1, and p 2 j,t  p M and v 2 j,t  v M as the market price and accurate value of it in stage 2.
Note that they are all positive values.
Let the total budget of the advertiser be B > 0 .
The auto-bidding problem can be modeled as a Constraint Markov Decision Process (CMDP) [15] < S , A , R, P ,  , C > .
The state s t , [ b t , T  t, B  b t ] 2 S is composed of three elements, including the budget left s t (1) =
b t , the time left s t (2) =
T  t and the budget consumed s t (3) =
B  b t .
The action a t 2 A , [ A min , A max ] is the bidding price at time step t , where A max > A min > 0 are the upper and lower bounds, respectively.
The reward function r t ( s t , a t ) and the constraint function c t ( s t , a t ) are the total value of impression opportunities won between time step t and t + 1 and the corresponding costs, respectively, and P denotes the state transition rule.
Note that R , C and P are all directly determined by the RAS.
Moreover,  2 [0 , 1] is the discounted factor.
We denote the auto-bidding policy as µ : S !
A , and let ⇧ be the policy space.
The goal of the auto-bidding problem is to maximize the total value of impression opportunities earned by the advertiser under the budget constraint, which can be expressed as
[Equation]
One can leverage standard RL algorithms to train a state-of-the-art auto-bidding policy [5] with the
VAS, where the constraint in (1) is met by terminating the training episode early once the budget runs out.
However, as stated before, this way of learning suffers from the challenge of IBOO.
Hence, we propose the SORL framework to avoid the IBOO in the following section.
Our Approach
The SORL framework consists of two algorithms, including the SER policy for online data collections and the V-CQL method for ofﬂine training of the auto-bidding policy based on the collected data.
The
SORL works in an iterative manner, alternating between online data collections and ofﬂine training.
4. Online Exploration: SER Policy
There are two requirements on the online exploration policy.
As the exploration policy is directly deployed in the RAS, the primary requirement is safety .
Unlike the safety in other realms such as robotics, it is not appropriate to construct an immediate-evaluated safety function to assess the safety degree of each state-action pair merely based on their values in auto-bidding.
Actually, any action in any state would be safe as long as the summation of all rewards of the whole episode maintains at a high level.
See Appendix F.1.3 for detailed explanations.
Let µ s be the auto- deployed that the online exploration policy as ⇡ e can
[Figure]
formally deﬁne the safety requirement as: the performance of ⇡ e cannot be lower than that of µ s by a threshold ✏ s > 0 , i.e., V ( µ s )  V ( ⇡ e )  ✏ s .
Besides, ⇡ e needs to be efﬁcient , i.e., collecting data that can give new feedbacks to the auto-bidding policy being trained , which constitutes the second requirement.
We next propose the SER policy for exploration that can satisfy both requirements.
4.1.1 Theory: Lipschitz Q Function
Our basic idea of ensuring the safety of exploration is to design a safety zone for taking actions around the outputs of µ s .
The motivation for this idea is our proof of the Lipschitz smooth property of the Q function in the auto-bidding problem.
We next report the theorem of this Lipschitz smooth property, as well as corresponding propositions and assumptions.
Let µ be an arbitrary auto-bidding policy.
According to the Bellman equation, we have:
[Equation]
Based on the mechanism of RAS, we formulate the reward function R , the constraint function C and the transition rule P as follows, and the proof can be found in Appendix D.1.
Proposition 1 (Analytical expressions of R , C and P ) .
Based on the characteristic of the two-stage cascaded auction in the RAS, we can formulate the reward function R as r t ( s t , a t ) =
P
[Equation]
p 1 j,t , a t v 2 j,t  p 2 j,t } v j,t , the constraint function C as c t ( s t , a t ) =
P j { a t v 1 j,t  p 1 j,t , a t v 2 j,t  p 2 j,t } p j,t , and the state transition rule P as s t +1 =
s t
+ [ 4 s t (1) , 4 s t (2) , 4 s t (3)] , where 4 s t (1) = 4 s t (3) =
P j { a t v 1 j,t
[Equation]
p
[Equation]
j,t , a t v 2 j,t
[Equation]
p
[Equation]
j,t } p j,t and
[Equation]
We make the following assumption on the impression opportunities, whose rationalities can be found in Appendix C.1.
Assumption 1 (Bounded Impression Distributions) .
Between time step t and t + 1 , we assume the numbers of winning impressions with action a t in the ﬁrst stage n t, 1 and the second stage n t, 2 can both be bounded by linear functions, i.e., n t, 1  k 1 a t , n t, 2  k 2 a t , where k 1 , k 2 > 0 are constants.
Based on this assumption, we claim that the reward function r t ( s t , a t ) is Lipschitz smooth with respect to actions a t at any given state s t .
See Appendix E.1 for the proof.
Theorem 1 (Lipschitz Smooth of r t ( s t , a t ) ) .
Under Assumption 1, the reward function r t ( s t , a t ) is
L r -Lipschitz smooth with respect to actions a t at any given state s t , where L r =
( k 1 + k 2 ) v M .
We make the following mild assumptions, whose rationalities can be found in Appendix C.2.
Assumption 2 (Bounded Partial Derivations of Q µ ( s t , a t ) ) .
We assume that the partial derivation of
Q µ ( s t , a t ) with respect to s t (1) and s t (3) is bounded, i.e.,  @ Q µ ( s t ,a t ) @
s t (1)   k 3 and
[Equation]
[Equation]
where k 3 , k 4 > 0 are constants.
Equipped with Theorem 1, we can prove that the Q function Q µ ( s t , a t ) is also Lipschitz smooth.
See
Appendix E.2 for proof.
Theorem 2 (Lipschitz Smooth of Q µ ( s t , a t ) ) .
Under Assumption 1 and 2, the Q function Q µ ( s t , a t ) is an L Q -Lipschitz smooth function with respect to the actions a t at any given state s t , where
L Q =
[ v M +  ( k 3 + k 4 ) p M ]( k 1 + k 2 ) .
This means that the decrease rate of Q µ , the subsequent accumulated rewards, due to action offset at any time step t is bounded by L Q , which gives us a way to design the safety zone.
Speciﬁcally, as shown in Fig. 2(a), the safety zone at state s t can be design as the neighborhood of action µ s ( s t ) , i.e., [ µ s ( s t )  ⇠ , µ s ( s t ) + ⇠ ] , where ⇠ > 0 is the range.
In this way, the online exploration policy ⇡ e can be designed as: sampling within the safety zone [ µ s ( s t )  ⇠ , µ s ( s t ) + ⇠ ] in certain  T consecutive time steps, and sticking to µ s in the rest of T   T time steps, i.e.,
[Equation]
sampling from [ µ s
[Equation]
s
[Equation]
[Equation]
µ
[Equation]
otherwise .
8 t,
[Equation]
where 0  t 1 < t 2  T  1 ,  T =
t 2  t 1 + 1 , and t 1 , t 2 2 N + .
In the following theorem, we give the upper bound of | V ( ⇡
[Equation]
)  V ( µ s ) | .
Theorem 3 (Upper Bound of | V ( ⇡ e )  V ( µ s ) | ) .
The expected accumulated reward V ( ⇡ e ) satisﬁes
[Equation]
See Appendix E.3 for the detailed proofs.
Hence, to meet the safety requirement, we design the range of the safety zone as ⇠ 
[Equation]
4.1.2 Sampling Method
With the safety guarantee, we can further design the sampling method in ⇡ e to increase the efﬁciency of exploration.
In the following, we use ˜ ⇡ x and ⇡ x to denote the sampling distribution and the corresponding exploration policy, respectively, where x represents the version.
Vanilla Sampling Method.
A vanilla sampling method in the safety zone can directly be sampling from a (truncated) Gaussian distribution ˜ ⇡ e, N = N ( µ s ( s t ) ,  2 ) with mean µ s ( s t ) and variance  2 .
However, ˜ ⇡ e, N resembles the policy µ s to a large extent, which makes the explorations conservative and cannot give sufﬁcient feedbacks to the auto-bidding policy being trained.
SER Sampling Method.
To lift up the efﬁciency of explorations, we shift the distribution ˜ ⇡ e, N towards the actions of the auto-bidding policy being trained, while constraining the deviations within a threshold ✏ e > 0 .
This can be formulated as a functional optimization problem:
[Equation] [Equation]
where the optimization variable ˜ ⇡
e denotes the shifted distribution, b
Q is the Q function of the auto-bidding policy being trained.
Using Euler equations, we can derive the form of ˜ ⇡
e from (5) as
[Equation]
, (6) where C ( s t ) =
R a t exp {  ( a t  µ s ( s t )) 2 2 + 1  b
Q ( s t , a t ) }
d a t acts as the normalization factor.
The complete deductions of ˜ ⇡ e are given in Appendix F.1.1.
We can interpret ˜ ⇡
e as a deviation from ˜ ⇡ e, N towards the Q function b
Q ( s t , a t ) , where ˜ ⇡ e, N further guarantees the safety and the Q function ensures the efﬁciency.
Note that  and  in (6) are both hyper-parameters that can control the randomness and deviations from µ s of the SER policy, respectively.
As shown in Fig.
2(b), the smaller the value of  , the larger the deviations of ˜ ⇡
e from µ s ; besides, the bigger the value of  , the greater the randomness of ˜ ⇡ e .
Hence, we can control the randomness and deviations from the safe policy µ s of exploration policy easily by adjusting the value of  and  .
In addition, we describe the way of practically implementing the sampling of distribution ˜ ⇡
e in Appendix F.1.2.
Then the complete SER policy is ⇡ e ( s t ) =
⇢ sampling from [ µ s ( s t )  ⇠ , µ s ( s t ) + ⇠ ] with ˜ ⇡ e , t 1  t  t 2 ; µ s
[Equation]
s t
[Equation]
otherwise .
8 t.
4. Ofﬂine Training: V-CQL
Due to the safety constraint on the explorations, the SER policy can only collect data within the safety zone and the data outside the safety zone will be missed.
Hence, we leverage a strong baseline ofﬂine RL algorithm, CQL, to address the extrapolation error and make the ofﬂine training effective .
Moreover, as the sampling distribution ˜ ⇡ e in the SER policy involves the Q function b
Q , the training result will affect the directions of further explorations.
However, on the one hand, as stated in Section 2, the OPE in the auto-bidding is not reliable; and on the other hand, the performance variance under different random seeds of existing ofﬂine RL algorithms can be large, as shown in Fig.
5.2 in the experiment.
Hence, we further design a novel regularization term in the CQL loss to increase the stability of ofﬂine training and thereby reduce the OPE process.
This forms the V-CQL method.
Speciﬁcally, we observe that the Q functions of the state-of-the-art auto-bidding policy are always in nearly quadratic forms.
In addition, we conduct simulated experiments where the auto-bidding policy is directly trained in a simulated RAS 6 with traditional RL algorithms.
We ﬁnd out the optimal Q functions in the simulated experiments are also in nearly quadratic forms.
See Appendix F.2.1 for illustrations.
Based on these observations, we assume that the optimal Q functions in the RAS are also in nearly quadratic forms.
Hence, the key idea of V-CQL is to restrict the Q function to remain nearly quadratic, and the regularization term in the V-CQL is designed as:
[Equation]
b
Q qua ( s k , · ))
P a exp( b
Q qua ( s k , a )) | {z } distribution(form) of b
Q qua ◆ , (8) where Q qua is selected as the Q function of the state-of-the-art auto-bidding policy, D KL ( · , · ) represents the KL-divergence, and  > 0 is a constant controlling the weight of R ( µ ) .
Moreover, we can interpret (8) from another aspect, i.e., R ( µ ) tries to limit the distance between the distribution of the Q function and the distribution of b
Q qua .
This can limit the derivation of the auto-bidding policy from the corresponding state-of-the-art policy, which can increase the training stability.
Complete implementations of the V-CQL are shown in Appendix F.2.2.
4. Iterative Updated Structure
Fig.
3 shows the whole structure of the SORL framework.
Speciﬁcally, the SORL works in an iterative manner, alternating between collecting data with the SER policy from the RAS and training the auto-bidding policy with the V-CQL method.
Warm Booting.
To start with, we use the safe policy µ s to boot the explorations in the RAS.
Denote the data collected by µ s as D s = { ( s k , a k , r k , s 0 k ) } k , and the auto-bidding policy trained by D s with the V-CQL as µ 0 .
Iteration Process.
Denote the SER policy and the auto-bidding policy in the ⌧ -th iteration as ⇡ e, ⌧ and µ ⌧ , and the data collected in the ⌧ -th iteration as D on, ⌧ , where ⌧ 2 { 1 , 2 , ... }
The design for the sampling distribution ˜ ⇡ e, ⌧ in iteration ⌧ is ˜ ⇡ e, ⌧ = 1 C ⌧ ( s t ) ˜ ⇡
e, N exp { b
Q ⌧ ( s t , a t ) /  ⌧ } , where
C ⌧ ( s t ) acts as the normalization factor and  ⌧ is the hyper-parameter used in iteration ⌧ , and
b
Q ⌧ is the Q function of µ ⌧ .
We note that the exploration policies in each iteration are safe, i.e., | V ( ⇡ e, ⌧ )  V ( µ s ) |  ✏ s , 8 ⌧ , since they all taking actions within the safety zone only with different sampling distributions.
We leverage the V-CQL to in each iteration improve the auto-bidding policy.
Speciﬁcally, at iteration ⌧ , we substitute b
Q qua in (8) with b
Q ⌧  1 , and train a new Q function b
Q b
Q ⌧ for the next iteration.
The auto-bidding policy are expected to be continuously improved.
A summary of the SORL framework is present in Appendix F.3.
Experiments
We conduct both simulated and real-world experiments to validate the effectiveness of our approach 7 .
The following three questions are mainly studied in the experiments: (1) What is the performance of the whole SORL framework?
Is the SER policy safe during iterations?
Can the V-CQL method continuously improve the auto-bidding policy and outperform existing ofﬂine RL algorithms and the state-of-the-art auto-bidding policy?
(2) Is the safety zone reliable?
Is the SER policy still safe when using Q functions of auto-bidding policies with bad performance?
(3) Does the V-CQL really help to reduce the performance variance compared to existing ofﬂine RL algorithms?
Experiment Setup.
We conduct the real-world experiments on one of the world’s largest E-commerce platforms, TaoBao.
See Appendix G.1 for details.
The simulated experiments are conducted in a manually built ofﬂine RAS and the corresponding VAS.
See Appendix A.2 for details.
The safe auto-bidding policy µ s used for warm booting and constructing the safety zone is trained by the state-of-the-art auto-bidding policy, USCB[5].
The safety threshold is set as ✏ s = 5% V ( µ s ) .
Performance Index.
The objective function V ( µ ) in (1), i.e., the total value of impression opportunities won by the advertiser in the episode, acts as the main performance index in our experiments and is referred as BuyCnt in the following.
In addition, we utilize three other metrics that are commonly used in the auto-bidding problem to evaluate the performance of our approach.
The ﬁrst metric the total consumed budget (ConBdg) of the advertiser.
The second metric is the return on investment (ROI) which is deﬁned as the ratio between the total revenue and the ConBdg of the advertiser.
The third metric is the cost per acquisition (CPA) which is deﬁned as the average cost for each successfully converted impression.
Note that larger values of BuyCnt, ROI, and ConBdg with a smaller CPA indicate better performance of the auto-bidding policy.
Baselines.
We compare our approach with the state-of-the-art auto-bidding policy, USCB [5], that is trained by RL in the VAS.
We also compare the V-CQL method with modern ofﬂine RL algorithms, including BCQ [11] and CQL [13].
Recall that many safe online RL algorithms are not suitable for explorations in the auto-bidding problem as stated in Section 2, we compare the SER policy with ⇡ e, N in our experiments.
5. Main Results
To Answer Question (1): We ﬁrst conduct simulated experiments with the SORL, and the results during iterations are shown in Fig.
Speciﬁcally, from Fig. 4(a), we can see that the decline rate in
BuyCnt of both the SER policy and the vanilla exploration policy are smaller than 5% , which veriﬁes the safety of the SER policy.
Moreover, the BuyCnt of the SER policy ⇡ e, ⌧ rises with iterations and is alway higher than that of ⇡ e, N , which indicates that the SER policy is more efﬁcient.
From Fig.
4(b), we can see that the BuyCnt of the auto-bidding policy µ ⌧ trained with V-CQL is higher than that of BCQ, CQL and USCB, which indicates the effectiveness of the V-CQL.
Besides, the BuyCnt rises with the number of iterations and converges to the optimal BuyCnt (i.e., the BuyCnt of the optimal policy) at the 5 -th iteration.
This validates the superiority of the whole SORL framework.
This indicates the effectiveness of the V-CQL method.
For real-world experiments, we utilize 10 ,advertisers to collect data from the RAS with ⇡ e and compare the auto-bidding policies in 4 iterations on 1 , 500 advertisers using A/B tests, and the results are shown in Table.
We can see that the performances of auto-bidding policies are getting better with iterations and exceeds the state-of-the-art algorithm, USCB, which validates the superiority of the whole SORL framework.
[Figure]
[Table]
5. Ablation Study
To Answer Question (2): We fully examine the safety of the
SER policy using the Q function b
Q of auto-bidding policies with different performance levels.
Speciﬁcally, in the simulated experiment
[Table]
we utilize seven different versions of auto-bidding policies { µ ( k ) } 7 k =1 with Q functions { b
Q ( k ) } 7 k =1 that have various per- formance levels to construct seven corresponding SER policies, where V ( µ (1) ) <
V ( µ (2) )
< ..., V ( µ (7) ) .
We also construct a vanilla exploration policy ⇡ e, N for comparison.
The hyperparameters for all exploration policies are  = 1 ,  = 0 .
We apply the ⇡ e, N and seven SER policies to the simulated RAS, and the BuyCnt of explorations policies are shown in Fig.
We can see that the BuyCnt of the SER policy rises with the performance level of the auto-bidding policy.
The worst BuyCnt of the SER policy drops about 3 .
52% compared with µ s , which meets the safety requirement.
Moreover, we can balance between the safety and efﬁciency of the SER policy by regulating the hyper-parameters  and  .
The corresponding results are shown in Appendix G.2.2.
[Table]
For real-world experiments, we apply ⇡ e and ⇡ e, N to 1 , 300 advertisers in the RAS and compare their performance to µ s in A/B tests, as shown in Table.
We can see that the varieties of the ConBdg and BuyCnt of ⇡ e are all with in 5% , and are better than those of ⇡ e, N , which indicates the safety of the SER policy.
To Answer Question (3): We leverage the V-CQL, CQL, BCQ and USCB to train auto-bidding policies under 100 different random seeds in the simulated experiment, and the results are shown in
Fig. 6(a).
We can see that the performance variance of the V-CQL is much smaller than those of other algorithms.
At the same time, the average performance of the V-CQL can maintains at a high level.
In addition, we leverage the V-CQL and CQL algorithm to train the auto-bidding policies based on the real-world data under 100 different random seeds.
To ensure fairness, we also utilize the USCB to train the auto-bidding policies in the VAS under exactly the same random seeds.
We carry out the
OPE for these three sets of auto-bidding policies, as shown in Fig.
6(b).
Recall that R/R ⇤ is the main
[Figure]
[Table]
main metric used in the OPE for auto-bidding (see Appendix
B).
We can see that the maximum R/R ⇤ of the V-CQL is much larger than that of the USCB, which indicates that the V-CQL is capable of training a better policy.
Although the maximum R/R ⇤ of the V-CQL and the CQL are about in the same level, the variance of R/R ⇤ of the V-CQL is smaller than that of the CQL.
For real-world experiments, we apply the auto-bidding policies trained by the V-CQL and USCB to 1 , 600 advertisers in the A/B test for seven days in the RAS, and the average values of the metrics are shown in Table 5.
We can see that the V-CQL outperforms USCB method in almost all metrics.
Moreover, we present the A/B test results under 10 random seeds in Table.
We can see that the
V-CQL outperforms the USCB under 9 seeds, and only slightly worse under the other one seed.
All these results indicate that the V-CQL can really help to reduce the performance variance and increase the training stability, while keeping the average performance at a high level.
[Table]
Conclusions
In this paper, we study the auto-bidding problem in online advertisings.
Firstly, we deﬁne the IBOO challenge in the auto-bidding problem and systematically analyze its causes and inﬂuences.
Then, to avoid the IBOO, we propose the SORL framework that can directly learn the auto-bidding policy with the RAS.
Speciﬁcally, the SORL framework contains two main algorithms, including a safe and efﬁcient online exploration policy, the SER policy, and an effective and stable ofﬂine training method, the V-CQL method.
The whole SORL framework works in an iterative manner, alternating between online explorations and ofﬂine training.
Both simulated and real-world experiments validate the superiority of the whole SORL framework over the state-of-the-art auto-bidding algorithm.
Moreover, the ablation study shows that the SER policy can guarantee the safety of explorations even under the guide of the auto-bidding policy with bad performance.
The ablation study also veriﬁes the stability of the V-CQL method under different random seeds.
Acknowledgments and Disclosure of Funding
This work is supported by Alibaba Research Intern Program.
The authors would like to thank
Mingyuan Cheng, Guan Wang, Zongtao Liu, Zhaoqing Peng, Lvyin Niu, Miao Xu, and Tianyu Wang for their valuable feedbacks and insightful discussions.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 6 and social impact in Appendix H. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] We believe there is no negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section 4.1.1 and Appendix C. (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix D and Appendix E. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We include the codes for simulated experiments in Section 5 in the paper.
The codes for real-world experiments are proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A]
Our work uses own data, and the data used in the simulation experiments and real-world experiments is released in supplementarymaterial.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]