Product Ranking for Revenue Maximization with Multiple Purchases
Abstract
Product ranking is the core problem for revenue-maximizing online retailers.
To design proper product ranking algorithms, various consumer behavior models are proposed to characterize the consumers’ behaviors when they are provided a list of products.
However, existing works assume that each consumer purchases at most one product or will keep viewing the product list after purchasing a product, which does not agree with the common practice in real scenarios.
We assume that each consumer can purchase multiple products at will in this work.
To model consumers’ willingness to view and purchase, we set a random attention span and purchase budget, which determines the maximal amount of products that he/she views and purchase, respectively.
Under this setting, we first design an optimal ranking policy when the online retailer can precisely model consumers’ behaviors.
Based on the policy, we further develop UCB-like algorithms with ˜ O ( √ T ) regret that estimates consumers’ behaviors and maximize revenue simultaneously in online settings.
Experiments on both synthetic and semi-synthetic datasets prove the effectiveness of the proposed algorithms. The source code is available at https://github.com/windxrz/product-ranking .
Introduction
Online retailing has become increasingly popular over the last decades [17, 27, 51].
The way of product ranking is the crux for online retailers because it determines the consumers’ shopping behaviors [17] and thus influences the retailers’ revenue [20, 48].
For instance, the probability of consumers’ purchasing from a firm or clicking an advertisement is strongly related to the display order [8, 3, 32].
Therefore, it is crucial for online retailers to design proper product ranking algorithms for revenue maximization.
The consumers’ behavior, when faced with a list of products, is the basis of the problem [17].
Generally speaking, existing works assume that the consumers view the list of products sequentially and select products according to a choice model [42, 41, 31, 37, 18, 25, 24, 36].
However, most of them assume that each consumer purchases at most one product and stop browsing immediately afterward, which does not agree with the common practice since a consumer may expect multiple purchases on a website or an app [33, 34, 16, 27].
Recently, several works consider a multiplepurchase setting while their targets are not revenue maximization [27, 15], or they assume that consumers always continue viewing the product list after purchasing any product, which is not practical in real scenarios [39].
In this paper, we propose a more realistic setting to characterize the consumer behaviors under multiple-purchase settings.
Following [13, 17, 39], we assume that the consumers view the product list sequentially and purchase a product when its utility exceeds a certain threshold.
Similar to [17], we assume the existence of the attention span for each consumer, which determines the maximal amount of products that the consumer is willing to view.
More importantly, instead of forcing the consumer to quit viewing once she purchases one product, we allow each consumer to conduct multiple purchases at will and explore product ranking models for revenue maximization.
Specifically, we constrain each consumer with a purchase budget to determine the maximal amount of purchases.
Both the attention span and the purchase budget are assumed to obey the geometric distribution [13, 39].
Under this setting, we study the optimal product ranking policy to maximize the online retailers’ revenue.
We first characterize the optimal ranking policy that achieves the maximal revenue when the online retailer knows the consumer parameters, including the purchase probabilities for each product and the distribution of the attention span and purchase budget.
Afterward, we investigate the problem in online learning frameworks where consumer parameters are unknown to online retailers.
In both non-contextual ( i.e. , all consumers share the same parameters) and contextual settings ( i.e. , consumers have personalized parameters), we provide UCB-like algorithms [1] that could fit consumers’ parameters and maximize revenue simultaneously.
We further theoretically prove that the algorithms achieve ˜ O ( √ T ) regrets in both settings.
Finally, we conduct experiments to verify the effectiveness of the algorithms.
To conclude, our contributions are highlighted as follows:
• We propose a novel setting to deal with multiple-purchase activities of consumers on online retailing scenarios.
• We characterize the optimal ranking policy and further design UCB-like algorithms with ˜ O ( √ T ) regret based on them in both non-contextual and contextual online settings.
• We conduct experiments on both synthetic and semi-synthetic datasets to prove the effectiveness of our method on the multiple-purchase setting.
Related works
Consumers choice model Traditional choice models adopt the multinomial logic (MNL) [12, 22, 42] models to characterize consumers’ behaviors.
However, these models can not describe the behaviors when the consumers are provided with a list of products.
As a result, several choice models are proposed recently.
In detail, Abeliuk et al.
[2] add a position bias term in the traditional MNL model.
Gallego et al.
[29], Aouad and Segev [4] suppose that the products are framed into a set of virtual web pages.
Consumers select a product, if any, from these pages following a general choice model (such as MNL).
Flores et al.
[28] propose the Sequential Multinomial Logit (SML), which assumed a two-stage MNL model and supposed that products are divided into two stages with prior knowledge.
Liu et al.
[40], Feldman and Segev [26], Gao et al.
[30] further extend the setting to multiple stages.
These works suppose that each stage contains multiple products while recently several works assume that a product is treated as a single stage [17, 13, 14].
These models are also closely related to the cascade model [12, 35, 37, 53, 19].
Gao et al.
[31] further propose the general cascade click model.
Following these works, we suppose that each stage contains only one product and the consumers views the products sequentially.
In addition, several works consider the setting where consumers may have multiple interactions with the platforms.
However, most of them focus on the click model [15, 16, 33–35] or MNL-based consumer choice models [47, 52, 49].
In addition, Ferreira et al. [27] maximize the number of consumers who engage with the site.
Liang et al.
[39] aim to maximize the revenue in the multiplepurchase setting while they assume that consumers always continue viewing the product list after purchasing any product, which is not practical in real scenarios.
Online learning and multi-armed bandit
Online retailers need to design proper algorithms to model consumers’ behaviors and maximize the revenue at the same time, which is closely related to the multi-armed bandit framework [46, 44].
Many works [5, 9, 11, 10, 23] have been proposed to deal with demand learning or price experimentation problems with the help of the framework in the field of revenue management.
Recently, more methods [39, 17, 13, 14, 6, 43] adopt similar techniques in the product ranking setting, which inspires the algorithm proposed in this paper.
Preliminaries
Let [ N ] ≜ { 1 , 2 , . .
, N } be the set of all products owned by an online-retailer (he).
The product k generates revenue r k when a consumer purchases it.
Let r max ≜ max k ∈ [ N ] r k be the maximal revenue for all products and r = { r k } Nk =1 .
We use the notation σ t ⊣ [ N ] to denote the ranking policy σ t on the products for the t -th consumer (she).
More precisely, ∀ k ∈
[ N ] , σ t,k ∈
[ N ] represents the product displayed in the k -th position for her.
Reversely, σ − 1 t,k denotes the position of the k -th product in the ranking list.
Furthermore, for any subset Q ⊆ [ N ] that represents the possible positions in the ranking list, we use σ t,Q to denote the corresponding set of the products w.r.t. the position set Q , i.e. , σ t,Q = { σ t,k : k ∈ Q } .
In addition, we use max( · ) to denote the maximal element in a set.
The consumers choice model When the t -th consumer arrives, she views the list of products sequentially.
The probability of product k being purchased by her is denoted as λ t,k and let λ t = { λ t,k } Nk =1 .
To characterize the consumers’ behaviors under multiple-purchase settings, we assume that the consumer is endowed with the attention span V t and purchase budget B t , which determine the maximal amount of the products that she will view and purchase, respectively.
In reality, the consumer always have random attention span and purchase budget [17].
Therefore, following [13, 39], we assume V t ∼ Geo(1 − q t ) and B t ∼ Geo(1 − s t ) are both geometric distributions with parameter 1 − q t and 1 − s t , respectively.
As a result, P ( V t = v ) =
q v − 1 t (1 − q t ) for any v ≥and P ( B t = b ) =
s b − 1 t (1 − s t ) for any b ≥ 1 .
With attention span V t and purchase budget B t , the consumer keeps viewing the products until she views V t or purchases B t products. Because of the attention span V t , the consumer will not purchase products whose positions are greater than V t in the ranking list.
Let Q ⊆
[ V t ] denote the possible positions of the purchased products by the consumer. The corresponding indices of purchased products are σ t,Q .
Therefore, the probability that the consumer purchases the subset of products σ t,Q is given by P
[Equation] [Equation]
Here the condition k ≤ max( Q ) requires that the ranking policy σ t displays the product k earlier than some products in the set Q .
Specifically, the probability P buy ( Q ; V t , B t , λ t , σ t ) is calculated based on the cardinality of Q .
When | Q | < B t , the consumer does not meet her purchase budget and stops browsing the list after viewing V t products.
When | Q | =
B t , the consumer stops browsing as long as she purchases the last product in the list Q .
When | Q | > B t , the probability becomes 0 since the amount of purchases | Q | exceeds her purchase budget.
Remark.
The setting of the purchase budget provides a rational way to characterize the consumers’ behaviors with multiple purchases and extends previous works that suppose consumers would purchase at most one product [13, 17].
Furthermore, when s t → 0 , the consumers would tend to purchase at most one product and our setting degenerates to the setting proposed in [13].
Thus the previous setting can be considered as a special case of our setting. In addition, Liang et al.
[39] assume that consumers always continue viewing the product list after purchasing any product, which is not practical in real scenarios.
By contrast, the purchase budget proposed in our paper leads to a different and more practical consumer choice model.
In addition, our model introduces an extra parameter ( i.e. , the parameter, which makes our problem more challenging.
Revenue optimization The online-retailer chooses a ranking policy σ t ⊣ [ N ] on the products for the t -th consumer.
For the consumer with fixed purchase probabilities for each product λ t , attention span V t , and purchase budget B t , the expected revenue could be written as
[Equation]
In reality, the attention span V t and purchase budget B t are random and can not be estimated accurately by the online retailer.
As a result, the retailer calculates the expected revenue for the t -th consumer by sampling V t from Geo(1 − q t ) and sampling B t from Geo(1 − s t ) .
[Equation]
To maximize the revenue, the optimal ranking policy is given by
[Equation]
For simplicity, we use σ ∗ t to denote σ ∗ t ( λ t , q t , s t ) .
Basic assumption We also need the following assumption to guarantee the tractability of the proposed problem.
In reality, consumers always have finite attention span, which indicates that q t is always smaller than 1 .
Assumption 3.1.
There exists a known constant ϵ q > 0 such that for all t > 0 , q t ≤ 1 − ϵ q .
Optimal ranking policy given consumers’ characteristics
In this section, we introduce the optimal ranking policy when the online retailer exactly knows the consumers’ parameters, including the purchase probability for each product λ , random attention span (parametrized with q ), and random purchase budget (parametrized with s ).
We drop the subscript t in this section for simplicity.
The optimal ranking strategy stems from the following theorem.
Theorem 4.1.
Under Assumption 3.1, for any 1 ≤ i < N , the following holds for the optimal ranking strategy σ ∗ ( λ , q, s ) .
[Equation] Remark.
The expected revenue of a single product σ ∗ i is λ σ ∗ i r σ ∗ i , which grows as the purchase probability and the revenue generated by the product increases.
Yet purchasing a product reduces the purchase probability of the following products, and thus the expected revenue of other products decreases as σ ∗ i increases.
Furthermore, when s → 0 , the term in Equation (5) become the same as that in [13, Theorem 1].
As a result, Theorem 4.1 consistently extends the results on the single purchase setting [13] to the multiple purchase setting. Liang et al.
[39] assume the online retailer will stop recommending products with a certain probability 1 − w t to avoid the cost brought by marketing fatigue (i.e., overexposure to unwanted marketing messages [13]).
As a result, the parameter w t is a part of the ranking policy instead of consumers’ characteristics and should be optimized to achieve maximal revenue.
In our paper, we focus on the multiple purchases with budgets setting and leave the extension to the case with the marketing fatigue as future work. According to Theorem 4.1, the online retailer can provide the optimal ranking policy σ ∗ by sorting the products in a descending order with value ( λ k r k ) / (1 − q + q (1 − s ) λ k ) for the k -th product.
The time complexity is O
( N log N ) .
Online learning of the ranking policy
In this section, we consider a more realistic scenario where the online retailer has no prior knowledge about consumers’ characteristics.
At each timestamp, a consumer arrives and the online retailer can only provide proper product ranking policies based on the feedbacks of previous consumers.
As a result, it is essential for the retailer to design online learning algorithms to model consumers’ behaviors and maximize the revenue in the meantime.
We consider two settings, i.e. , the noncontextual setting where all consumers share the same parameters (Section 5.1) and the contextual setting where consumers have personalized behaviors (Section 5.2).
Before introducing the details of the algorithms, we first formally define the notations to characterize the consumers’ behaviors.
Notations for consumers’ behaviors Let Φ t be the random variable that denotes the amount of products viewed by the t -th consumer and Γ t = { γ t,k } Φ t k
=1 be the purchase activities.
Specifically, γ t,k = 1 if
and only if the product at the k -th position ( i.e. , σ t,k ) is purchased.
Let random variable η t,k denote whether the consumer continues viewing products after she views the k -th product but does not buy it and η t = { η t,k } Φ t k =1 .
Let random variable µ t,k denote whether the consumer continues viewing products if she buys the k -th product and µ t = { µ t,k } Φ t k
=
So η t,k is observed only when γ t,k = 0 and µ t,k is observed only when γ t,k = 1 .
Let Ber( · ) denote the Bernoulli distribution and we have the following result on the distribution of these random variables.
Lemma 5.1.
γ t,k are i.i.d.
random variables and γ t,k ∼ Ber (cid:0) λ σ t,k (cid:1) .
η t,k are i.i.d.
random variables and η t,k ∼ Ber( q t ) .
µ t,k are i.i.d.
random variables and µ t,k ∼ Ber( q t s t ) .
Lemma 5.1 is the basis of the algorithms introduced later that estimate parameters λ , q , and s from historical data.
In addition, compared with s t , it is easier to estimate q t s t from data and we denote it as w t ≜ q t s t .
5.1 Non-contextual online setting
In this subsection, we assume that all consumers share the same parameters, including the purchase probabilities on different products and the parameters on the distribution of the attention span and purchase budget.
To be specific, we assume that there exist λ , q, s such that λ t = λ , q t = q , and s t = s .
A) Estimation of parameters
To estimate the parameters at timestep t , we leverage the observed behavior of consumers before timestep t .
We first calculate the following statistics that can help provide unbiased estimators of the consumer parameters.
In detail, let C t,k be the total number of times that the product k is observed and c t,k be the number of times the product k is purchased, i.e. , C t,k ≜ (cid:80) tu =1 (cid:80) Φ t i
=
I [ σ t,i = k ] and c t,k ≜ (cid:80) tu =1 (cid:80) Φ t i =
I [ σ t,i =
k ] I [ γ t,i = 1] .
In addition, Let D qt be times that consumers view a product and not purchases it and d qt be the times that consumers continue viewing the list after viewing a product without purchasing it, i.e. , D {+qt ≜ (cid:80) tu =1 (cid:80) Φ t i =
+}
{+I [ γ t,i = 0] and d qt ≜ (cid:80) tu =1 (cid:80) Φ t i =
 I [ γ t,i = 0] η t,i .+} Furthermore, let D wt be times that consumers purchase a product and d wt be the times that consumers continue viewing the list after purchasing a product, i.e. , D wt ≜ (cid:80) tu =1 (cid:80) Φ t i =
I [ γ t,i = 1] and d wt ≜ (cid:80) tu =1 (cid:80) Φ t i =
I [ γ t,i = 1] µ t,i .
With these statistics, the parameters λ , q , and w = qs are estimated as follows.
[Equation]
According to the Hoeffding’s inequality, we provide the following proposition that gives the error bound of our parameter estimation approach.
Proposition 5.2.
For any t ≥ 1 , with probability at least 1 − 2( N + 2) t − 2 , the following holds:
[Equation]
B) Algorithm
Following the classic optimism in the face of uncertainty principle [1], we design a UCB-like algorithm that learns consumer behaviors and maximizes revenue simultaneously.
In detail, according to Proposition 5.2, we learn the optimistic estimators of the parameters given as follows
[Equation]
Let σ t be the optimal offline policy from Theorem 4.1 with λ = ˜ λ t − 1 , q = ˜ q t − 1 , and s = ˜
w t − 1 / ˜
q t − 1 . t t , t , η t , µ t .
As shown in Algorithm 1, when the t -th consumer arrives, the online retailer displays the products based on Theorem 4.1 with ˜ λ
t − 1 , ˜ q t − 1 , and ˜ w t − 1 / ˜
q t − 1 shown in Step 4.
Afterward, the online retailer updates the statistics according to consumers’ feedbacks as shown in Steps 5 and 6.
The optimistic estimators are then calculated in Step 7.
C) Regret analysis
To theoretically analyze the performance of Algorithm 1, we first introduce the regret, which measures the total difference between the maximal revenue achieved by σ ∗ and the cumulative reward by an online algorithm after T rounds.
[Equation]
Reg T is a random variable and the randomness comes from the uncertainty in consumers’ behaviors.
As a result, we focus on E [Reg T ] , which is a common routine in bandit literature [45].
The performance of the algorithm is given by the following theorem.
Theorem 5.3.
Under Assumption 3.1 (with parameter ϵ q ), for any T > 1 , the regret achieved by Algorithm 1 is bounded by [Equation] (10) where C is a absolute constant (independent of problem parameters) and r max = max k ∈ [ N ] r k .
Remark.
We analyze different parameters’ impacts on the regret bound in Equation (10). Firstly, the regret grows at ˜ O ( √ T ) , which is a standard result in online learning algorithms [45]. Secondly, the regret bound linearly related to N , which is similar to the results in cascade bandit models [53] and revenue management literature [17]. Finally, the bound depends on r max /ϵ q , which measures the maximal expected revenue the online retailer could achieve from each consumer.
5.2 Contextual online setting
We consider a more realistic scenario where the consumers’ behaviors are various and depend on their own features.
In detail, let x t ∈ R m x be the feature of the t -th consumer with dimension m x and y t,k ∈ R
m
y be the joint feature of the t -th consumer and k -th product with dimension m y .
y t,k can be the concatenation of consumer features and product features or fusion through a transformation such as matrix multiplication.
In this paper, we adopt concatenation for simplicity.
Let ∥·∥ denote the ℓ 2 norm.
Following [17], we consider the classic linear bandit setting [7, 21, 38] and the ground-truth data-generating process is given by the following assumptions.
Assumption 5.1.
There exists constant β λ ∈ R m y , β q ∈ R m x , and β s ∈ R m x , such that
[Equation]
Assumption 5.2.
For any t > 0 and k ∈ [ N ] , ∥ x t ∥ ≤ 1 and ∥ y t,k ∥ ≤ 1 .
In addition, there exists a constant U > 0 such that ∥ β λ ∥ , ∥ β q ∥ , ∥ β s ∥ ≤ U .
Remark.
These two assumptions are common in revenue maximization and linear bandits [1, 45].
Assumption 5.1 can be satisfied with kernel functions for complex data-generating processes and
Assumption 5.2 can be achieved by normalization.
Thus the assumptions are rational and easily attainable in real applications.
A
Estimation of
parameters
For the estimation of λ t and q t , we use the ridge regression method to estimate β λ and β q .
Given the observation of the t -th consumer’s behaviors, the estimation ˆ β t,λ , ˆ β t,q is given by
[Equation]
Here Σ t,λ =
[Equation] tu [Equation]
Φ
[Equation]
=
[Equation]
[ γ
[Equation]
= 0] x u
[Equation]
x
[Equation]
and
[Equation] tu [Equation]
Φ
[Equation]
k =
[Equation]
I [ γ
[Equation]
k = 0] η u,k x u . I m x and I m y are the identity matrices with dimension m x and m y , respectively. For the estimation of s t , similar to Section 5.1, we estimate w t =
q t s t instead.
Note that w t =
q t s t = x
⊤ t β q β ⊤ s x t = vec( x t x ⊤ t ) ⊤ vec( β q β ⊤ s ) , where vec( · ) denotes the vectorization operation that maps a matrix to a vector.
As a result, define z t ≜ vec( x t x ⊤ t ) and β w ≜ vec( β q β ⊤ s ) and we can get w t =
z ⊤ t β w .
Similarly, we estimate β w by conducting ridge regression as follows
[Equation]
Here Σ
[Equation]
I m 2 x is the identity matrix with dimension m 2 x .
Let ∥ β
∥
A = (cid:112) β ⊤ Aβ for any positive definite matrix A .
Proposition 5.4 gives the estimation error bound.
Proposition 5.4.
Under Assumptions 5.1 and 5.2 (with parameter U ), for any t ≥ 1 , with probability at least 1 − 3 t − 2 , the following holds:
[Equation]
Here τ ( t, m, α ) =
[Equation]
B)
Algorithm
Given the estimation of β λ , β q and β w we estimate λ t,k , q t and w t as follows.
[Equation]
Here Proj [0 , 1] ( · ) means to project the input into the interval [0 , 1] .
As shown in Algorithm 2, when the t -th consumer arrives, the online retailer observes the consumer features x t and y t,k for k ∈ [ N ] before learns the intermediate feature z t and calculate ˜ λ t,k , ˜ q t , and ˜ w t according to Equation (16).
Then the ranking policy σ t is given as shown in Step 6.
Then the statistics are calculated in Step 6 and 7 and finally the parameters ˆ β t,λ , ˆ β t,q , and ˆ β t,w are estimated in
Step 9 for the next iteration.
C) Regret analysis
We analyze the performance of Algorithm 2 by investigating its regret bound.
For a sequence of T consumers with features X T = { x t } Tt =1 and Y T = {{ y t,k } Nk =1 } Tt =1 , the regret is given by
Reg T ( X T , Y T ) =
T (cid:88) t =
Revenue ( σ ∗ t ; λ t , q t , s t ) − Revenue ( σ
[Equation]
Observe consumer features x t and y t,k for k ∈ [ N ] .
Let z t
=
vec
x t x ⊤ t ) .
Calculate λ t,k , ˜ q t ,
˜ w t according to Equation (16).
Let σ t be the optimal ranking policy from Theorem 4.1 with λ = ˜ λ
t , q = ˜ q t , and s = ˜ w t / ˜ q t .
σ t
Φ t , Γ t , η t , µ t .
estimated
Equations
10 end
Similar to the non-contextual setting, Reg T ( X T , Y T ) is a random variable and we focus on E [Reg T ( X T , Y T )] .
As a result, the performance of the algorithm is given by the following theorem.
Theorem 5.5.
Under Assumptions 3.1 (with parameter ϵ q ), 5.1, and 5.2 (with parameter U ), for any
T > 1 , the regret achieved by Algorithm 2 is bounded by
[Equation]
[Equation]
Here C is a absolute constant (independent of problem parameters), r max = max k ∈
[ N ] r k , and χ ( T, m, α ) = [Equation] [Equation] m
[Equation]
Remark.
We analyze different parameters’ impacts on the regret bound in Equation (19).
Firstly, the regret grows at ˜ O ( √ T ) , a standard result in online learning algorithms [45].
Secondly, the regret depends ˜ O ( N ) , which shares a similar result with the revenue management literature [17].
Thirdly, similar to Theorem 5.3, the bound depends linearly on r max /ϵ q that measures the maximal expected revenue the online retailer could achieve from each consumer. Fourthly, the bound depends on the dimensions of features ( i.e. , m x and m y ) by ˜ O ( √ m y ) and ˜ O ( m x ) because the feature dimension for the estimation of w t = q t s t is m 2 x
and then the result is consistent with [17, 53]. Finally, the bound depends on O ( U 2 + U ) since ∥ β w ∥ ≤ U 2 and ∥ β q ∥ , ∥ β λ ∥ ≤ U , which agrees with [17].  Experiments In this section, we conduct experiments on synthetic data to verify the performances of the online ranking policies.
Reports and discussions of more results including the expected revenue and the ratio over the expected revenue of the optimal ranking policy are in Section B.
Baselines We adopt the following baselines. Firstly, Cao and Sun [13] considered the single purchase setting, which is the simplified version of our multiple purchase setting. We denote it as Single Purchase .
Secondly, we adopt the method in [39] (denoted as Keep Viewing ), which assumes that consumers always keep viewing the ranking list after purchasing products.
Furthermore, we implement two explore-then-exploit algorithms (denoted as Explore Then Exploit A and Explore
Then Exploit B , respectively) to verify the advantages of our method that could balance exploration and exploitation.
6.1 Synthetic data
Data-generating processes For the non-contextual setting
, we consider N = 50 and 300 products and T = 100 , 000 consumers. The revenue for each product r k is uniformly distributed in [0 , 1] and the purchase probabilities for each product are uniformly sampled from [0 , 0 . 3] . We set q = 0 . 9 and s = 0 .
5 and 0 . 8 to test the performance of different models on characterizing consumers’ behaviors with multiple purchases. [Figure] For the contextual setting, we consider N = 50 and 300 products and T = 100 , 000 consumers.
The revenue r k is also uniformly distributed in [0 , 1] . Both the dimension of consumer features and products features are set to 5 ( i [Figure] e. , m x = 5 ).
The joint feature y t,k (the feature of the t -th consumer and k -th product) is a concatenate of the corresponding consumer and product feature so that m y = The consumer features and product features are uniformly sampled from [0 . 8 / √ m x , 1 / √ m x ] and [0 , 1 / √ m x ] .
Afterward, the coefficients β λ , β q , and β s in Equation (11) are uniformly sampled from [0 , 1] . To ensure λ t,k , q t , s t have similar range in the contextual setting, we normalize the coefficients such that the maximal elements in λ t,k and q t (denoted as λ max and q max ) are 0 . 3 and 0 . 9 , respectively. Similarly, by normalizing β s , we set the maximal element in s t (denoted as s max ) to 0 .
5 and 0 . 8 . Results and analysis
We implement the experiments for 5 distinct simulations by resampling consumers’ behaviors while their basic characteristics remain unchanged.
Specifically, in both the non-contextual and contextual settings, the purchase probabilities ( i.e. , λ t ) and parameters on the span attention and purchase budget ( i.e. , q t and s t ) are the same. However, the same consumer in distinct simulations may purchase different products due to randomness. We then evaluate our method and baselines and present the regret via calculating the regret (according to Equations (9) and (17)). The results are shown in Figure 1 and our method outperforms all baselines in both settings with various parameters.
On the one hand, the Single Purchase and Keep Viewing baseline considers different consumers’ behaviors settings and is not directly applicable here for the more realistic multiple purchase setting. On the other hand, compared with the explore-thenexploit based algorithms, our method integrates the advantages from traditional UCB-like algorithms, leading to the better exploration-exploitation trade-off for smaller regret. In addition, as introduced in Section B, Explore Then Exploit B incorporates learning processes during the exploration phase, leading to a better performance compared with Explore Then Exploit A.
6.2 Semi-synthetic dataset
We utilize the Ad Display / Click Data from Taobao 1 , which contains ad display / purchase logs ( million records) of 1,140,000 anonymized users from the website of Taobao for 8 days.
Because the behavior logs only include the purchase information of the users on a category of a brand, we view a (brand, category) pair in the dataset as a product and the average prices of the ads in this pair as the revenue for the product. Because we can not obtain consumers’ behaviors when we offer a new product ranking policy, following [14], we first estimate the parameters ( i.e. , λ , q , s in the non-contextual setting and β λ , β q , β s in the contextual setting) with all of the data. Afterward, we use the estimated parameters to simulate consumers’ behaviors when we provide them with different ranking lists of products.
[Figure]
Data processing Since each consumer may launch and leave the platform many times during thedays, we need to distinguish different launching activities of the same consumer.
Specifically, we suppose that a consumer stops viewing the product list if it has been more than 10 minutes since her last interaction with the platform. Afterward, we can estimate the parameters in both non-contextual and contextual settings. In the non-contextual setting, we calculate the statistics C t,k , c t,k , D qt , d qt , D wt , and d wt from the data and estimate λ , q , s directly according to Lemma 5.1. The estimation is q = 0 .
870 and s = 0 . Afterward, we choose N = 50 and 300 products with prices no more than 200 and purchase probabilities between [0 . 05 , 0 . 2] . Similar to the synthetic data, we set T = 100 , 000 .
In the contextual setting, we set the gender, age level, consumption grade, and shopping level as consumers’ features and the amount of the ads, the logarithm of the average, standard deviation, maximum, and median of the ads’ prices in the (brand, category) pair as products’ features. With the statistics C t,k , c t,k , D qt , d qt , D wt , and d wt , we use the ridge regression with regularization strength 1 to estimate β λ , β q , and β s . Afterward, we sample N = 50 and 300 products whose prices are no more than 200 and sample T = 100 , 000 consumers. Results and analysis Similar to the synthetic experiments, we implement the experiments fordistinct simulations by resampling consumers’ behaviors while their basic characteristics remain unchanged.
The results of the regret are shown in Figure 2. As shown in the figure, our method outperforms all baselines in different scenarios. Firstly, the Single Purchase and Keep Viewing baselines are not directly applicable here since they consider different consumers’ behavior settings.
It is worth mentioning that the estimated s is large from the dataset. As a result, the consumers prefer to keep viewing products if they purchase any product, making the Keep Viewing baseline outperforms Single Purchase in this setting. Secondly, our method combines the benefits of existing UCB-like algorithms, resulting in a superior exploration-exploitation trade-off with less regret compared with explore-then-exploit-based methods.
In addition, similar to the analysis in the synthetic experiments,
Explore Then Exploit B outperforms Explore Then Exploit A in most cases.
Conclusion
To conclude, we propose a novel setting to deal with multiple-purchase activities of consumers on online scenarios.
We characterize the optimal ranking policy and further design UCB-like algorithms with ˜ O ( √ T ) regret based on them in both non-contextual and contextual online settings.
We conduct extensive experiments to prove the effectiveness of our method.
Acknowledgments
Peng Cui’s research was supported in part by National Key R&D Program of China (No.
2018AAA0102004, No. 2020AAA0106300), National Natural Science Foundation of China (No.
U1936219, 62141607), Beijing Academy of Artificial Intelligence (BAAI).
Bo Li’s research was supported by the National Natural Science Foundation of China (No.72171131), the Tsinghua University Initiative Scientific Research Grant (No. 2019THZWJC11), Technology and Innovation Major
Project of the Ministry of Science and Technology of China under Grants 2020AAA0108400 and 2020AAA0108403.
One potential limitation is that we suppose the distribution of the attention span and purchase budget obey the geometric distribution. While the assumption is common [13, 39], we can relax it in future work, similar to [17, 14]. This paper does not have critical negative societal impacts because the purpose of the paper is to design an optimal ranking policy for online retailers.
Besides the online retailers, consumers may also benefit because Equation (5) indicates that a product with a higher purchase probability may appear in a higher order in the ranking list.
As a result, consumers may find the products shown in the ranking list highly relevant.
B More experimental results B.1 Implementation details Baselines
• Single Purchase [13].
Cao and Sun [13] supposes that the consumers always purchase at most one product.
As a result, we implement them by setting ˜ s t = 0 for all consumers. The estimations of ˜ λ t,k and ˜ q t are the same with our method. • Keep Viewing [39]. Liang et al.
[39] supposes that the consumers always keep viewing the list if they purchase one product.
To get the optimal ranking policy with the knowledge of consumers’ characteristics, we can sort the products with value ( λ k r k ) / (1 − q − (1 − q ) λ k ) [39, Theorem 1]. Afterward, we use similar techniques to estimate ˜ λ t,k and ˜ q t and maximize the revenue in the online settings. • Explore Then Exploit A . Following [13], we implement two explore-then-exploit based algorithms as baselines.
For the Explore-then-exploit-A method, we first adopt an exploration stage to ensure that each product is displayed to consumers at least δ log T times. (Here δ is a hyper-parameter.) To be specific in the exploration stage, we rank the products according to their current amount of display.
Afterward, we use the estimated parameters to provide the optimal ranking policy according to Theorem 4.1 in the main body of the paper. • Explore Then Exploit B . This method is a variant of the Explore-then-exploit-A method. In detail, in the exploration stage, after ranking the products according to their amount of display, we further rank the products that have been displayed to consumers at least δ log
T times according to Theorem 4.1 in the main body. Hyper-parameters We add hyper-parameter ξ λ , ξ q , and ξ w to determine the level of exploration for our method and two baselines (Single purchase and Keep viewing). The other two baselines do not include the hyper-parameter ξ because they do not have the dynamic exploration step. In detail, in the non-contextual setting, the parameters ˜ λ t,k , ˜ q t , and ˜ w t of our method are calculated as ˜ λ t,k ≜ min
[Equation]
In the contextual setting, the parameters are also calculated according to the hyper-parameter ξ
[Equation]
For the synthetic dataset, we search the hyper-parameter ξ λ ∈ { 0 . 1 , 0 . 3 , 0 . 5 } , ξ q , ξ w ∈ { 0 . 05 , 0 . 1 , [Figure] A v e r a g e r e v e nu e [Figure]
[-Φ u k =
-][Figure] R e v e nu e r a t i o [Figure]
We further report two more metrics to verify the effectiveness of our method. In detail, Average revenue ( T ) measures the average revenue achieved by different algorithms till the T -th consumer.
Revenue ratio ( T ) measures the ratio of the achieved revenue over the optimal ranking policy. Formally, the metrics are calculated as follows. [Equation] For both metrics, higher values indicate better performances.
The results of these two metrics on both the synthetic and Semi-synthetic datasets are shown in Figures 3, 4, 5, and 6.
This figures further demonstrate that our method outperforms all baselines in both contextual and non-contextual settings with various parameters ( i.e. , s , s max , and N ).
[Figure]
A v e r a g e r e v e nu e
[Figure]
[Figure]
R e v e nu e r a t
i o
[Figure]
C Omitted proofs C.1 Proof of Theorem 4.
Proof.
We drop all the subscript t for simplicity.
We first write the expected revenue function.
Revenue ( σ ; λ , q, s ) =
[Equation]
∼ Geo(1 − s )
[ R ( V, B ; λ , σ )] =
N (cid:88) v =
N (cid:88) b =
P ( V = v ) P ( B = b )   (cid:88)
Q ⊆ [ v ] , | Q |≤ b
P buy ( Q ; v, b, λ , σ )  (cid:88) k ∈ Q r σ k     = N (cid:88) k = r
σ k   N (cid:88) v = N (cid:88) b = P ( V = v ) P ( B = b )   (cid:88)
Q ⊆ [ N ] ,k ∈ Q P buy ( Q ; v, b, λ , σ ) 
 
 = N (cid:88) k
= r σ k  
N (cid:88) v = k N (cid:88) b =
P ( V = v ) P ( B = b )   (cid:88) Q ⊆ [ N ] ,k ∈ Q P buy ( Q ; v, b, λ , σ ) 
   .
For any Q k ⊆ [ N ] that satisfies k ∈ Q k , define Q − k ≜ { i ∈ Q k : i < k } and Q + k ≜ { i ∈ Q k : i > k } . We further define  
[Equation]
(cid:40)(cid:16)(cid:81) i
[Equation]
max
[Equation] λ
[Equation]
) (cid:17) , if | Q + k | > 0 , 1 , if | Q + k | = 0 .
As a result,
N [Equation]

[Equation]   .
Here P buy k +1 (cid:0) Q + k ; v, b − | Q − k | − 1 , λ , σ (cid:1) in the second equality denotes the probability of purchasing
Q + k with attention span v purchase budget b after purchasing | Q − k | + 1 [-/ 
-]products, i.e.
P buy k +1 (cid:0) [Equation] [Equation]
[Equation] = [Equation] As a result, when | Q − k | , v , b , λ , and σ are fixed, (cid:88)
[Equation]
Hence, λ σ k
N (cid:88) v = k
N
[Equation]
b =
P (
[Equation]
= v ) P ( B = b )   (cid:88) Q − k ⊆
[Equation]
− 1] (cid:88)
[Equation] [Equation]
[ k ] G
[Equation]
  = λ
σ k
N (cid:88) v = k N (cid:88) b = P ( V = v ) P ( B = b )   b − 1 (cid:88) u =   (cid:88)
Q
[Equation]
G
[Equation] ) 
   (cid:88)
Q +
[Equation]
⊆
[Equation] [Equation] P buy [Equation]
Q [Equation] k ; v, b − u −
[Equation]
λ
[Equation]
σ (cid:1)     = λ σ k N (cid:88) v = k
N (cid:88) b = P ( V = v ) P ( B = b )   b − 1 (cid:88) u
=   (cid:88)
Q − k [Equation]
For any k ≥ 1 and 0 ≤ u ≤ k − 1 , define [Equation] [Equation] As a result,
Revenue ( σ ; λ , q, s ) =
[Equation]
In addition, it is easy to verify
that H σ ( k, u ; λ ) has the following recurrence relation H σ (1 , 0; [Equation] = 1 .
[Equation]
From the definition of G − k and H σ , it is easy to check that the value of H σ ( k, u ; λ ) does not depend on the order of σ 1 , σ 2 , . . .
, σ k − 1 .
As a result, for any 1 ≤ i < N , let σ be a ranking policy that differs from σ ∗ only in the i -th and ( i + 1) -th position, i.e. ,
[Equation]
Due to the optimality of σ ∗
[Equation]
we have
[Equation]
Hence,
[Equation]
Now the claim follows.
C.2 Proof of Lemma 5.
Proof.
The claim holds obviously for γ t,k .
For η t,k , suppose the consumer purchased u products after viewing the k -th product, then
[Equation]
As a result, P ( η t,k = 1) does not depend on the historical purchase activities and is always q t .
Hence, η t,k are i.i.d.
Bernoulli random variables and η t,k ∼ Ber( q t ) .
Similarly, for µ t,k , suppose the consumer purchased u products after viewing the k -th product, then
[Equation]
As a result, P ( η t,k = 1) does not depend on the historical purchase activities. Hence, µ t,k are i.i.d. Bernoulli random variables and µ t,k ∼ Ber( q t s t ) . C.3 Proof of Proposition 5.
Proof.
According to Lemma 5.1 and Hoeffding’s inequality, for any t > 0 and 1 ≤ k ≤ N , we have
[Equation]
Now the claim follows.
C.4 Proof of Theorem 5.
We first define
[Equation]
We also need the following two propositions. Proposition C.1. Suppose ˜ λ ⪰ λ , ˜ q ≥ q , and ˜ s ˜ q ≥ sq , then [Equation]
Proof of Proposition C.1.
For any σ ⊣ [ N ] , 1 ≤ n ≤ k , 0 ≤ u ≤ k − n , define
[Equation]
As a result
[Equation]
J σ (1 , k, u ; λ ) =
H σ ( k, u ; λ ) .
In addition
[Equation]
it is easy to check that
[Equation]
In addition
[Equation]
for any n ≥
[Equation]
define
[Equation]
[Equation] It is easy to verify that R σ (1; λ , q, s ) = Revenue( σ ; λ , q, s ) .
Furthermore,
[Equation]
Now we prove R σ ∗ ( n ; λ , q, s ) ≤ R σ ∗ ( n ; ˜ λ , ˜ q, ˜ s ) , ∀ n ∈ [ N ] by induction.
Firstly, it is easy to check that R σ ∗ ( N ; λ , q, s ) ≤ R σ ∗ ( N ; ˜ λ , ˜ q, ˜ s ) .
Suppose the condition holds for R σ ∗ ( n ; λ , q, s ) ≤
R σ ∗ ( n ; ˜ λ , ˜ q, ˜ s ) .
Consider the case for n − 1 .
Notice that R σ ∗ ( n − 1; λ , q, s ) ≥ R σ ∗ ( n ; λ , q, s ) (otherwise it will increase the revenue if removing the ( n − 1) -th product in the list), we can get that
[Equation]
[Equation]
Hence r σ n − 1 − q (1 − s ) R σ ∗ ( n ; λ , q, s ) ≥ 0 .
As a result,
[Equation]
As a result
[Equation]
Revenue
[Equation]
[Equation]
[Equation]
Proposition C.2. Under Assumption 3.1, suppose ˜ λ ⪰ λ , 1 − ϵ q ≥ ˜ q ≥ q , and ˜ s ˜ q ≥ sq , then Revenue (cid:16) σ t ; ˜ λ , ˜ q, ˜ s (cid:17) − Revenue ( σ t ; λ , q, s ) ≤ r max ϵ q N (cid:88)
[Equation]
[Equation]
Proof of Proposition C.2.
For simplicity, we use σ to denote σ t in this proof. According to Equation (30) [Equation] and H σ ( k, u ; λ ) is defined in Equation (29).
As a result
Revenue (cid:16)
[Equation]
Note that
[Equation]
E (cid:2)
[Equation]
For any 1 ≤ k ≤ N ,
[Equation]
Here the first inequality is due to the triangle inequality of | · | and the second inequality is according to the fact
that ˜ λ
σ k ≤ 1 .
Now for any 1 ≤ k ≤ N , define
[Equation]
We can get
[Equation]
As a result
[Equation] for any 1 < k ≤ N [Equation] Here the last inequality is due the triangle inequality of | · | .
Note that [Equation] [Equation] Here the first two inequalities are due to the triangle inequality of | · | and the third inequality is due to that fact that ˜ q, ˜ s ≤ 1 .
Combining Equations (56) and (57), we can get
[Equation]
) Combining the above inequality with Equation (49) and (53)
[Equation]
we can get that
[Equation]
[Table]
[Equation] Now the claim follows [Equation] Proof of Theorem 5.3.
For each timestamp t , define the favorable event as
[Equation] According to Proposition 5.
[Equation]
we have P ( ξ t ) ≥ 1 − 2( N + 2) t − 2 . Let ξ Ct denote the complement event of ξ t .
[Equation] On the one hand, consider the first term in the RHS of Equation (61), when the event ξ t − 1 is satisfied, we have q ≤ ˜ q t − 1 , z ≤ ˜ w t − 1 , and λ k ≤ ˜ λ k , ∀ k ∈ [ N ] . As a result, according to Proposition C.1, Revenue
[Equation] ; λ [Equation] q
[Equation]
s )
[Equation] Revenue [Equation] Hence,
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
ϵ
[Table]
Let χ Vt ≜ (cid:80) N k =1 O qt,k ≤ N
and we have D t,v = D V t − 1 + χ Vt .
As a result, E (cid:34) T (cid:88) t = N (cid:88) k = max (cid:40)(cid:115)
D qt − 1 ,(cid:41)
O qt,k (cid:35) =
E (cid:34) T (cid:88) t = max (cid:40)(cid:115)
D qt − 1 ,(cid:41) χ Vt (cid:35) ≤ 1 + N +
T (cid:88) t =
N (cid:115) ( t − 1) N ≤ 1 + N + √ N (cid:32)
[Equation]
for some constants C 1 and C 2 .
Similarly
[Equation] we have [Equation] In addition
[Equation]
let o λt,k ≜ O λt,σ − 1 t,k represent whether the product k is viewed in the t -th timestamp. As a result, C t,k = C t − 1 ,k + o λt,k .
[Equation]
[Equation] for a constant C 3 . On the other hand, [Equation]
≤
[Equation]
for a constant C 4 .
Now the claim follows after combining Equations (63), (64), (65), (66), and (67).
C.5 Proof of Proposition 5.
Proof.
The claim follows from standard routines in linear bandit methods.
We first consider the ˆ β t,λ − β λ term.
We can view the tuple ( t, k ) as the new timestamp.
Let ϵ t,k ≜ y ⊤ t,σ t,k β λ − γ t,k .
According to Lemma 5.1, E [ ϵ t,k | y t,σ t,k , H t,k ] = 0 where H t,k denotes the historical observations before the t -th consumer views the k -th product. In addition, it is bounded in [ q t − 1 , q t ] and hence sub-gaussian with R = 1 / 2 . As a result, according to [1, Theorem 2], under Assumption 5.2 with parameter U , for any δ ∈ (0 , 1) , for any t ≥ 0 , [Equation]
According to [1, Lemma 10], we have
[Equation]
As a result, by letting δ = t − 2 , for all t ≥ 0 , at least 1 − t − 2 ,
[Equation]
Similarly
[Equation]
we have
[Equation]
In addition, because [Equation] we have [Equation]
Now the claim follows
[Equation] C.6 Proof of Theorem 5.  Proof. Similar to the proof of Theorem 5.3, for each timestamp t
[Equation] define the favorable event as ξ [Equation] [Table]
[Equation] [Table] [Equation] According to Proposition 5.4, we have P ( ξ t ) ≥ 1 − 3 t − 2 .
Let ξ Ct denote the complement event of ξ t .
[Equation]
On the one hand, consider the first term in the RHS of Equation (75), when the event ξ t is satisfied, we have q t ≤ ˜ q t ,
z t ≤ ˜ w t , and λ t,k ≤ ˜ λ t,k , ∀ k ∈ [ N ] .
As a result, according to Proposition C.1,
Revenue ( σ ∗ t ; λ t , q t , s t ) ≤ Revenue (cid:16) σ ∗ t ; ˜ λ t , ˜ q t , ˜ s t (cid:17) ≤ Revenue (cid:16) σ t ; ˜ λ t , ˜ q t , ˜ s t (cid:17) . Hence, [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
k =
[Table]
[Equation] We consider the [Equation] term first and this part is similar to the proof of [50, Lemma 1].
Define g t,k = (cid:13) (cid:13) y t,σ t,k (cid:13) (cid:13) Σ − 1 t − 1 ,λ for all t ∈
[ T ] , k ∈ [ N ] that satisfy O λt,k = 1 . Because
[Equation]
we have for all 1 ≤ k ≤ Φ t , [Equation] As a result (cid:18) [Equation]
Hence, (det
[Equation]
Σ [Equation] ,λ )) [Equation]
Nm [Equation] On the other hand, due to Assumption 5.2, trace (Σ t,λ ) = trace
[Equation]
According to the [1, Lemma 10], we have trace(Σ t,λ ) /m
y ≥ (det(Σ t,λ )) 1 /m y . As a result,
[Equation] [Equation] g [Equation]
By taking the logarithm, we can get that
[Equation]
Since
[Equation]
we have g 2 t,k ≤ α − 1 λ log(1 + g 2 t,k ) / log(1 + α − 1 λ ) . As a result,
[Equation]
Hence, according to the Cauchy–Schwarz inequality
[Equation]
we have [Equation] As a result [Equation]
we have
[Equation]
Using similar techniques
[Equation]
we can prove that
[Equation]
On the other hand,
[Equation]
for a constant C 1 .
Now the claim follows after combining Equations (77), (88), (89), and (90).