P ROVABLY E FFICIENT R EINFORCEMENT L EARNING FOR O
NLINE
A DAPTIVE I NFLUENCE
M AXIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Online inﬂuence maximization aims to maximize the inﬂuence spread of a content in a social network with an unknown network model by selecting a few seed nodes.
Recent studies followed a non-adaptive setting, where the seed nodes are selected before the start of the diffusion process and network parameters are updated when the diffusion stops.
We consider an adaptive version of content-dependent online inﬂuence maximization problem where the seed nodes are sequentially activated based on real-time feedback.
In this paper, we formulate the problem as an inﬁnite-horizon discounted MDP under a linear diffusion process and present a model-based reinforcement learning solution.
Our algorithm maintains a network model estimate and selects seed users adaptively, exploring the social network while improving the optimal policy optimistically.
We establish (cid:101) O
( √ T ) regret bound for our algorithm.
Empirical evaluations on synthetic and real-world networks demonstrate the efﬁciency of our algorithm
I NTRODUCTION
Inﬂuence Maximization (IM) (Kempe et al., 2003; Kitsak et al., 2010; Centola & Macy, 2007), motivated by real-world social-network applications such as viral marketing, has been extensively studied in the past decades.
In viral marketing, a marketer selects a set of users (seed nodes) with signiﬁcant inﬂuence for content promotion.
These selected users are expected to inﬂuence their social network neighbors, and such inﬂuence will be propagated across the network.
With limited seed nodes, the goal of IM is to maximize the information spread over the network.
A typical IM formulation models the social network as a directed graph and the associated edge weights are the propagation probabilities across users.
Inﬂuence propagation is commonly modeled by a certain stochastic diffusion process, such as independent cascade (IC) model and linear threshold (LT) model (Kempe et al., 2003).
A popular variant is topic-aware IM (Chen et al., 2015; 2016) where the activation probabilities are content-dependent and personalized, i.e., edge weights are different when propagating different contents.
Classical inﬂuence maximization solutions are studied in an ofﬂine setting, assuming activation probabilities are given (Kempe et al., 2003; Chen et al., 2009; 2010).
However, this information may not be fully observable in many real-world applications.
Online inﬂuence maximization (Chen et al., 2013; Wen et al., 2017; Vaswani et al., 2017) has recently attracted signiﬁcant attention to tackle this problem, where an agent learns the activation probabilities by repeatedly interacting with the network.
Most existing works on online inﬂuence maximization are formulated as a multi-armed bandits problem making a non-adaptive batch decision: at each round, the seed nodes are computed prior to the diffusion process by balancing exploring the unknown network and maximizing the inﬂuence spread; the agent observes either edge-level (Chen et al., 2013; Wen et al., 2017; Wu et al., 2019) or node-level (Vaswani et al., 2017; Li et al., 2020) activations when the diffusion ﬁnishes and updates its model.
Combinatorial multi-armed bandits (Chen et al., 2013; Wang & Chen, 2017) and combinatorial linear bandits (Wen et al., 2017; Wu et al., 2019) algorithms have been proposed as solutions, where most works follow independent cascade model with edge-level feedback.
In contrast to the non-adaptive setting, adaptive inﬂuence maximization allows the agent to select seed nodes in a sequential manner after observing partial diffusion results (Golovin & Krause, 2011;
Tong et al., 2016; Peng & Chen, 2019).
The agent can achieve a higher inﬂuence spread since the decision adapts to the real-time feedback of diffusion.
In viral marketing, the agent could observe partial diffusion feedback from the customer and adjust the campaign for the rest of budgets based on current diffusion state.
Unfortunately, online inﬂuence maximization in an adaptive setting is under-explored.
Previous bandit-based solutions cannot be applied because the decisions of bandit algorithms are independent of the network state.
In this paper, we study the content-dependent online adaptive inﬂuence maximization problem: at each round, the agent selects a user-content pair to activate based on current network state, observes the immediate diffusion feedback, and updates its policy in real-time.
The network’s activation probabilities are content-dependent and are unknown to the agent.
The agent’s goal is to maximize the total inﬂuence spread.
We formulate this problem as an inﬁnite-horizon discounted Markov decision process (MDP), where the state is users’ current activation status under different contents (user-content pairs), an action is to pick a user-content pair as the new seed, and the total reward is the discounted sum of active user counts.
Speciﬁcally, we study the problem under the independent cascade model with node-level feedback.
Similar to combinatorial linear bandits (Wen et al., 2017;
Vaswani et al., 2017), we formulate a tensor network diffusion process where activation probabilities are assumed to be linear with respect to both user and content features.
To tackle the problem of node-level feedback, we propose a Bernoulli independent cascade model, a linear approximation to the classic IC model which requires edge-level feedback to learn.
We propose a model-based reinforcement learning (RL) algorithm to learn the optimal adaptive policy.
Our approach builds on prior work of bandit-based inﬂuence maximization algorithms (Chen et al., 2013; Wen et al., 2017; Wu et al., 2019) and has the following distinct features: (1) Our adaptive IM policy makes decisions and updates policy on the ﬂy, without waiting till the end of diffusion process; (2) Our algorithm takes into consideration real-time feedback from the network, thus approaching a dynamic-optimal policy and outperforming bandit-based static-optimal solutions; (3) Our algorithm learns from node-level feedback, which greatly relaxes the common edge-level feedback assumption in previous works with IC model; (4) Our policy can handle content-dependent networks and select the best content for the right user for the campaign; (5) To improve computation efﬁciency, we adopt the slow switching strategy (Abbasi-Yadkori et al., 2011) that only update model parameter for O ( d log T ) times, where d is the feature space dimension.
Our contributions are summarized as follows:
• We propose a linear tensor diffusion model for content propagation in social networks and formulate the problem as an inﬁnite-horizon discounted MDP.
• We propose a tensor-regression-based RL inﬂuence maximization algorithm with optimistic planning that learns an adaptive policy from node-level feedback, which selects the content and next seed user based on current state of the network.
• We proved a (cid:101) O
( d √ T/ ∆ + √ dNKT ) 1 regret of our algorithm, where T is the total rounds, N is the number of users, K is the number of contents, ∆ is the coefﬁcient for diffusion decay, d is the dimension related to user and content feature.
To our best knowledge, this is the ﬁrst sublinear regret bound for online adaptive inﬂuence maximization.
• We empirically validated on synthetic and real-world social networks that our algorithm explores the unknown network more thoroughly than conventional bandit methods, achieving larger inﬂuence spread.
Related Works.
The classical works on (ofﬂine) inﬂuence maximization (Kempe et al., 2003; Chen et al., 2009; 2010) assume the network model, i.e., the activation probabilities, is known to the agent and the goal is to maximize the inﬂuence spread, i.e., total number of activated users.
IM has been studied in a non-adaptive setting where the agent chooses the seed nodes before the diffusion starts (Kempe et al., 2003;
Chen et al., 2009; 2010; Bourigault et al., 2016; Netrapalli & Sanghavi, 2012;
Saito et al., 2008), or an adaptive setting where the agent sequentially selects the seed nodes adaptive to current diffusion results (Golovin & Krause, 2011; Tong et al., 2016; Han et al., 2018; Peng &
Chen, 2019; Tong & Wang, 2020).
Online inﬂuence maximization (Chen et al., 2013; 2015; Lei et al., 2015; Lugosi et al., 2019; Perrault et al., 2020; Zuo et al., 2022) is proposed to learn network model while selecting seed nodes in the non-adaptive setting.
Existing works on online IM studies mostly follow IC model and edge-level feedback (Chen et al., 2013; Wang & Chen, 2017; Wen et al., 2017;
Vaswani et al., 2017; Lugosi et al., 2019).
Chen et al.
and Wang & Chen (2017) formulated the online IM problem as combinatorial bandits problem and proposed combinatorial upper conﬁdence bound (CUCB) algorithm to estimate the activation probabilities of edges in a tabular manner.
Wen et al.
(2017) assumed a linear parameterization on each edge with known edge features and proposed a linear bandits-based solution.
Our paper is the ﬁrst to consider online inﬂuence maximization in the adaptive setting and formulate it as an RL problem.
We also can handle the more challenging node-level feedback.
Some recent works also explored settings beyond IC model and edge-level feedback.
Li et al.
(2020) studied online IM with linear threshold model, and proposed a linear bandits-based solution to model the linearity in LT model for node-level feedback.
We also leveraged the linearity in diffusion model to handle node-level feedback similar to Li et al.
but for
IC model.
Vaswani et al.
(2017) considered diffusion model-independent setting using a heuristic objective function, but without theoretical guarantee of the heuristic.
Olkhovskaya et al.
(2018) studied UCB-based algorithm for node-level feedback, but their algorithm is designed only for certain random graph models such as stochastic block models and Chung–Lu models.
Our analysis is related to regret analysis of model-based reinforcement learning, which have been studied in various settings such as tabular MDP (Auer et al., 2008), linear/kernel MDP (Yang &
Wang, 2020; Yang et al., 2020), factored MDP (Rosenberg & Mansour, 2021), general model class (Ayoub et al., 2020), etc.
We provide a ﬁrst problem-speciﬁc analysis for inﬂuence maximization.
Our analysis differs from existing regret analysis in a couple of ways.
First, although we focus on a linear model for network diffusion, the state-to-state transition of the IM is highly nonlinear , thus the value and Q functions for IM do not admit a linear model and invalidate linear/kernel MDP approaches.
Second, due to the nature of network diffusion process, the state and its value can grow unboundedly for large networks, causing unbounded variance at the same time.
Our analysis is specially tailored to such growth process over large networks and derive regret bound by focusing a high probability event where states stay bounded.
To our best knowledge, this is a ﬁrst IM-speciﬁc regret analysis for controlling unbounded growth process over large networks.
P ROBLEM F ORMULATION
We present a tensor network diffusion process to model user feature-dependent content featuredependent network propagation.
Our goal is to both select seed users and customize contents for inﬂuence maximization.
Further, we formulate IM into an RL problem to enable much more delicate control of the network diffusion process based on real-time feedback.
2. T ENSOR N ETWORK D IFFUSION P ROCESS
Consider a social network of N users, where the network structure may be hidden.
Let there be
K choices of contents.
Let s i,k ∈ { 0 , 1 } denote the status of an user-content pair, i.e., s i,k = 1 if user i is actively tweeting content k .
The full state of the network is denoted by s ∈ { 0 , 1 } N × K , a binary matrix.
We focus on the asymptotic regime of large networks , i.e., N can be arbitrarily large or even N → ∞ .
We assume that each content can be propagated from one user to multiple users following an independent network diffusion process.
Assumption 1 (Bernoulli Independent Cascade Model) .
Let s (cid:48) be the next state.
For each k ∈ [ K ] , we assume there is an underlying connectivity matrix A k ∈ R n × n such that
[Equation]
And we assume s (cid:48)
i,k ’s are independent conditioned on s .
Here A ki,j measures the level of inﬂuence user j has over user i for the k -th content.
Therefore, the aggregate “inﬂuence” received by user i is (cid:80) j A ki,j s j,k .
We model the status of user i as a Bernoulli variable, which is parameterized by the aggregate “inﬂuence” received by user i .
Our model is closely related to the independent cascade model (Kempe et al., 2003).
In IC model, the activation probability takes of the form P ( s (cid:48)
i,k = 1 | s ) = 1 − (cid:81) j
(1 − A ki,j s j,k ) .
A limitation is that efﬁcient estimation of IC model requires edge-level observations (Chen et al., 2013; Wang & Chen, 2017).
Assumption 1 can be viewed as an linearized approximation to IC model, i.e., 1 − (cid:81) j
(1 − A k
i,j s j,k ) ≈ (cid:80) j
A k i,j s j,k when all the A values are tiny (see Assumption 3).
In
Appendix F, we extend the assumption to the generalized linear setting and establish the regret bound for our algorithm.
Consider a parameterized network diffusion model based on user features and content features.
Let the i -th user be associated with a user feature vector x
i ∈ R d 1 , for all i ∈ [ N ] .
Let the k -th content be associated with a content feature θ k ∈ R d 2 , for all k ∈
[ K ] .
We assume that the inﬂuence is linear with respect to both user and content feature.
Assumption 2 (Content-Dependent Linear Tensor Model) .
There exists a d 1 × d 1 × d 2 tensor
T ∗ ∈ R d 1 d 1 d 2 such that
[Equation]
where ⊗ denotes outer product and (cid:104) , (cid:105) denotes inner product.
Note that this is different from the linear MDP model commonly studied in the theoretical RL literature (Jin et al., 2020).
We focus on large networks where N can be arbitrarily large.
We also assume each individual user has bounded inﬂuence over its neighbors and the diffusion process has a natural decay property.
Assumption 3 (Uniform transition probability upper bound) .
There exists a constant C > 0 such that (cid:107) A (cid:107) ∞
≤ C NK .
Assumption 4 (Diffusion decay) .
There exists ∆ > 0 such that (cid:80) i ∈
[ N ] A ki,j ≤ 1 − ∆ for all k, j .
Assumption 4 says that inﬂuence from any seed user has a discounting nature; without this assumption, some seed user may have inﬁnite-long inﬂuence and make the diffusion process unbounded.
This assumption also implies that, the “inﬂuence" of any seed user-content pair would last O (1 / ∆) time steps on average.
2. R EINFORCEMENT L EARNING M ODEL
We formulate the inﬂuence maximization problem as an inﬁnite-horizon discounted MDP.
Deﬁne the state space as S = { 0 , 1 } N × K where 1 refers to activated user-content pair.
At each timestep, the agent observes the current network state s and picks an action a ∈ A :=
[ N ] × [ K ] to activate one user-content pair.
Let s a be the post-action state, i.e., s a = s + 1 a .
Then the state of network transitions following the network diffusion process, i.e., Assumptions 1,2.
Since users are activated independently from one another, the state-transition law of the MDP admits a product structure:
P
[Equation]
(cid:89)
[Equation]
At each state-action pair, the agent receives a reward r ( s, a )
= (cid:80) i,k
v i,k s i,k measuring the amount of inﬂuence over the network.
For examples, if we let v i,k ≡ 1 , then we have r ( s, a ) = (cid:107) s (cid:107) 1 , which counts the number of active users.
Without lost of generality, we assume v i,k ≤ 1 .
Let π : S (cid:55)→
A be a decision policy.
We measure the value of policy π at state s as a cumulative sum of discounted rewards
[Equation]
Recall Assumption 4 that the inﬂuence of any action lasts 1 / ∆ time steps.
Thus, a natural choice of the discount factor to be γ = 1 − o (∆) .
Finally, the policy optimization problem is to ﬁnd π ∗ =
argmax π V π ( s ) .
Relation between Discounted MDP and Bandit IM model.
The discounted MDP formulation differs from the bandit IM optimization in two ways.
(1) Our policy is dynamic and makes statedependent decisions, while the bandit approach would make a batch of decisions only at the beginning of the diffusion process; (2)
In both cases, the optimization objectives are sums of total inﬂuences from all seed users.
The difference lies in how to measure the per-seed inﬂuence.
In IM bandit, the per-seed inﬂuence is a cumulative sum calculated after the diffusion process is over.
In our formulation, the per-seed inﬂuence is a cumulative γ -discounted sum of rewards from this seed’s descendants.
If we choose γ = 1 − o (∆) , these values differ by only o (1) and we can make the difference arbitrarily small.
M ODEL - BASED RL FOR I NFLUENCE M AXIMIZATION (M ORIMA )
To reduce the statistical complexity, we adopt a model-based RL approach for exploring the unknown network and learning the optimal policy.
Our approach alternates between model estimation and policy update.
Our algorithm calculates a bonus function based on the collected data and and add it to the reward, which dynamically trades-off between exploitation and exploration.
We also adopt a slow switching technique to reduce computational burden.
Tensor ridge regression for model estimate.
Under the linear tensor model (Assumption 2), we can use tensor ridge regression to perform model-based RL.
This reduces the statistical complexity since the dimension of the unknown parameter is smaller.
Furthermore, this approach only requires node-level feedback, While previous bandits approaches for IC model require edge-level feedback (Chen et al., 2013; Wen et al., 2017; Wu et al., 2019).
Speciﬁcally, let s a be the altered state after applying action a .
Observe that, conditioned on ( s, a ) , the random variable s (cid:48)
i,k satisﬁes a linear relation:
[Equation]
Denote for short φ i,k ( s, a ) = x i ⊗ (cid:16)(cid:80)
j x j · ( s a ) j,k (cid:17) ⊗ θ k ∈ R d 1 d 1 d 2 , and φ ti,k
= φ
i,k ( s t , a t ) .
At time t , after observing the history ( s 1 , a 1 , . . .
, s t − 1 , a t − 1 , s t ) , we estimate the tensor model by :
[Equation]
where (cid:107)T (cid:107) 22 is calculated by vectorizing T .
This allows an analytical solution:
[Equation]
where
[Equation]
Notice that the sizes of the covariance matrix Σ t − 1 and the right-hand-side term B t − 1 are d × d and d , respectively, where d = d 21 d 2 (cid:28) N .
Optimistic Planning with truncated-reward model.
To avoid the worst-case O ( NK ) reward, we identify a high probability upper bound Λ for the rewards and truncate the reward as (cid:101) r
( s, a ) =
min { r ( s, a ) , Λ } .
Then based on the ridge regression estimation (cid:98) T t , we add a bonus term b t ( s, a ) to the truncated reward (cid:101) r and solve for an optimistic Q-function Q ∗ (cid:98) T t , (cid:101)
r + b t ( s, a ) using the model estimate.
Speciﬁcally, we can choose Λ = 6∆ 2 log(4 NKT 3 ) .
For (cid:98) T t , we deﬁne the reward bonus as
[Equation]
where we use the notation 1 ∧ x = min { 1 , x } and
[Equation]
) with L being an upper bound of (cid:107) φ ti,k (cid:107) 2 and d = d 21 d 2 .
This choice of β t ensures with high probability, Q ∗ (cid:98) T t , (cid:101)
r + b t ( s, a ) is an upper bound of (cid:101) Q ∗ ( s, a ) , which is the optimal Q-function for ground-truth transition with truncated-reward.
We calculate the optimal truncated Q-function Q ∗ (cid:98) T t , (cid:101) r + b t ( s, a ) using value iteration with truncation (Algorithm 2).
Slow switching.
To reduce computation overhead, we adopt a slow switching technique from bandit and RL literatures (Abbasi-Yadkori et al., 2011; Zhou et al., 2021b).
The idea is that we only update model and policy when enough new data has been collected, via checking the covariance matrix.
Speciﬁcally, say the most recent switching happens at time t , we choose to switch at time t (cid:48) only if
[Equation]
After switching, we calculate the optimistic Q-function Q
[Equation]
=
[Equation]
+
[Equation]
( s, a ) .
Then we pick actions greedily using Q (cid:48) t , i.e., a = argmax a Q t (cid:48)
( s, a ) , until the next switching.
: else 11:
Set Q t =
Q t − 1 .
12: end if 13:
Take the greedy action with respect to Q t : a t = argmax a Q t ( s t , a ) .
14: end for
Q
( s a
( s a γ E s (cid:48) ∼ P T ( ·| s,a
) max a (cid:48) Q ( s (cid:48) , a (cid:48)
Full algorithm.
We put together the pieces and present the full Algorithm 1.
The algorithm makes only O
( d log( T )) model updates and policy updates until time T .
Each model update can be done efﬁciently using least square regression.
Policy updates require solving a new planning problem which can be combinatorially hard.
In practice, one can solve the planning problem using approximate dynamic programming (Powell, 2007) or Monte-Carlo Tree Search (MCTS) methods (Browne et al., 2012), and we will use a two-step lookahead scheme in our experiments.
For theoretical analysis, we assume access to a planning oracle that is able to ﬁnd the optimal policy with respect to a known model (cid:98) T .
Relaxing such assumption to an approximated planning oracle can be also be done with minor algorithmic and analysis modiﬁcations.
R EGRET A NALYSIS
In this section, we provide regret analysis for Algorithm 1.
We deﬁne the regret for the inﬁnite-horizon discounted MDP as in (Zhou et al., 2021b).
Deﬁnition 1.
For any possibly non-stationary policy π , the inﬁnite-horizon discounted regret is deﬁned as
Regret( T ) =
T (cid:88) t = ∆ t , where ∆ t = V ∗ ( s t ) − V πt ( s t ) , where V ∗ is the optimal value function, and V πt is deﬁned as
[Equation]
Now we present our main theorem.
Theorem 2.
Let Assumptions 1-4 hold.
With probability at least 1 − δ , Algorithm 1 satisﬁes the following regret upper bound:
[Equation]
where d = dim( T ∗ )
=
d 21 d 2 .
We see that the dominant term of the regret is (cid:101)
O
[Equation]
Notice that the worst-case reward would scale with NK , while we managed to reduce the scaling of the regret to 1 / ∆ 2 .
4. P ROOF SKETCH OF T HEOREM
Next, we provide a proof sketch and defer the complete proof to Appendix C.
Proof.
High probability upper bounds for the size of active user-content pairs.
We utilize the diffusion decay assumption (Assumption 4) to provide a high probability upper bound on the number of active user-content pairs.
We show that for any policy π , with probability at least 1 − p , we have for all t ≥ 1 ,
[Equation]
We see that although we have in total NK user-content pairs, the number of active ones is constrained by a constant intrinsic to the network diffusion dynamics.
Sharper bounds for the conﬁdence region.
We derive a batched version of Bernstein-type self-normalized bounds from (Zhou et al., 2021b) and show that with high probability for all t , (cid:107) (cid:98) T t − T ∗ (cid:107) Σ t − 1 ≤ β t , where β t can be chosen as (cid:101) O ( σ √ d + 1) and σ 2 is the upper bound of var[( s t +1 )
i,k | s t , a t ] .
Combing Eqn.
(7) and Assumption 3, we have var
[Equation]
Then β t = (cid:101) O
( (cid:113) dCNK / ∆ + 1) , which improves upon β t = (cid:101) O ( √ d ) given by the sub-Gaussian type self-normalized bounds.
Surrogate regret of the truncated-reward model.
Since we essentially run our algorithm against the truncated-reward model, we deﬁne the surrogate regret as (cid:94) Regret( T ) =
(cid:80) Tt =1 ( (cid:101) V ∗ ( s t ) − (cid:101)
[Equation]
πt ( s t )) , where (cid:101) V ∗ and (cid:101) V πt are computed using the truncated reward (cid:101) r ( s, a ) =
min { r ( s, a ) , Λ } .
By Eqn.
(7), with probability at least 1 − 1 / (2 NKT ) , under any policy, we have r ( s t , a t ) ≤ (cid:107) s t (cid:107) 1 ≤ Λ = (cid:101) O
(1 / ∆ 2 ) for all t ≤ T .
This means with high probability we have r ( s t , a t ) =
(cid:101) r ( s t , a t ) and hence the true regret and the surrogate regret is similar.
Speciﬁcally, we will show Regret(
T ) ≤ (cid:94) Regret( T ) + 1 / (1 − γ ) .
Bonus term.
Let P ( s (cid:48) | s, a ) be the true transition probability and (cid:98) P ( s (cid:48) | s, a ) be the empirical estimate.
As a typical result in MDP theory, we require b ( s, a ) ≥ γ (cid:107) V (cid:107) ∞ · (cid:107) P ( ·| s, a ) − (cid:98) P ( ·| s, a ) (cid:107) 1 to ensure optimism.
We exploit the fact that P ( s (cid:48) | s, a ) and (cid:98) P ( ·| s, a ) is factorized, i.e., P ( ·| s, a ) = ⊗ Ni =1 ⊗ Kk =
P i,k ( ·| s, a ) and (cid:98) P ( ·| s, a ) =
⊗ Ni =1 ⊗ Kk =1 (cid:98) P i,k ( ·| s, a ) , which stem from the independence assumption (Assumption 1).
This gives us (cid:107) P ( ·| s, a ) − (cid:98) P ( ·| s, a ) (cid:107) 1 ≤ (cid:80) Ni =1 (cid:80)
Kk =1 (cid:107) P i,k ( ·| s, a ) − (cid:98) P
i,k ( ·| s, a ) (cid:107) 1 .
Notice that P i,k ( ·| s, a ) is a Bernoulli distribution, then by Assumption 2 we have (cid:107) P i,k ( ·| s, a ) − (cid:98) P i,k ( ·| s, a ) (cid:107) 1 ≤ 2(1 ∧ |(cid:104)T ∗ − (cid:98) T , φ i,k ( s, a ) (cid:105)| ) .
Therefore, the bonus term can be chosen as Eqn.
and we ensure optimism at each time.
Regret decomposition.
We have the following regret decomposition for the surrogate regret.
[Equation]
where M = M ( T ) is the total number of switches and we will show that M = (cid:101) O ( d ) .
Then the dominant term of the regret is
[Equation]
where t s denotes the last switch up to time t
[Equation]
and the last inequality follows from a variant of Elliptical
Potential Lemma.
Plug in the choice of β T and we derive the result.
4. E XTENSION TO GENERALIZED LINEAR INDEPENDENT CASCADE MODELS
In this subsection, we brieﬂy show how to extend our algorithm and regret bound to generalized linear IC models.
The details can be found in Appendix F.
Assumption 5 (Generalized Linear Independent Cascade (GLIC) Model) .
Let s (cid:48) be the next state.
For each k ∈ [ K ] , we assume there is an underlying connectivity matrix A k ∈ R n × n such that
[Equation]
where µ : R → R satisﬁes µ (0) = 0 and 1 /κ ≤ µ (cid:48) ≤ 1 for some κ ≥ 1 .
And we assume s (cid:48)
i,k ’s are independent conditioned on s .
Given Assumption 5, there are two major differences extending M ORIMA to GLIC models: (1) new objective of tensor model estimation under GLM, which does not have analytical solutions; and (2) new optimistic planning, which is analyzed following new tensor estimate.
We have the following regret bound for M ORIMA under GLIC.
Theorem 3.
Let Assumption 5, Assumptions 2-4 hold.
With probability at least 1 − δ , Algorithmsatisﬁes the following regret upper bound:
[Equation]
where d = dim( T ∗ )
=
d 21 d 2 .
E XPERIMENTS
We experiment with M ORIMA (Algorithm 1) for inﬂuence maximization on both synthetic networks and a large-scale Twitter social network.
Synthetic networks.
We run benchmark experiments on two synthetic networks.
The ﬁrst cascade network (see Appendix E Figure 2a), consisting of N = 300 users and K = 4 content types, is constructed to bear a hierarchical structure with users with high, medium, and low inﬂuences, with 9 -dimensional user feature and 3 -dimensional content features (see Appendix E for details).
The second network is constructed to have a star-like structure, consisting of N = 70 users of various inﬂuence levels and K = 1 contents (see AppendixE Figure2b for details).
Experiment result of the ﬁrst network is reported in Figure 1(a), and result of the second star network is reported in Appendix
E Figure 3.
Twitter social network.
We further conduct experiments using the Twitter Social Network dataset (Leskovec & Krevl, 2014), which represents real-world social networks.
The dataset contains ∼ 80k nodes and ∼ 1 million directed edges, where a directed edge ( u, v ) means the node u follows the node v on Twitter.
We randomly sample multiple sub-graphs from the Twitter network and construct K
[Figure]
networks corresponding to different topics/content types.
For the sampled networks, we ﬁrst pick out n 1 nodes with most out-degrees and then include all their (cid:96) -hop neighbors, with n 1 = 8 , (cid:96)
= 5 .
We construct a connectivity tensor over these networks by randomly drawing edge weights from [0 , 0 . 1] and normalizing tensor row sums to 0 .
Next we apply non-negative Tucker decomposition (Kossaiﬁ et al., 2019) to extract the Tensor core model T ∗ , user features (of dimension d 1 = 10 ) and content features (of dimension d 2 = 2 ).
Thus, we have generated a large-scale topic-aware
Twitter diffusion network with K = 3 content types, N = 1966 nodes, and 38023 edges for dynamic inﬂuence maximization.
Implementation and Baselines.
Exactly solving for the optimal policy, even if the network is fully known, requires solving a combinatorially hard planning problem and is intractable.
In our experiment, we adopt the two-step lookahead approximate dynamic programming scheme (Powell, 2007) as the planning oracle for Algorithm 2 of M ORIMA .
In the implementation of Algorithm 1, we set γ = 0 .
9 , λ = 1 for synthetic networks and λ = 0 .
01 for the Twitter network.
For comparison with M ORIMA , we also test the following baselines: (i) the naive random policy that uniformly selects a user-content pair to activate; (2) the IMLinUCB (Wen et al., 2017) which is combinatorial linear bandits baseline that was originally designed for non-adaptive online IM.
To make a fair comparison with the two-step lookahead oracle, we run IMLinUCB every 2 timesteps and play the 2 selected actions spontaneously; (3) M ORIMA without slow switching, where we force the Q-function to be updated at each time step; (4) M ORIMA with known A k s - purely planning with a fully know model - it would be a performance upper bound of the reinforcement learning algorithm.
Results and analysis.
We report the averaged discounted cumulative rewards and its empirical conﬁdence region of our experiments in Figure 1, where each test is repeated on synthetic networks for 20 times and Twitter network for 5 times.
In Figure 1 (a), we observe that the discounted sum of reward of M ORIMA reaches the same level of the performance upper bound with true A in less than 100 rounds on synthetic network, showing that our algorithm can quickly explore the unknown network and learn to make optimal decisions.
In Figure 1 (b), we observe that M ORIMA can still match the performance of the planning oracle in large real social network with thousands of users.
Across all experiments, our reinforcement learning-based M ORIMA signiﬁcantly outperforms
IMLinUCB because it can adaptively make decisions based on current state while IMLinUCB does not take state into consideration.
Further, slow switching did not hurt the performance of M ORIMA while greatly reducing the computation complexity from O ( T ) parameter updates to O ( d log T ) .
C ONCLUSION
In this paper, we study the problem of content-aware online adaptive inﬂuence maximization and formulate the problem as an inﬁnite-horizon discount MDP.
We propose M ORIMA , a model-based reinforcement learning algorithm that learns optimal policy using only node-level feedback under the
IC model.
We provide a (cid:101) O ( d √ T/ ∆ + √ dNKT ) regret bound for our algorithm, which is the ﬁrst sublinear regret bound of online adaptive inﬂuence maximization problem.
We empirically
A L IMITATIONS AND B ROADER I MPACT
As a common limitation for all previous work on inﬂuence maximization, it is computationally infeasible to ﬁnd the exact optimal policy, especially when applied to real-world networks with millions of nodes. Therefore, the scalability of our proposed algorithm comes at the cost of using approximations. Given a certain computational budget constraint, the optimality of the policy needs to be traded-off towards affordable computational and storage complexity.
In our experiments, we use Monte-Carlo methods, parallel computation and randomized tree search (dynamic programming) methods for approximating the optimal policies. While the real-world social network graph in our experiment is in the same scale as the graphs used in previous online IM studies, scaling up to larger network is an important future work of ours. In the paper, we propose a model-based RL algorithm to learn the optimal policy for online adaptive inﬂuence maximization problems, which can be applied to advertisements for promoting beneﬁcial ideas, new knowledge, and innovative products across social networks. However, such algorithms might be exploited to propagate fake news or rumors through the social networks.
Addressing the ethical concerns also needs to be considered in future work.
B A S HARPER BOUND FOR THE C ONFIDENCE R EGION B.
[Equation]
Lemma B.1 (Conﬁdence Region) .
Let Assumptions 1-4 hold.
With probability at least 1 − δ , we have for all t ≥ 1 ,
[Equation]
where
[Equation]
and L = sup (cid:107) φ ti
[Equation]
k (cid:107) 2 , d
= d 21 d
[Equation]
Before the proof of Lemma B.1, we introduce two lemmas below:
Lemma B.2 (High probability bounds for the number of active user-content pairs) .
Let Assumptions 14 hold.
For any possibly non-stationary policy π , with probability at least 1 − δ , we have for all t ≥ 1 ,
[Equation]
Lemma B.3 (Bernstein-type self-normalized bound, batched version (Zhou et al., 2021a)) .
Let {F t } ∞ t
=
1 be a ﬁltration, { x it , y it } t ≥ 1 , 1 ≤ i ≤ m be a stochastic process such that x it ∈ R
d is F tmeasurable and y i t ∈ R is F t +1 -measurable.
Assume that conditioned on F t , { y 1 t , · · · , y m t } are independent, and
[Equation]
then with probability at least 1 − δ , the following holds simultaneously for all t ≥ 1 :
[Equation] where (cid:98) T
[Equation] =
[Equation]
−
[Equation] = [Equation] and β t = 8 σ (cid:112)
d log(1 + mtL 2 / ( dλ )) log(4 m 2 t 2 /δ )
+ 4 R log(4 m 2 t 2 /δ )
+ √ λ
(cid:107)T ∗ (cid:107) 2 .
proof of Lemma B.1.
We use Lemma B.3 for batched stochastic process { φ ti,k , ( s t +1 ) i,k } .
Notice that we can choose σ 2 to be the upper bound of var[( s t +1 )
i,k | s t , a t ] , and var[( s t +1 )
i,k | s t , a t ] ≤ E [( s t +1 ) i,k | s t , a t ] = (cid:88)
[Equation]
where we used the assumption that A ki,j ≤ CNK .
By Lemma B.2, we have with probability at least 1 − δ/ 2 , for all t ,
[Equation]
Therefore, when the above inequalities hold, we have
[Equation]
By Lemma B.3 with m =
NK , R = 1 , and σ 2 = CNK
(3 / ∆) 2 log 4 t 2 δ , we have
[Equation]
which is smaller than the result stated in the lemma.
B. D EFERRED PROOFS IN S UBSECTION B.
proof of Lemma B.2.
First, we bound the expectation of (cid:107) s t (cid:107) 1 .
By the transition, we have
[Equation]
Therefore,
[Equation]
Recall that we have assumed that for any content k and any user j , (cid:88) [Equation] j
[Equation]
1 − ∆ .
Then we have
E [
[Equation]
1 |
[Equation]
t , a t ] ≤ (
[Equation]
∆) (cid:88)
[Equation]
where the last inequality holds since the action alters at most one entry of the state.
Next, notice that conditioned on ( s t , a t ) , (cid:107) s t +1 (cid:107) is the summation of NK independent Bernoulli random variables.
By Bernstein inequality, we have with probability at least 1 − δ t ,
[Equation]
Since the variance of a Bernoulli random variable is bounded by its expectation
[Equation]
we have [Equation] Therefore
[Equation]
by Equation (9) [Equation] we have
[Equation]
where we used at + b/t ≥ 2 √ ab for the last inequality.
Finally, we set δ t = δ 2 t 2 so that (cid:80) t δ t ≤ δ and take union bound over all t ≥ 1 .
By solving the recursion, we have with probability at least 1 − δ ,
[Equation]
for all t ≥ 1 .
proof of Lemma B.3.
We consider a “serialized” stochastic process.
Let G t,i =
σ ( F t , y 1 t , . . .
, y i − 1 t ) .
When 1 ≤ i ≤ m , we have G t,i ⊆
G t,i +1 ; while when i = m
+ 1 , we have G t,m +
= σ ( F t , y 1 t , . . .
, y mt ) ⊆
G t +1 , 1 = F t +1 .
Then we know that
[Equation]
is a ﬁltration.
Clearly we have x it is G t,i -measurable and y
it is G t,i +1 -measurable.
By the conditional independence assumption, we also have
[Equation]
Therefore, by Theorem 4.1 of Zhou et al.
(2021a), we have with probability at least 1 − δ , for all t ≥ 1 and i = 1 , . . . , m , [Equation] and
[Equation] where [Equation] T t,i = Σ − 1 t,i B t,i ,
[Equation] t,i = [Equation] + t − 1 (cid:88) τ
= m (cid:88)
j = x jτ ( x jτ ) (cid:62) + i (cid:88)
j = x jt ( x jt ) (cid:62) , B t,i = t − 1 (cid:88) τ
= m (cid:88)
j = x jτ y jτ + i (cid:88)
j = x jt y jt , and β t
[Equation]
Then the result of Lemma B.3 follows by setting i = m .
C P ROOF OF T HEOREM
Additional Notation.
Let t 1 = 1 , and for s ≥ 1 , the next switching time t s +1 is recursively deﬁned as
[Equation]
Denote the set of switching times by W = { t 1 , t 2 , . .
. , t M } where M is the total number of switches.
We have 1 = t 1 < t 2 < · · · < t M ≤ T < t M +1 .
We slightly abuse the notation and use t s to denote the last switch up to time t , i.e., t s ≤ t < t s +1 .
Then by slow switching we mean Q t =
Q ∗ (cid:98) T ts , (cid:101) r + b ts .
Recall the deﬁnition of the regrets
Regret( T ) =
T (cid:88) t
= ( V ∗ ( s t )
− V πt ( s t )) , (cid:94) Regret( T ) =
T (cid:88) t
= ( [Equation]
where V, Regret are deﬁned with the original untruncated model and (cid:101) V , (cid:94) Regret are deﬁned with the truncated-reward model.
Key Lemmas.
Before the proof of Theorem 2, we introduce several key lemmas.
Lemma C.1 (optimism) .
Let Assumptions 1-4 hold.
Set the bonus term to be
[Equation]
Then with probability at least 1 − δ , we have the optimistic condition (cid:101) Q ∗ ( s, a ) ≤ Q t ( s, a ) holds for all t ≥ 1 .
Furthermore, we have for any V ( s ) such that 0 ≤ V ( s ) ≤ Λ / (1 − γ ) ,
[Equation]
Lemma C.2 (surrogate regret) .
Let Assumptions 1-4 hold.
Assume that T log(1 /γ ) ≥ log(2 NKT ) .
For any policy π , we have the following connection of the regrets of the two MDPs.
[Equation]
Lemma C.3 (regret decomposition (Zhou et al., 2021b)) .
Let Assumptions 1-4 hold.
Assume at each time step t , the results of Lemma C.1 holds.
Then with probability at least 1 − δ , we have the following regret decomposition
[Equation]
where E T is the switching error
[Equation]
Lemma C.4 (bounding the number of switches) .
Let Assumptions 1-4 hold.
The total number of the switches M incurred by Algorithm 1 is bounded as
[Equation]
where L = sup (cid:107) φ ti,k (cid:107) 2 .
Next we state the proof of Theorem 2.
proof of Theorem 2.
Combing Lemma C.1, Lemma C.2, and Lemma C.3, we have with probability at least 1 − 2 δ , when T log(1 /γ ) ≥ log(2 NKT ) ,
Regret( T ) ≤
[Equation]
γ
[Equation]
(cid:88) t
[Equation]
[Equation]
Next we provide an upper bound for (cid:80)
Tt =1 b t s ( s t , a t ) .
By Lemma C.1 we know that
[Equation]
For any ( i, k ) , deﬁne Σ t,i,k = Σ t − 1 + (cid:80) i − 1 j =1 (cid:80) Kl
=1 φ tj,l ( φ tj,l ) (cid:62) + (cid:80) kl
=1 φ ti,l ( φ ti,l ) (cid:62) .
By the deﬁnition that t s +1 = min { t | det(Σ t − 1 ) > 2 det(Σ t s − 1 ) } , we have det(Σ t s +1 − 2 ) ≤ 2 det(Σ t s − 1 ) .
Therefore, when t s ≤ t < t s +1 − 1 , we have [Equation]
By Lemma D.4, this implies
[Equation]
Then we have
[Equation]
where the last inequality follows from Lemma D.3.
This implies
[Equation]
Next we bound the switching error E T .
Since there are in total M switches, we know that there are at most M non-zero terms in the summation of E T .
Then we have
[Equation]
Plugging the result of Lemma C.4, we have E T =
(cid:101) O ( Λ1 − γ d ) .
Therefore, we have the ﬁnal regret upper bound when T log(1 /γ ) ≥ log(2 NKT ) :
Regret( T ) ≤ (cid:101) O (cid:16)
[Equation]
, where d = dim( T ∗ ) =
d 21 d 2 .
When T log(1 /γ ) ≤ log(2 NKT ) , the above inequality trivially holds.
C. D EFERRED PROOFS
proof of Lemma C.1.
For simplicity, deﬁne (cid:98) P = P (cid:98) T t , which is the estimated transition distribution obtained using (cid:98) T t .
Notice that P i,k ( ·| s, a ) is a Bernoulli distribution with success probability (cid:104)T ∗ , φ i,k ( s, a ) (cid:105) , while P i,k ( ·| s, a ) is a Bernoulli distribution with success probability (cid:104) (cid:98) T t , φ i,k ( s, a ) (cid:105) .
Therefore, we have
[Equation]
By Lemma B.1, with probability at least 1 − δ , we have (cid:107)T ∗ − (cid:98) T t (cid:107) Σ t − 1 ≤ β t for all t ≥ 1 .
Then by
Cauchy Inequality, the above term can be further bounded as
[Equation]
By Lemma D.2, we have
[Equation]
Then by Lemma D.1, we know the desired result holds.
proof of Lemma C.2.
Deﬁne π ∗ as the optimal policy under the original model.
Then we have
[Equation]
where the inequality holds because (cid:101) V ∗ ( s t ) is the optimal V-function with respect to the truncatedreward model.
Notice that by Lemma B.2 with δ 0 = 1 / (2 NKT ) , we know that for policy π ∗ , with probability at least 1 − δ 0 , we have (cid:107) s i (cid:107) 1 ≤ Λ = 6∆ 2 log(4 NKT 3 ) for all 1 ≤ i ≤ T .
Therefore, with probability at least 1 − δ 0 , r ( s i , a i ) =
(cid:101) r
( s i , a i ) for all 1 ≤ i ≤ T .
Then
[Equation]
where the last inequality holds when T log(1 /γ ) ≥ log(2 NKT ) .
On the other hand, we have
[Equation]
Therefore,
Regret( T ) ≤ (cid:94) Regret( T ) + T (cid:88) t = [( V ∗ ( s t ) − (cid:101) V ∗ ( s t )) − ( V πt ( s t ) − (cid:101) V πt ( s t ))] ≤ (cid:94) Regret( T ) + 1 − γ .
proof of Lemma C.3.
Deﬁne V t ( s ) = max a Q t ( s, a ) .
By the assumption that (cid:101) Q ∗ ( s, a ) ≤ Q t ( s, a ) , we have (cid:101) V ∗ ( s ) ≤ V t ( s ) .
Then
[Equation]
where the last equality holds since a t ∈ argmax a Q t ( s t , a ) , i [Equation] e., we take the action a t greedily according to Q t .
The optimal truncated Q-function Q ∗ (cid:98) T ts , [Equation] r + b ts ( s, a ) (Algorithm 2) satisﬁes the following truncated
Bellman equation:
[Equation]
Then we have
[Equation]
For the second term, since | V t ( s (cid:48) ) | ≤ Λ / (1 − γ ) , by Lemma C.1, we have
[Equation]
For the last term, we have
[Equation]
where
[Equation]
Therefore, we have
[Equation]
For the last term above
[Equation]
we have
[Equation]
Notice that { ξ t } is a martingale difference sequence.
Therefore, by Azuma-Hoeffding inequality, we have with probability at least 1 − δ ,
[Equation]
To summarize, we have
[Equation]
which implies
[Equation]
proof of Lemma C.4.
On the one hand, we have det
[Equation]
On the other hand [Equation] we also have
[Equation]
λ d [Equation] Therefore, M < 1 log2 d log (cid:16) d +
[Equation]
[Equation] d (cid:17) + 1 . D
[Equation]
L EMMAS
Lemma D.1.
Let P ( s (cid:48) | s, a ) and (cid:98) P ( s (cid:48) | s, a ) be two transition probabilities.
Assume that 0 ≤ r ( s, a ) ≤ Λ .
Let Q ∗ be optimal Q-function for MDP M P ,r .
Let (cid:98) Q ∗ be the optimal truncated Q-function for
M (cid:98) P ,r + b , which satisﬁes the following equation
[Equation]
Then if
[Equation]
we have
[Equation]
all
[Equation]
Furthermore, we have for any V ( s ) such that 0 ≤ V ( s ) ≤ Λ / (1 − γ ) ,
[Equation]
Lemma D.2 (Factorization) .
If P ( ·| s, a ) = ⊗ Ni =1 P i ( ·| s, a ) , (cid:98) P ( ·| s, a ) = ⊗ Ni =1 (cid:98) P i ( ·| s, a ) , then (cid:107) P ( ·| s, a ) − (cid:98) P ( ·| s, a ) (cid:107) 1 ≤
N (cid:88) i = (cid:107) P i ( ·| s, a ) − (cid:98) P i ( ·| s, a ) (cid:107) 1 .
Lemma D.3 (Lemma 11 in (Abbasi-Yadkori et al., 2011)) .
For any { x t } T t =1 ⊆ R d , let Σ t
= λ I + (cid:80)
Tt =
1 x t x (cid:62) t , then we have
T (cid:88) t = (1 ∧ (cid:107) x t (cid:107) 2Σ − 1 t − 1 ) ≤ 2 d log dλ + TL 2 dλ , where L = sup (cid:107) x t (cid:107) 2 .
Lemma D.4 (Lemma 12 in (Abbasi-Yadkori et al., 2011)) .
Let A,
B ∈ R d
× d be two positive deﬁnite matrices and A (cid:23) B .
Then for any x ∈ R d , we have
[Table]
E A DDITIONAL E XPERIMENTS E. S YNTHETIC C ASCADE N ETWORK
We conduct experiments on cascade synthetic networks as shown in Figure 2a.
There are K =identical networks are associated with different contents; each cascade network has N = 300 nodes with high/medium/low inﬂuential power, 5 / 20 / 275 nodes respectively.
More speciﬁcally, the ﬁve high nodes can activate theirselves and each others at the next time step with probability 0 .
1 ; the high nodes can activate the medium nodes with probability 0 .
2 ; the medium nodes can only activate the low nodes with probability 0 .
Therefore, these networks would appreciate the high nodes as better actions with more delayed reward.
In addition, the d 2 = 3 -dim content features for K =contents are { (1 , 0 , 0) , (0 , 1 , 0) , (0 , 0 , 1) , (0 . 3 , 0 . 3 , 0 . 4) } ; the d 1 = 9 -dim user features are one-hot for each type of nodes associated with each dimension of content features.
Finally, the underlying linear dynamic T ∗ can be easily computed for desired probabilities of edges.
The experiment result of the cascade synthetic networks is reported in Figure1a.
[Figure]
[Figure]
MORIMA MORIMA with known A
IMLinUCB random
[Figure]
E. S YNTHETIC S TAR - SHAPE N ETWORK
We conduct additional experiments on another synthetic network as shown in Figure 2b, where a better action has more delayed reward and making decision at each time step is appreciated.
This synthetic network has three inﬂuential nodes out of total N = 70 nodes, while the center one is the best choice and has the ability to activate the other two inﬂuential nodes.
Also the central inﬂuential node has delayed but higher expected reward than expected reward by activating either of other two neighboring inﬂuential nodes.
For simplicity, we set only one content is available here, K = 1 and d 2 = 1 .
The d 1 = 6 user features are one-hot vectors, indicating their neighborhood subgraph.
The discount factor of reward is γ = 0 .
In Figure 3, we compare the performance of our MORIMA to MORIMA with known A k as upper bound and IMLinUCB as baseline.
The details of these algorithms are the same as in section 5.
MORIMA exhibits its great power to explore the unknown graph efﬁciently; its learning curve at the ﬁrst 40 time steps overlaps with MORIMA while knowing the true dynamics A k .
In addition, the sum of cumulative rewards of MORIMA and the upper bound stay at the same high level.
Furthermore, we observe that the learning procedure of IMLinUCB takes much longer and converges to a much lower level.
The classic IM setting, activating k seeds at once for every k step, shows its limit while adaptive decision making leads to a better result.
We ran all experiments on our internal cluster with 8 CPUs, 128G memory per task.
F E XTENSION TO G ENERALIZED L INEAR M ODEL
In this section, we show how to extend our algorithm and regret bound to generalized linear models.
We ﬁrst re-state below the modiﬁed assumption on the transition model.
Assumption 6 (Generalized Bernoulli Independent Cascade Model) .
Let s (cid:48) be the next state.
For each k ∈ [ K ] , we assume there is an underlying connectivity matrix A k ∈ R n × n such that
[Equation]
where µ : R → R satisﬁes µ (0) = 0 and 1 /κ ≤ µ (cid:48) ≤ 1 for some κ ≥ 1 .
And we assume s (cid:48)
i,k ’s are independent conditioned on s .
F. MORIMA FOR G ENERALIZED L INEAR M ODEL
Next, we state the changes to our algorithm.
Under this assumption, we cannot simply use ridge regression to get an empirical estimation of T ∗ .
Instead, we estimate the tensor model by
[Equation]
where φ ti,k and Σ t are deﬁned as in Section 3.
We still perform optimistic planning with respect to the truncated-reward model, where the reward bonus term b t ( s, a ) is replaced with
[Equation]
[Equation]
which is the original bonus term multiplied by 2 κ .
We adopt the same slow switching method as before.
F. R EGRET A NALYSIS
We have the following regret bound, which is the original bound multiplied by κ .
Theorem 4 (Re-state Theorem 3) .
Let Assumption 5, Assumptions 2-4 hold.
With probability at least 1 − δ , Algorithm 1 satisﬁes the following regret upper bound:
[Equation]
where d = dim( T ∗ )
=
d 21 d 2 .
F. P ROOF S KETCH OF T HEOREM
The proof of Theorem 3 only differs from the proof of Theorem 2 slightly.
Next we examine through the proof of Theorem 2 and state the corresponding lemmas in the generalized linear model setting.
First, we have exactly the same result for the high probability bounds for the number of active user-content pairs.
Lemma F.1 (High probability bounds for the number of active user-content pairs) .
Let Assumption 5,
Assumptions 2-4 hold.
For any possibly non-stationary policy π , with probability at least 1 − δ , we have for all t ≥ 1 ,
[Equation]
The next two lemmas justify the choice of the bonus term.
Lemma F.2 (Conﬁdence Region) .
Let Assumption 5, Assumptions 2-4 hold.
With probability at least 1 − δ , we have for all t ≥ 1 ,
[Equation]
where β t is deﬁned as in Eqn.
Lemma F.3 (Optimism) .
Let Assumption 5, Assumptions 2-4 hold.
Set the bonus term to be
[Equation]
Then with probability at least 1 − δ , we have the optimistic condition (cid:101) Q ∗ ( s, a ) ≤ Q t ( s, a ) holds for all t ≥ 1 .
Furthermore, we have for any V ( s ) such that 0 ≤ V ( s ) ≤ Λ / (1 − γ ) ,
[Equation]
proof of Theorem 3.
We can verify that Lemma C.2, Lemma C.3, and Lemma C.4 also hold for the generalized linear model.
Therefore, the exact same proof of Theorem 2 applies with Lemma C. replaced with Lemma F.3.
The result only differs by a factor of κ .
F. D EFERRED PROOFS OF L EMMAS
proof of Lemma F.1.
Notice that we have assumed µ (0) = and µ (cid:48) ≤ 1 .
Therefore for z > 0 , µ ( z ) =
µ ( z ) − µ (0) ≤ z −
= z .
Then
[Equation]
Then the result holds by applying the same proof of Lemma B.2.
proof of Lemma F.2.
We have
[Equation]
where the last but one inequality holds by applying Lemma B.3 with the same variance upper bound as in the proof of Lemma B.1.
proof of Lemma F.3.
Let (cid:98) P = P (cid:98) T t .
By the assumption that µ (cid:48) ≤ 1 , we still have (cid:107) P i,k ( ·| s, a ) − (cid:98) P i,k ( ·| s, a ) (cid:107) 1 ≤ 2(1 ∧ |(cid:104)T ∗ − (cid:98) T t , φ i,k ( s, a ) (cid:105)| ) ≤ 2(1 ∧ (cid:107)T ∗ − (cid:98) T t (cid:107) Σ t − 1 · (cid:107) φ i,k ( s, a ) (cid:107) Σ − 1 t − 1 ) .
By Lemma F.2, with probability at least 1 − δ , we have L t ( T ∗ ) ≤ β t and then L t ( (cid:98) T t ) ≤ β t for all t ≥ 1 .
Therefore, we have
[Equation]
[Table]
≥ 1 /κ · (cid:107) Σ t − 1 · ( (cid:98) T t − T ∗ ) (cid:107) Σ − 1 t −= 1 /κ · (cid:107) (cid:98) T t − T ∗ (cid:107) Σ t − 1 , where we use Lagrange Mean Value Theorem in the ﬁrst equality and µ (cid:48) ( · ) ≥ 1 /κ .
Then the desired result holds by applying Lemma D.2 and Lemma D.1 in the same way as the proof of Lemma C.1.