SlateFree: a Model-Free Decomposition for Reinforcement Learning with Slate Actions
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We consider the problem of sequential recommendations, where at each step anagent proposes some slate of N distinct items to a user from a much larger catalogof size K >> N .
The user has unknown preferences towards the recommendationsand the agent takes sequential actions that optimise (in our case minimise) someuser-related cost, with the help of Reinforcement Learning.
The possible itemcombinations for a slate is (cid:0) KN (cid:1) , an enormous number rendering value iterationmethods intractable.
We prove that the slate-MDP can actually be decomposedusing just K item-related Q functions per state, which describe the problem in amore compact and efﬁcient way.
Based on this, we propose a novel model-free
SARSA and Q-learning algorithm that performs N parallel iterations per step,without any prior user knowledge.
We call this method SlateFree , i.e. free-ofslates, and we show numerically that it converges very fast to the exact optimumfor arbitrary user proﬁles, and that it outperforms alternatives from the literature.
Introduction
In many real-life applications, an agent needs to optimally adapt to a random environment through thechoice of multi-dimensional actions over time.
A speciﬁc, common scenario is that of a personalisedrecommender system (RS) which should pick a set of N ≥ 1 items among a much larger corpusof size K >> N , given the user’s recent and past viewing history.
An illustrative example of thischallenge to solve the top- N recommendations problem is Google’s YouTube algorithm, wherethe current corpus contains several tens of billions of videos Goodrow [2021] and the number ofrecommended items per view may vary (based on scrolling) but in general involves N > 20 items.
Insuch applications the group of N items recommended per step is often called a slate .
As Shani et al.
[2005] observed, the RS problem is essentially of sequential nature.
The recommendation of a speciﬁc slate at some point in time offers not only immediate gains if some recommendeditem is clicked, but can also generate future beneﬁts by guiding the user towards a path of moreinteresting items as the user session evolves.
Shani et al.
[2005] formulated this problem within theframework of Markov Decision Processes (MDPs) (Puterman [1994]), and tried to solve it understrong assumptions of independence.
Naturally, when the RS needs to learn unknown user preferences, it can do so by observing user-item interactions over time, and the tools of Reinforcement
Learning (RL) are most appropriate, see Taghipour et al.
[2007] for an early effort.
The optimal slate selection problem per step is in fact combinatorial: the user’s choice is affected bythe combination (and possibly the order) of the items in the slate, not just their individual importanceto the user Aouali et al.
In such control problems over long horizon, the challenge withslate actions is that the corresponding combinatorial action space is immense and the search for anoptimal solution quickly becomes intractable.
Even for a small catalog of size K = 100 items and a recommendation slate of size N = 4 , there are (cid:0) 1004 (cid:1)
≈ 4 million unordered slates as possible actions.
Consequently, the number of slate actions in the YouTube example is astronomical.
To tackle the dimensionality explosion in such value iteration algorithms, Ie et al.
[2019] (motivatedby previous work from Sunehag et al.
[2015]) have shown that the slate-value function can be exactlydecomposed into K individual Q item-values, per state.
Such decomposition could actually rendertemporal-difference (TD) learning with slates tractable.
However, this proposal is based on priorknowledge about the user behaviour given any slate and single item choice.
In essence this is nota model-free algorithm.
It can be implemented if either a user model is assumed, or if the userpreference choice per slate is learned from history, which doubles the learning effort of RL and needsto keep in memory N (cid:0) KN (cid:1) unknowns per user, one unknown per slate and per item choice.
Our contribution.
We introduce in this work a novel exact decomposition of the Q values for slates,into K individual item-Q values and propose a tractable TD-learning (SARSA- and Q-) algorithm,named here SlateFree , which allows to solve efﬁciently learning and control problems of arbitraryslate dimension N > 1 using value-iteration.
The important difference compared to Ie et al.
[2019] isthat our decomposition is entirely model-free in the sense that it does not require any prior knowledgeover the user behaviour, and it allows to include costs that depend on both state and action.
Theproposed decomposition without assuming any independence, simpliﬁes Q-learning considerably:
(i)
It keeps K state-item Q functions per state in memory, instead of (cid:0) KN (cid:1) , a massive reduction.
(ii)
The optimal slate in the exploitation consists of the N -out-of- K items with best item-Q function.
The novel RL algorithm is based on deﬁnitions of state-item Q functions, item-costs and transitions.
It performs N-parallel Q-updates per step, one per item included in the slate.
This way, the relevanceof an individual item is updated every time this is included in a slate.
The method reminds of theindependent learners in multi-agent systems, by Claus and Boutilier [1998], one agent per dimension.
The SlateFree has considerable performance features.
It can learn and take optimal actions overtime for any unknown user behaviour.
It is shown to be insensitive to the slate-size N , thus allowingto scale for arbitrary dimensions.
The MDP decomposition is proved in this paper to hold undercertain assumptions: (i) the slates are unordered sets of distinct items, and (ii) the user behaviour is
Markovian, i.e. the user choice is based only on the current state and recommended slate.
Numericalevaluations of the novel RL algorithm show empirically that it ﬁnds the optimum even when the costis a function of both the state and the chosen action-slate, something not possible in other works.
In Section 2 we introduce state-item values as marginal quantities and prove the decomposition of the
Bellman equations in the MDP setting.
In Section 3 we present the decomposed SARSA- and RLalgorithms, referred to from now on, jointly, as SlateFree .
In Section 4 we show numerically theexactness of the solution compared to vanilla-RL for users with various preference behaviour andillustrate signiﬁcant performance improvements against SlateQ in Ie et al.
We also illustratehow SlateFree converges for any type of user and the convergence speed does not depend on thesize of the slate.
We further illustrate how the algorithm behaves in situations where the cost is afunction of both state and action-slate.
The code is available on Google Colab SlateFree Authors[2022a] and here SlateFree Authors [2022b].
Related literature.
The established solution for static recommender systems is based on collaborativeﬁltering, as in Deshpande and Karypis [2004] or matrix factorisation, as in Takács et al.
Sincethe problem is actually dynamic, Reinforcement Learning (Sutton and Barto [2018]) is at the momentwidely applied to propose more effective or more diverse recommender systems (Karatzoglou et al.[2013], Rohde et al.
[2018], Zhou et al.
[2020], Warlop et al.
To overcome the curse ofdimensionality in the action space a deep reinforcement learning approach is taken: Zheng et al.[2018] work with a value-based approach and approximate the Q-value by a neural network, whereas
Liu et al.
[2020b], Liu et al.
[2020a] use an actor-critic architecture for policy-based opimisation,where the actor network outputs a continuous feature vector, which can be mapped to an item, thusavoiding the discrete formulation.
RL problems with continuous and high-dimensional action spaces have been recently approachedby policy iteration methods.
Deterministic policy gradient by (Silver et al.
[2014]) is shown toconsiderably outperform standard policy updates.
This method was combined with an efﬁcientmapping to discrete actions by Dulac-Arnold et al.
[2015], so that problems like the search for top- Nrecommendations can be efﬁciently resolved.
Chen et al.
[2019] adapt the REINFORCE algorithmwith reward independence assumptions.
de Wiele et al.
[2020] work with amortised inference to maximise over a smaller subset of possible actions.
Metz et al.
[2017] propose an autoregressivenetwork architecture to sequentially predict the action value for each action dimension, which requiresmanual ordering of the actions.
Tavakoli et al.
[2018] propose a neural architecture with many networkbranches, one for each action dimension.
The special structure of slate-recommendations has givenrise to problem speciﬁc solutions, like the one by Sunehag et al.
[2015], who introduce a formulationthat beneﬁts from the fact that at each step the user chooses a single item, for a given action slate.
Their approach cannot scale because it needs to keep in memory one value-function per slate.
In avery interesting recent approach Ie et al.
[2019] show how the slate-value function can be exactlydecomposed into individual Q item-values and introduce the method SlateQ. They construct optimalslates from the individual item-values by solving a Linear Program (LP) per step.
As mentioned, theirdecomposition is based on prior knowledge of user choice behaviour.
Decomposition of slate-MDPs
We ﬁrst introduce the slate-MDP, deﬁned as ( S , A , P , C , λ ) and describe the process for the specialapplication of the recommender system.
Time is slotted with current step t .
The state S t = s at timet will be here the currently viewed item, so the state-space S
= K is the full item catalog of size K .
But we can use more general states, e.g. the history of the last m -viewed items m > 1 , so S (cid:54) =
K .
The action A t = ω is an N -sized unordered slate of recommended items.
The set of possible actions
A is the set of all possible unordered N -sized slates, where in each slate ω ∈
A no item is duplicated.
Here, “unordered” means that only the set of recommended items in the slate is important, not theirorder.
The state transition function P : S ×A×S → [0 , 1] is the probability, given the current state sand recommended slate ω , that the user moves to state s (cid:48) , by either picking one of the recommendeditems, or rejecting them and selecting some item from the search bar.
The general cost functionis C : S ×
A → R and λ ∈ (0 , 1) is the discount rate.
The objective is to ﬁnd an optimal policyπ : S ×
A → [0 , 1] to minimise the expected cumulative discounted cost from any initial state s ∈ S ,which is the value-function of state s (alternatively one could work with rewards and maximisation)
[Equation]
In the above, E π is the expectation under given policy π , the current time-step is t and the cost atfuture step t + k is c t
+ k =
c
( S t + k , A t + k ) .
The randomness is due to the user choice behaviour.
Weconsider a stationary policy π , which is a distribution over actions given the current state.
It does notdepend on time t .
This is a randomised policy in general,
[Equation]
If the mass is concentrated on a single slate-action ω , the policy is called deterministic and we denoteit by π ds (or just d ).
Given a state s , it holds (cid:80) ω ∈A ( s ) π s ( ω )
= 1 .
Observe that we introduced anaction space
A ( s ) per state s , because for our recommender application the currently viewed item sshould not be included in the recommendation slate.
The state-action function Q π ( s, ω ) of pair ( s, ω ) ∈ S
× A is the expected cumulative discountedcost, starting from state s , taking action ω and following policy π ,
[Equation]
From Sutton and Barto [2018] and Puterman [1994] we know that the state-value functions satisfythe recursive system of Bellman equations (just policy π evaluation here), ∀
[Equation]
s, ω
[Equation] [Equation]
where in the last equation we replaced with the value function in s (cid:48) because for stationary randomisedpolicies it holds V π ( s
[Equation]
The P [ s (cid:48) | s, ω ] in (4) models the random user choice behaviour when visiting state s and exposed toslate ω , which is considered known in the MDP setting.
Notice here, that the process is Markovianexactly because the user is Markovian, meaning that their choice is only based on the current stateand action and not the past.
2.1 Item frequencies, transition probabilities, and state-item functions
For the decomposition we need to introduce some new marginal quantities to shift the analysis fromslates to items.
Since the policy π is stationary and randomised, given state s it randomly recommendsamong feasible slates, each containing a different set of items.
Obviously, the item j ∈ K can appearin several action-slates.
Deﬁnition 1.
The frequency of a recommended item j ∈ K at state s ∈ S , under policy π , is deﬁnedthrough the randomised probabilities of slate-actions in (2) as
[Equation]
where A ( s ; { j } ) ⊆
A ( s ) is the set of actions at state s , that necessarily include item j .
The indicatorfunction 1 ( j ∈ ω ) =
if j is included in the slate, otherwise 0 .
What we call “frequency” is in factthe probability to randomly select some slate-action that includes item j , when at state s .
It holds,
[Equation]
where we use the fact that each slate contains N distinct items, i.e. there are no duplicates.
Deﬁnition 2.
The transition probability given item j inside the recommended slate is deﬁned as
[Equation]
For the transition probability given state s and some action including item j we prove three Properties:
Property 1.
The single-item transition probability depends on the policy π .
It satisﬁes,
[Equation]
Proof.
We can write the slate as ω =
( ω − j , j ) which contains item j and ω − j are the remaining
N − 1 entries.
It holds due to conditioning, that
[Equation]
where the superscript π is included, because P π [ ω | s, j ] depends on the policy π .
Using this expression,
[Equation]
Summing at the left-hand side over all ω that contain j , we get the marginal P
π
[ s (cid:48) | s, j ] .
Property 2.
The single-item transition probability is a marginal probability of P [ s (cid:48) | s, ω ] , and it holds(cid:88)
[Equation] [Equation]
Proof.
It holds that π s ( ω ) :
=
P π [ ω | s ] .
We can use the conditional probability formula(cid:88)
[Equation]
1 ( j ∈ ω ) P [ s (cid:48) | s, ω ] = (cid:88) ω ∈A ( s )
P [ s (cid:48) , ω | s ] 1 ( j ∈ ω ) =
P [ s (cid:48) , ω ∈ A (
s ; { j } ) | s ] =
P [ s (cid:48) | s, ω ∈ A (
s ; { j } )] · P [ ω ∈ A ( s ; { j } ) | s ]
Def. 2 , Def. 1 =
P π
[ s (cid:48) | s, j ] r πs,j .
Property 3.
If r πs,j > 0 , then P [ s (cid:48) | s, j ] is a probability mass function, (cid:80) s (cid:48)
∈S P π
[ s (cid:48) | s, j ] = 1 .
Proof.
Using Deﬁnition 2 and Deﬁnition 1 we can write(cid:88)
[Equation]
[Equation] [Equation]
Def
[Equation]
Deﬁnition 3.
The marginal cost-item function c π ( s, j ) that depends on policy π is deﬁned as
[Equation]
In the special case that the cost is just a function of the current state, c π
( s, j ) =
c ( s ) , ∀ j ∈ K .
Finally, we give the following special deﬁnition for the state-item function Q π ( s, j ) :
Deﬁnition 4.
The state-item function Q π ( s, j ) is deﬁned from the state-action functions Q π ( s, ω ) as
[Equation]
Notice that this deﬁnition is different from what would be the most natural one, i.e. to be thevalue-function starting from state s , and taking some initial slate-action that necessarily includes itemj and following policy π .
The Deﬁnition 4 is in fact a marginal quantity, i.e., the expectation over allstate-value functions that include item j normalised by the frequency r πs,j > 0 .
Notice here that for items j such that r π s,j = 0 , the state-item functions Q π ( s, j ) are not well deﬁned.
We will see next that this is not actually a problem for the decomposition.
2.2 Decomposed Bellman equations
Theorem 1.
[SlateFree-MDP]
The Bellman equations in (4) for state-action functions with slates
Q π ( s, ω ) , are equivalent to the following system of equations with state-item functions Q π ( s, j ) from
Def. 4, cost-item functions from Def. 3, and transition probability given some item j from Def.
Q π ( s, j ) =
[Equation]
If the cost is a function of just the current state
[Equation]
we replace in the above by c π ( s, j )
=
c ( s ) , so thatthe cost does not depend on the policy.
Proof.
We multiply both sides of (4) by π s ( ω ) 1 ( j ∈ ω ) and sum over all feasible slate-actions ω ,
[Equation]
Then, we replace the left-hand side by the function Deﬁnition 4, the ﬁrst term of the right-hand sideby the cost-item Deﬁnition 3 and in the second term we use Property 2, to ﬁnd
[Equation]
Now, use the equality V π ( s (cid:48)
[Equation]
= (cid:80)
[Equation]
) , multiply it from both sides by 1 ( k ∈ ω )and sum over k ∈ K .
We use the fact that the slate size is N and again Deﬁnition 4, so we get(cid:88)
[Equation]
By replacing the above expression for V π ( s (cid:48) ) in (14) we get the expression in (13), as long as r π s,j > 0for the ( s, j ) pair.
In the case that r π ˜ s,(cid:96) = 0 for some pair (˜ s, (cid:96) ) , notice that
regardless of its value
Q π (˜ s, (cid:96) ) < ∞ , it will always contribute r π ˜ s,(cid:96) Q π
(˜ s, (cid:96) ) = 0 when found at the right-hand side of (13),hence the pairs with zero frequencies do not affect the equations of others.
For their own state-itemvalue, any solution Q π (˜ s, (cid:96) ) − c
[Equation]
˜ s ) − λ
[Equation]
s
[Equation]
∈S P π [˜ s (cid:48) | ˜ s, (cid:96) ]
V π (˜ s (cid:48) )
= κ < ∞ satisﬁes (14), hence alsothe one for κ = 0 .
This way we result in the validity of (13) for any possible state-item pair.
2.3 Optimality equations
We know from Puterman [1994, Prop.6.2.1] that the discounted MDPs always have a stationarydeterministic optimal policy.
We denote from now on deterministic policies by index d (we may omitπ ) and the optimal policy by d ∗ (we may omit d ).
Then by deﬁnition,
[Equation]
for a unique
[Equation]
A special case is when we follow the optimal deterministic policy, so that
[Equation]
When more than one action-slates have the minimum Q ( s, ω ) ties are broken arbitrarily.
For thedeterministic and optimal policies the value function starting from state s is equal to
[Equation]
Theorem 2.
[Optimal SlateFree-MDP]
The Bellman optimality equations for slates are equivalentto the following system of equations with state-item functions from Def. 4, cost-item functions from
Def. 3, and transition probability given some item j from Def. 2, under the optimal policy π = d ∗
[Equation]
and it holds Q d ∗ ( s, j ) =
V d ∗ ( s ) , ∀ j ∈ ω ∗ ( s ) inside the optimal slate.
For state-only dependent cost,we replace above by c d ∗ ( s, ω ∗ ( s )) =
c ( s ) .
Proof.
For deterministic (and optimal) policies the quantities in Section 2.1 related to items become:
Item-frequencies (from Deﬁnition 1)
[Equation]
Transition probability (from Deﬁnition 2):
[Equation]
meaning that the transition probability given some item in the slate, is equal to the transitionprobability given the whole information about the slate.
This probability is the same, conditioned onany item inside the slate.
Hence P d [ s (cid:48) | s, j ] =
P d [ s (cid:48) | s, ω d ( s )]
= P d [ s (cid:48) | s ] .
Cost – if it depends on the action ω (from Deﬁnition 3):
[Equation]
State-item function (from Deﬁnition 4):
[Equation]
In words, given state s , all items included in the deterministic (resp. optimal) slate have the samestate-item function value, equal to that of the whole slate.
Lemma 1.
For any stationary policy d (and also the optimal d ∗ ), it holds that Q d ( s, j ) =
V d ( s ) ,∀ j ∈ ω
d ( s ) .
Speciﬁcally for the optimal,
[Equation]
Proof of Lemma 1.
For stationary deterministic policies we have from (22) that Q d ( s, j ) =
Q d ( s, ω d ( s )) , ∀ j ∈ ω
d ( s ) .
It also holds from (17) that V d ( s ) =
Q d ( s, ω d ( s )) .
We now continue to the proof of Theorem 2.
Applying the optimal deterministic policy to the stateaction equations for item-frequencies from Theorem 1 (see formulation in 14) we get, ( c ∗ :
= c d ∗ )
Q d ∗ ( s, j ) =
c ∗ ( s, j ) + λ (cid:88)
[Equation]
From (23) in Lemma 1 it holds that Q d ∗ ( s (cid:48) , j ) =
V d ∗ ( s (cid:48) ) , ∀ j ∈ ω ∗ ( s (cid:48) ) .
Then, necessarily
Q d ∗ ( s (cid:48) , j ) ≤ Q d ∗ ( s (cid:48) , k ) , ∀ k / ∈ ω ∗ ( s (cid:48) ) , otherwise V d ∗ ( s (cid:48) ) would not be the optimal value.
Inother words, V d ∗ ( s (cid:48) )
=
min (cid:96) ∈K Q d ∗ ( s (cid:48) , (cid:96) ) .
From (11) we get (21) c ∗ ( s, j ) =
c d ∗ ( s, ω ∗ ( s )) for allj ∈ ω ∗ ( s ) .
For (cid:96) / ∈ ω ∗ ( s ) , we know that r ∗ s,(cid:96) = 0 and
Decomposed SARSA and Q-learning for slate actions
Consider a sequence of states, slate-actions and costs over discrete time-slots t = 1 , 2 , . . .
as( S 1 = s, A 1 = ω, c
=
c
( s ) , S
= s (cid:48) , A 2 = ω (cid:48) , c
2 =
c ( s (cid:48) ) , . . . ) .
The transition from state S 1 to
S 2 depends on the slate recommended by the agent, and the unknown user behaviour to select oneof the items in the slate; the user is allowed to disregard the slate and select another item of theirown preference.
The vanilla SARSA Sutton and Barto [2018] is an on-policy TD (0) method, whichupdates the state-action values Q ( s t , ω t ) as follows
[Equation]
The slate-actions ω t +1 can follow the (cid:15) -greedy exploration policy.
We denote it by π (cid:15) ; based onthis, the greedy slate
ω ∗ ( s ) that minimises Q ( s, ω ) is chosen with probability 1 − (cid:15) and a uniformlyrandom slate ω ∈ A ( s ) is chosen with probability (cid:15) .
This implementation requires per state s ∈ S ,all (cid:0) KN (cid:1) combinations of Q-values stored in memory.
Furthermore, all these combinations need to betraversed when searching for the minimum in the greedy step.
SlateFree updates.
We can use the decomposition of the Bellman equations in Theorem 1, to proposea SlateFree-SARSA policy.
We remind that a state-action pair ( s, ω ) corresponds to N state-itempairs ( s, j ) one per j ∈ ω .
The update can be written based on (13),[ SlateFree − SARSA ]
For all N items in the slate j ∈ ω t :
[Equation]
For the (cid:15) -greedy policy, each time state s t is visited, the transition to s t +1 is sampled from theunknown transition probability P (cid:15) [ s t +1 | s t , j ] , which depends on user preferences, but also on thepolicy (cid:15) -greedy, which can change over time in the transient regime.
The frequencies r (cid:15)s (cid:48) ,k in (13)do not appear above, because the new action batch ω t +1 is a sample of the policy π (cid:15) (and thefrequencies r (cid:15) ).
In fact it can be easily shown that (cid:80) k ∈ ω t +1 Q ( s t +1 , k ) /N is just a one-sampleunbiased estimator of (cid:80) k
∈K r s (cid:48) ,k Q ( s (cid:48) , k ) /N .
The cost c (cid:15) ( s t , j ) also depends on the (cid:15) -greedy; in thespecial case that it depends on the state only, we replace c (cid:15) ( s t , j ) =
c ( s t ) , otherwise the cost per itemwill evolve over time and needs to be recalculated using Def. 3, keeping track of ˜ r (cid:15) , ˜ π estimators.
Similar to SARSA, we can introduce a decomposed version of the Q-learning algorithm (Watkins and
Dayan [1992]), which is an off-policy TD (0) method, where the Q functions are updated based onthe optimal action policy, although the actions may follow some other (say (cid:15) -greedy) policy.
Then asabove, we can use Theorem 2 to propose the updated step of the state-item functions following (18),[ SlateFree − Q ]
For all N items in the slate j ∈ ω t :
[Equation]
In the special case that it depends on the state only, we replace by c ( s t , ω t ) =
c ( s t ) .
The implementation of SlateFree (both -SARSA and -Q variations) requires per state s at most K values storedin memory (if we avoid self-loops, then recommending the same item is not an option).
Finding the best slate.
In the exploitation phase of the (cid:15) -greedy policy, we need to decide which
N -slate is optimal.
We are given, however, for each state s , not the state-action values Q ( s, ω ) butrather the K state-item values Q ( s, j ) .
Since we are looking for a stationary deterministic optimalpolicy, then we can apply the results from Section 2.3.
Speciﬁcally, we have proved in Lemma 1 that
Q d ∗ ( s, j ) = min ω ∈A ( s ) Q d ∗ ( s, ω ) , ∀ s ∈ ω ∗ ( s ) , meaning that all state-item values will be equal, forthe items included in the optimal slate (or more generally in the slate of the deterministic policy).
Hence, we need only select in the greedy phase, the N items with smallest Q ( s, j ) values, both in the-SARSA and -Q version of SlateFree .
The update steps in (25) and (26) have important novelties compared to alternatives, as in e.g. Ieet al.
[2019, eq.14, 15].
They are strictly model-free and do not need any prior knowledge overthe environment.
Also, costs that depend on both state and action are allowed.
Hence, SlateFreeuses N parallel updates to learn any stationary environment over time, using a more compact Qfunction representation, compared to the non-
Numerical evaluation
In this section, we evaluate numerically the performance of SlateFree (both -Q and -SARSAvariations), against two methods from the literature: (i) the standard Q-learning and SARSA tabularmethod (called Vanilla-Q and Vanilla-SARSA , where all possible (cid:0) KN (cid:1) action-slate combinations areaccounted for, each having its own Q-value per state; furthermore against (ii) the proposed in Ieet al.
[2019] method SlateQ with greedy slate selection in exploitation phase.
Our environment isa recommendation system where a user starts their viewing episode from a certain item, and thesystem recommends a slate of N (unordered) items, excluding the currently viewed item.
Eachepisode has average length (1 − λ ) − 1 steps.
The experience is repeated over T epis ≥ 1 episodes.
The great challenge is to test the SlateFree method on user behaviours whose solution needs to becombinatorially searched.
We evaluate three such artiﬁcial user choice models:
User-1 The user has a ﬁxed retention probability α ∈ [0 , 1] per step to select one of the N
recommended items uniformly at random, and (1 − α ) probability to disregard the recommendations and select on their own for one of the K library items uniformly at random.
User-2 The user has a set X of undesired items that they would never click on.
Then, this behaviour
is similar to user-1, with the difference that user-2 will choose either from the recommendeditems (with probability α ), or from the whole library (with 1 − α ) one item uniformly atrandom among all items, ignoring in both cases those items in set X .
User-3 The user has a set Y of must-include global items.
This user does not follow a retentionprobability α .
Instead, they will select among the recommended items at random, as long as
at least one item from Y is included in the recommendation slate.
A. Small scenario.
We ﬁrst study a small size scenario with K = 10 and N = 4 .
The number ofpossible combinations per state is (cid:0) K − 1 N (cid:1) = 126 , where we exclude recommendation of the currentlyviewed item.
With SlateFree we get a reduction in memory for the Q-table from 10 × 126 to10 × 9 .
The costs per item are all high
+ z i , where z i ∼ Uniform (0 , 4) , but there are four itemswith lower cost, namely c 0 = 5 + z 0 , c 1 =
+ z 1 , c 7 = 4 + z 7 and c 9 = 8 + z 9 (remember it is aminimisation problem).
The discount is ﬁxed in all experiments to λ = 0 . 85 .
For user-1 and -2, theretention is α = 0 .
For user-2 the exclusion set X = { 0 , 1 , 8 } .
For user-3 we select X
=
Y asin user-2.
The learning rate is γ = 0 .
004 and the (cid:15) -greedy GLIE strategy has ﬁxed (cid:15) = 0 .
for theexploration probability.
We consider as number of episodes for the evaluation T epis = 600 K .
Eachepisode is a walk of the user on the library of K items, with mean length (1 − λ ) − 1 = 6 , 67 views.
The evaluation for the three types of users and T epis =
600 K is shown in Fig.
1 (TOP-row).
We useon the x-axis logarithmic scale of episodes.
The value per episode has high variance and so we smooththe results within a window of 200 episodes.
SlateFree-Q and -SARSA converge for any user typein around 10 K episodes, an order of magnitude faster than the Vanilla-Q and Vanilla-SARSA.
Also,the average value after convergence is the same for the two methods, indicating that SlateFreeconverges to the optimal value function.
Both SlateFree and SlateQ converge to the optimal value,but we will see this is not true for larger catalog and dimension instances.
Our method shows fasterand steeper convergence than SlateQ in all users, because SlateQ updates the item-Q values only forthe single selected item, whereas SlateFree for all N items included in the slate.
B. Larger Scenario.
Next we evaluate the convergence in a more difﬁcult scenario with K = 100and N = 10 , which corresponds to (cid:0) 9910 (cid:1) ≈ 15 · 10 12 combinations.
This problem is not tractable for
Vanilla-Q or Vanilla-SARSA.
Hence, we only show results for SlateFree-Q , SlateFree-SARSAand SlateQ in Fig.
1 (BOTTOM-row).
The value per episode has high variance and so we smooththe results within a window of 1000 episodes.
Now, both SlateFree-Q and -SARSA can solve allthree user cases within ≈ 500 K episodes, whereas SlateQ seems to learn and improve over time, butcannot solve for any user, at least within the T epis = 1 M episodes.
C. Insensitivity in N .
Our evaluations show that the convergence time given some library size Kbecomes almost insensitive to the dimension size N .
To illustrate this, we simulate user-3 for acatalog size K = 10 and various sizes of N ∈ { 1 , 2 , 3 , 4 , 5 } .
The results are illustrated in Fig. 2(left).
One can observe that surprisingly the slowest converging curve is for N = 1 , whereas forthe higher N almost all curves converge before T epis = 10 K .
The reason for the poor convergencebehaviour of N = 1 is probably due to the fact that each step in the episode contributes a singleupdate of the state-item functions, whereas for N > 1 the multiple parallel updates accelerate the process considerably.
This is the same reason why SlateFree-Q in Fig.
1 shows a steeper learningcurve compared to SlateQ, where the latter updates a single state-item function per step.
D. Effect of parallel updates.
We investigate the role of N parallel updates in the convergence of the
SlateFree-Q algorithm.
More speciﬁcally, for user-1, and the small scenario K = 10 and N = 4we evaluate the algorithm using a different number of updates per step at each evaluation.
Aside theproposed algorithm which updates all four items in the slate per step
, in the others we allow threeitems per step, two items, and ﬁnally a single item to update.
We plot our results in Fig.
2 (centre).
Weobserve that the complete method with all four updates converges in 10 K episodes already (shown inred).
For three updates per step, the method seems to converge (green curve) to a value close to theoptimum, albeit very slowly.
For two and a single update (brown and pink curves) we observe thatthe method gradually improves over the episodes but even in 150 K events it has not converged to theoptimum.
To conclude, the plot shows that it is necessary to do all N parallel updates per step for themethod to converge to the best possible value, and fast.
E. Dependence of cost on both state and action-slate.
We study now how the performance of
SlateFree is affected when the cost depends on both the current state and the entire action-slatec ( s t , ω t ) .
Such an option is not supported by SlateQ Ie et al.
We now modify the cost so thata penalty = 42 is applied to all Q ( s t , j ) where j ∈ ω t are the items participating in the recommendedslate, whenever the user does not follow (rejects) the recommendation slate.
Obviously this penalty isslate-dependent.
We illustrate the performance of all methods in Fig.
2 (right).
We observe that thedecomposed SlateFree converges to the optimal solution for both -Q and -SARSA variations, sameas Vanilla-Q and Vanilla-SARSA.
SlateQ from Ie et al.
[2019] fails to converge to the minimal value.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] In Section 1 we include references where the code can be downloaded from Dropbox SlateFree Authors [2022b].
One can alsorun the code on Google Colab SlateFree Authors [2022a].
In both cases the Apache 2.
licence has been included on the relevant jupyter-notebook.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] The limitations are described inlines 60-64 in page 2, Section 1.
These include the necessary assumptions to prove themain theorems.
In the simulations we show that the method performs in practice verywell even when assumption (ii), related to cost, is not satisﬁed.
The necessity of Nparallel updates is described and illustrated in page 9, Section 4-D.(c)
Did you discuss any potential negative societal impacts of your work?
[No] The workis about the solution of a standard mathematical problem of Slate-MDPs and has boththeoretical and practical ﬂavour, but general applicability to various practical cases, nota speciﬁc one.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] As mentionedabove, these are given in lines 60-64 in page 2, Section 1.
These assumptions are furtherrepeated when necessary in the presentation of Theorems 1 and 2.
In the simulationswe show that the method performs in practice very well even when assumption (ii),related to cost, is not satisﬁed.
We also argue about the necessity of N parallel updates.(b)
Did you include complete proofs of all theoretical results?
[Yes] The available 9 pageswhere sufﬁcient to include all proofs.3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The codeis included as a URL in the references.
Speciﬁcally, the code can be downloadedfrom Dropbox through reference SlateFree Authors [2022b
] and can be used online in
Google Colab SlateFree Authors [2022a].
It is the exact code that we used to produce
all ﬁgures.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were
chosen)?
[Yes] All details and hyperparameters are explicitly stated in the Numerical
Evaluation Section
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[No] The randomness in the initialization of the algorithmcould be related to the initial value of the Q-matrix.
We did not explicitly run many
experiments with various initialisations, but we let this option available on the code.
The aim is to identify whether and how fast our algorithm converges to the optimalpolicy compared to alternatives.
Various initialisations could start closer or further
away from the
optimal solution but this would hold for all policies/methods tested andthe relative results would be similar to what we show.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No] The experiments we perform are of
small scale to validate the theory.
We only used local CPUs and the x-axis of the plots
are in "number of learning episodes", which is a measure of time.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[No] All code was developed by the authors.
(b) Did you mention the license of the assets?
[No]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
We include the code for the numerical evaluations as a URL on the paper references (for
Dropbox SlateFree Authors [2022b] and for Google Colab SlateFree Authors [2022a
])(d ) Did you discuss whether and how consent was obtained from
people
whose
data
you’re using/curating? [No] Does not apply .(e
) Did you discuss
whether the
data you are using/ curating contains personally identiﬁableinformation or offensive content
No
Data content is not relevant to our numericalevaluation.
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[No] Does not apply.
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[No] Does not apply.
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[No] Does not apply.
[Table]
A Appendix
[Table]
No appendix accompanies the submission.
The 9 pages and extra references are self-sufﬁcient.
Two
[Table] URLs (for Dropbox SlateFree Authors [2022b] and for Google Colab SlateFree Authors [2022a])link to the code used for the numerical evaluations.