E FFICIENT R EINFORCEMENT L EARNING IN R ESOURCE A LLOCATION P ROBLEMS T HROUGH P ERMUTATION I NVARIANT M ULTI - TASK L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
One of the main challenges in real-world reinforcement learning is to learn successfully from limited training samples.
We show that in certain settings, the available data can be dramatically increased through a form of multi-task learning, by exploiting an invariance property in the tasks.
We provide a theoretical performance bound for the gain in sample efﬁciency under this setting.
This motivates a new approach to multi-task learning, which involves the design of an appropriate neural network architecture and a prioritized task-sampling strategy.
We demonstrate empirically the effectiveness of the proposed approach on two real-world sequential resource allocation tasks where this invariance property occurs: ﬁnancial portfolio optimization and meta federated learning.
I NTRODUCTION
Sample efﬁciency in reinforcement learning (RL) is an elusive goal.
Recent attempts at increasing the sample efﬁciency of RL implementations have focused to a large extent on incorporating models into the training process: Xu et al.
; Clavera et al.
; Zhang et al.
; Berkenkamp et al.
(2017); Ke et al.
; Yarats et al.
; Huang et al.
Chua et al.
; Serban et al.
The models encapsulate knowledge explicitly, complementing the experiences that are gained by sampling from the RL environment.
Another means towards increasing the availability of samples for a reinforcement learner is by tilting the training towards one that will better transfer to related tasks: if the training process is sufﬁciently well adapted to more than one task, then the training of a particular task should be able to beneﬁt from samples from the other related tasks.
This idea was explored a decade ago in Lazaric & Ghavamzadeh (2010) and has been gaining traction ever since, as researchers try to increase the reach of deep reinforcement learning from its comfortable footing in solving games outrageously well to solving other important problems.
Yu (2018) discusses a number of methods for increasing sample efﬁciency in RL and includes experience transfer as one important avenue, covering the transfer of samples, as we do here, transfer of representation or skills, and jumpstarting models which are then ready to be quickly, i.e. with few samples, updated to different tasks.
D’Eramo et al.
(2020) address the same idea, noting that multi-task learning can improve the learning of each individual task, motivated by robotics-type tasks with underlying commonality, such as balancing a single vs. a double pendulum, or hopping vs. walking.
We are interested in exploiting the ability of multi-task learning to solve the sample efﬁciency problem of RL.
Our setting does not apply to all problem classes nor does it seek to exploit the kind of physical similarities found in robotics tasks that form the motivation of Lazaric & Ghavamzadeh (2010);
D’Eramo et al.
Rather, we show that there are a number of reinforcement learning tasks with a particular fundamental property that makes them ideal candidates for multi-task learning with the goal of increasing the availability of samples for their training.
We refer to this property as permutation invariance.
It is present in very diverse tasks: we illustrate it on a ﬁnancial portfolio optimization problem, whereby trades are executed sequentially over a given time horizon, and on the problem of meta-learning in a federated supervised learning setting.
Permutation invariance in the ﬁnancial portfolio problem exhibits itself as follows: consider the task of allocating a portion of wealth to each of a number of ﬁnancial instruments using a trading policy. If the trading policy is permutation invariant, one can change the order of the instruments without changing the policy.
This allows one to generate multiple portfolio optimization tasks from a given set of ﬁnancial instruments. A commonality between applications that have this property is that they concern sequential resource allocation: at each time step, the resource allocation scores the quality of each available candidate entity (for example a ﬁnancial instrument in the above example), then based on those scores, apportions out the resource (the total wealth to invest, in the above example) among the entities at that time step, so that over the horizon of interest, the reward is maximized.
Sequential resource allocation problems include applications such as sequential allocation of budget, sequential allocation of space, e.g. in IT systems, hotels, delivery vehicles, sequential allocation of people to work slots or appointments, etc.
Many such applications possess permutation invariance in that the ordering of the entities, i.e. where the resources are allocated, can change without changing the resulting optimal allocation.
We show that under this form of permutation invariance, it is possible to derive a bound on the performance of the policy.
The bound is an extension of that of Lazaric & Ghavamzadeh (2010), and while similar to, provides additional information beyond the bound of D’Eramo et al.
We use the bound to motivate an algorithm that allows for substantially improved results as compared with solving each task on its own.
The bound and the algorithm are ﬁrst analyzed on a synthetic problem that validates the bound in our theorem and conﬁrms the multi-task gain that the theory predicts. Hessel et al.
; Bram et al.
(2019) have cautioned against degrading of the performance on each task when some tasks bias the updates to the detriment of others in multi-task learning.
They claim that some tasks have a greater density or magnitude of in-task rewards and hence a disproportionate impact on the learning process.
In our setting, deleterious effects of some tasks on others could also arise.
The algorithm we propose handles this through a form of prioritized sampling, where priorities are put on the tasks themselves, and acts like a prioritized experience replay buffer, applied to a multi-task learning problem.
We show empirically that the priorities thus deﬁned protect the overall learning problem from the deleterious effects that unrelated or unhelpful tasks could otherwise have on the policy.
The contributions of this work are as follows: (1) we identify the permutation invariance property of the class of reinforcement learning problems involving sequential resource allocation, (2) we deﬁne a method to increase sample efﬁciency in these reinforcement learning problems by leveraging this property of permutation invariance; (3) we provide a theoretical performance bound for the class of problems; (4) we validate experimentally the utility of permutation variance on sample efﬁciency as well as the validity of the bound on a synthetic problem; and (5) we illustrate two real-world RL resource allocation tasks for which this property holds and demonstrate the beneﬁts of the proposed method on sample efﬁciency and thus also on the overall performance of the models.
R ELATED WORK
A notable ﬁrst stream of work on leveraging multi-task learning for enhancing RL performance on single tasks can be found in Wilson et al.
(2007); Lazaric & Ghavamzadeh (2010) which consider, as we do,
that there is an underlying MDP from which the multiple tasks can be thought to derive.
They use however a Bayesian approach and propose a different algorithmic method than ours.
Our results extend performance bounds by Lazaric et al. (2012) on single-task RL.
As noted by Yu (2018), jumpstarting, or distilling experiences and representations of relevant policies is another means to increasing sample efﬁciency in solving a new but related problem.
Rusu et al.
(2016) uses this idea in so-called progressive neural networks and Parisotto et al.
(2015) leverage multiple experts to guide the derivation of a general policy.
With a similar objective, Teh et al.
(2017) deﬁne a policy centroid, that is, a shared distilled policy, that captures the commonalities across the behaviors in the tasks.
In all of these distillation-type methods, the tasks considered are simple or complex games.
Teh et al.
(2017) note that their policy centroid method, distral, is likely to be affected by task interference, in that differences across tasks may degrade the performance of the resulting policy of any of the constituent tasks.
This topic was studied by Hessel et al.
; Bram et al.
Hessel et al.
(2018) proposed a solution to this by extending the so-called PopArt normalization van Hasselt et al.
(2016) to re-scale the updates of each task so that the different characteristics of the task-speciﬁc reward do not skew the learning process.
Bram et al.
(2019) use a different approach that learns attention weights of the sub-networks of each task and discards those that are not relevant or helpful.
Vuong et al.
; D’Eramo et al.
(2020) are, like our work, concerned with sharing of experiences to facilitate a more sample-efﬁcient learning process.
Vuong et al.
(2019) suggest identifying the shared portions of tasks to allow sharing of samples in those portions.
The work of D’Eramo et al.
(2020) is in some ways quite similar to ours: the authors’ goal is the same and they derive a bound as we do on the performance in this setting.
However, their setting is different in that their tasks have both shared and task-speciﬁc components, and their bound becomes tighter only as the number of tasks increases.
In our setting, we do not require a task-speciﬁc component, and we are able to show how the distance between the MDPs of each task, in addition to the number of tasks, affects the strength of the bound.
Recently, permutation invariance has been exploited in deep multi-agent reinforcement learning (Liu et al., 2019) where the invariance properties arise naturally in a homogeneous multi-agent setting.
Their work employs permutation invariance in learning the critic whereas in our case the entire learned policy employs permutation invariance.
P RELIMINARIES
We begin by deﬁning notation.
For a measurable space with domain X , let S ( X ) denote the set of probability measures over X , and B ( X ; L ) the space of bounded measurable functions with domain
X and bound 0 < L < ∞ .
For a measure ρ ∈ S ( X ) and a measurable function f : X → R , the l 2 ( ρ ) -norm of f is (cid:107) f (cid:107) ρ , and for a set of n points X 1 , · · · , X n ∈ X , the empirical norm, (cid:107) f (cid:107) n is (cid:107) f (cid:107) 2 ρ
=
(cid:90) f
( x ) 2 ρ ( dx ) and (cid:107) f (cid:107) 2 n = 1 n n (cid:88) t
= f
( X t ) 2 .
Let (cid:107) f (cid:107) ∞
=
sup x ∈X | f ( x ) | be the supremum norm of f .
Consider a set of MDPs indexed by t .
Each MDP is denoted by a tuple M t =
(cid:104)X , A , R t , P t , γ (cid:105) , where X , a bounded closed subset of the s -dimensional Euclidean space, is a common state space; A is a common action space,
R t : X × A → R is a task speciﬁc reward function uniformly bounded by R max , P t is a task speciﬁc transition kernel such that P t ( ·| x, a ) is a distribution over X for all x ∈ X and a ∈ A , and γ ∈ (0 , 1) is a common discount factor.
Deterministic policies are denoted by π : X → A .
For a given policy π , the MDP M t is reduced to a Markov chain M πt =
(cid:104)X , R πt , P πt , γ (cid:105) with reward function
R πt ( x ) =
R t ( x, π ( x )) , transition kernel P πt ( ·| x ) =
P t ( ·| x, π ( x )) , and stationary distribution ρ πt .
The value function V πt for MDP t is deﬁned as the unique ﬁxed-point of the Bellman operator
T πt : B ( X ; V max =
R max / (1 − γ ))
→ B ( X ; V max ) , given by
[Equation]
Let π ∗ t denote the optimal policy for M t .
The optimal value function V π ∗ t t for M t is deﬁned as the unique ﬁxed-point of its optimal Bellman operator T π ∗ t t which is deﬁned by
[Equation]
To approximate the value function V , we use a linear approximation architecture with parameters α ∈ R d and basis functions ϕ i ∈ B ( X ; L ) for i
= 1 , · · · , d . Let ϕ ( · ) =
( ϕ 1 ( · ) , · · · , ϕ d ( · ))
T ∈ R
d be the feature vector and F the linear function space spanned by basis functions ϕ i .
Thus, F = { f α | α ∈ R d and f α ( · ) =
ϕ ( · ) T α } .
Consider a learning task to dynamically allocate a common resource across entities U t ⊆
U .
Each t corresponds to a task, but for now take t to be an arbitrary ﬁxed index.
At each time step n , the decision maker observes states x n = ( x i,n ) i ∈U t of the entities, where x i,n is the state of entity i , and takes action a n = ( a i,n ) i ∈U t , where a i,n is the share of the resource allocated to entity i .
The total resource capacity is normalized to 1 for convenience.
Therefore, allocations satisfy 0 ≤ a i,n ≤ 1 and (cid:80) i ∈U t a i,n = 1 .
We consider policy π θ ( x n ) parameterized by θ .
Assume that we have access to the reward function R t as well as a simulator that generates a trajectory of length N given any arbitrary policy π θ .
The objective of the learning task is to maximize
[Equation]
In many settings, N is small and simulators are inaccurate; therefore, trajectories generated by the simulator are poor representations of the actual transition dynamics.
This occurs in batch RL where trajectories are rollouts from a dataset.
In these cases, policies overﬁt and generalize poorly.
T HEORETICAL R ESULTS
We introduce ﬁrst a property that we term permutation-invariance for the policy network that can be shown to help signiﬁcantly reduce overﬁtting.
Deﬁnition 1 (Permutation Invariant Policy Network)
A policy network π θ is permutation invariant if it satisﬁes π θ ( σ ( x ))
=
σ ( π θ ( x )) for any permutation σ .
Permutation invariant policy networks have signiﬁcant advantages over completely integrated policy networks.
While the latter are likely to ﬁt correlations between different entities, this is not possible with permutation invariant policy networks as they are agnostic to identities of entities.
Therefore, permutation invariant policy networks are better able to leverage experience across time and entities, leading to greater efﬁciency in data usage.
Moreover, observe that if the transition kernels can be factored into independent and identical transition kernels across entities, then the optimal policy is indeed permutation invariant.
Our main theoretical contributions start with an extension of results from Lazaric et al.
(2012), where a ﬁnite-sample error bound was derived for the least squares policy iteration (LSPI) algorithm on a single task.
Lazaric et al.
(2012) provided a high-probability bound on the performance difference between the ﬁnal learned policy and the optimal policy, of the form c 1 + c 2 / √ N , where c 1 and c 2 are constants that depend on the task and the chosen feature space, and N is the number of training examples.
We extend their result by showing that, as long as tasks are (cid:15) -close to each other (with respect to a similarity measure we deﬁne later), the error bound of solving each task using our multi-task approach has the form c 1 + c 2 / √ NT + c 3 (cid:15) , where T is the number of tasks and cis a task-dependent constant. Speciﬁcally, our theorem provides a general result and performance guarantee with respect to using data from a different but similar MDP. Deﬁnition 1 provides a basis for generating many such MDPs.
Finally, the beneﬁt of doing so shall be provided by Corollary 2.
Thus, provided (cid:15) is small, a given task can beneﬁt from a much larger set of NT training examples.
In addition to the assumptions of Lazaric et al. (2012), we extend the deﬁnition of second-order discounted-average concentrability, proposed in Antos et al.
and deﬁne the notion of ﬁrst-order discounted-average concentrability.
The latter will be used in our main result, Theorem 1.
Assumption 1 There exists a distribution µ ∈ S ( X ) such that for any policy π that is greedy with respect to a function in the truncated space ˜ F , µ ≤ Cρ πt for all t , where C < ∞ is a constant.
Given the target distribution σ ∈ S ( X ) and an arbitrary sequence of policies { π m } m ≥ 1 , let
[Equation]
We assume that C (cid:48) σ,µ , C (cid:48)(cid:48) σ,µ < ∞ , and deﬁne ﬁrst and second order discounted-average concentrability of future-state distributions as follows:
[Equation]
Theorem 1 (Multi-Task Finite-Sample Error Bound) Let M =
(cid:104)X , A , R, P, γ (cid:105) be an MDP with reward function R and transition kernel P .
Assume A ﬁnite.
Denote its Bellman operator by
[Equation]
Given a policy π , deﬁne the Bellman difference operator between M t and M to be D πt V =
T π t V − T π V .
Apply the LSPI algorithm to M , by generating, at each iteration k , a path from M of size N , where N satisﬁes Lemma 4 in Lazaric et al.
Let V − 1 ∈ ˜ F be an arbitrary initial value function, V 0 , · · · , V K − 1 ( ˜ V 0 , · · · , ˜ V K − 1 ) be the sequence of value functions (truncated value functions) generated by the LSPI after K iterations, and π k be the greedy policy w.r.t.
the truncated value function ˜
V k − 1 .
Suppose also that
[Equation]
and
[Equation]
π
[Equation] ˜ [Equation] Then, for constants c 1 , c 2 , c 3 , c 4 that are dependent on M , with probability 1 − δ (with respect to the random samples):
[Equation]
The proof is deferred to the Appendix.
Theorem 1 formalizes the trade off between drawing fewer samples from the exact MDP M t , versus drawing more samples from a different MDP M .
Importantly, it shows how to beneﬁt from solving a different MDP, M , when: (a) additional samples can be obtained from M , and (b) M is not too different from M t .
In particular, the distance measure is simply the distance between the Bellman operators of the MDPs, which can be bounded if the difference in both the transition and reward functions are bounded.
In recent work, a performance bound for multi-task learning was given in Theorem 2 and 3 of D’Eramo et al.
However, the authors used a different setup containing both shared and task-speciﬁc representations, and their focus was on showing that the cost of learning the shared representation decreases with more tasks.
They did not show how the similarity or difference across tasks affects performance.
In contrast, our setup does not contain task-speciﬁc representations, and our focus is on how differences across MDPs impact the beneﬁt of having more tasks (and consequently more samples).
We show this in Corollary 1 and Corollary 2.
Remark 1 While our theoretical results are based on LSTD and LSPI and assume ﬁnite action space, our approach is applicable to a wide range of reinforcement learning algorithms, including policy gradient methods and to MDPs with continuous action spaces.
Deriving similar results for a larger family of models and algorithms remains an interesting, albeit challenging, future work.
Permutation invariant policy networks allow using data from the global set of entities U .
Since the policy network is agnostic to the identities of the entities, one can learn a single policy for all tasks, where each task t ∈ [ T ] is a resource allocation problem over a subset of entities U t .
For notational simplicity, assume that all tasks have the same number of entities, and all trajectories are of equal length N .
Our approach can, however, be readily extended to tasks with different numbers of entities and different trajectory lengths.
Permutation invariance allows a large set of MDPs to leverage the result of Theorem 1.
In the next section we shall provide an algorithm, motivated by the following corollaries, and a prioritized sampling strategy for this setting that drives signiﬁcantly greater sample efﬁciency for the original task.
The sampling strategy also helps to stabilize the learning process, reducing the risk of deleterious effects of the multi-task setting, as discussed by Teh et al.
and addressed in works such as Hessel et al.
; Bram et al.
Corollary 1 Let [ T ] be a set of similar tasks such that their distance from the average MDP, given by
[Equation]
is bounded by (cid:15) as deﬁned in Theorem 1.
Let N be the number of samples available in each task.
Let π K be the policy obtained at the K th iteration when applying LSPI to the average MDP.
Then, the suboptimality of the policy on each task is O (1 / √ NT )
+ O ( (cid:15) )
+ c for some constant c (where suboptimality is deﬁned according to Theorem 1).
Recall that each task is formed by selecting a subset U t of entities from the global set U .
We thus have the following sample gain that can be attributed to the permutation invariance of the policy network.
Corollary 2 (Gain in Sample Efﬁciency from Permutation Invariance) Let M = |U| and m =
|U t | .
Given ﬁxed M and m , there are T = (cid:0) Mm (cid:1) ≥ (cid:0) Mm (cid:1) m different tasks.
Then, by Cor. 1, assuming all pairs of tasks are weakly correlated, the potential gain in sample efﬁciency is exponential in m .
Disregarding correlation between samples from tasks with overlapping entities Corollary 1 and
Corollary 2 together suggest that the (up to) exponential increase in the number of available tasks can signiﬁcantly improve sample efﬁciency as compared to learning each task separately.
E XPLOITING P ERMUTATION I NVARIANCE THROUGH M ULTI - TASK R EINFORCEMENT L EARNING
Our approach to exploiting permutation invariance is via multi-task reinforcement learning, where each “task” corresponds to a particular choice of subset U t ⊂ U . Furthermore, for each task, we enforce permutation invariance among the entities i by forcing the neural network to apply the same sequence of operations to the state input x i of each instrument through parameter sharing.
The proposed method, shown in Algorithm 1, learns a single policy by sampling subsequences of trajectories from the different MDPs.
At each step, we sample a task t according to a distribution deﬁned by task selection policy p .
Then, a minibatch sample B t is drawn from the replay buffer for task t , and gradient descent is performed using the sampled transitions B t (alternatively, samples can be generated using policy rollouts for the speciﬁc task).
Separate replay buffers maintained for each task are updated only when the corresponding task is being used.
In contrast with other active sampling approaches in multi-task learning, our approach maintains an estimate of the difﬁculty of each task t as a score, s t .
After each training step, we update the score for only the sampled task based on minibatch B t , avoiding evaluation over all the tasks.
The scoring functions depend on the sampled minibatch; to reduce ﬂuctuations in scores for each task, exponential smoothing is applied s t ← γs t + (1 − γ ) · scorer ( B t ) .
We propose a stochastic prioritization method that interpolates between pure greedy prioritization and uniform random sampling.
Our approach is similar to prioritized experience replay (PER) by Schaul et al.
but while classical PER prioritizes samples, we prioritize tasks.
The probability of sampling task t is p t =
s αt / (cid:80) t (cid:48)
s αt (cid:48) , where the exponent α determines the degree of prioritization, with α = 0 corresponding to the uniform case.
We correct for bias with importance-sampling (IS) weights w t = 1 / ( Tp t ) β , that compensate for non-uniform probabilities if β = 1 .
We normalize weights by 1 / max t w t .
Tasks on which the reward variance is high can be interpreted as having more challenging samples, hence reward variance can be used as a scoring function.
Algorithm
Initialize policy network π θ
Initialize replay buffers R 1 , . .
, R T
Initialize time steps n 1 ← 1 , . .
. , n T ←loop
Select a task t ∼ p to train on
Sample a random minibatch B t of transitions ( x n , a n , r n , x n +1 ) from R t
Update policy θ using B t and chosen RL approach (correcting for bias using IS weights w )
Update score s t ← γs t + (1 − γ ) · scorer ( B t ) Update ALL selection probabilities p and IS weights w for n = n t , . . , min { n t + n e , N } do
For task t , select action a n according to current policy and exploration noise
Execute action a n , and observe reward r n and new state x n +
Store transition ( x n , a n , r n , x n +1 ) in R t end for
If n < N , update n t ← n + 1 , otherwise, update n t ←end loop
E XPERIMENTS 6. S YNTHETIC DATA
With the aim of validating the theory presented in Section 4, we deﬁne a synthetic example to explore the efﬁciency gain afforded by permutation invariance.
To do so, we control of the deviation (cid:15) between any two tasks, thereby empirically validating the main theoretical results.
Consider a resource allocation problem where the observed state x i for each entity i ∈ { 1 . . . m } is a single scalar x i ∈ [0 , 1] . The action space is the probability simplex, where each action [Figure]
a = ( a 1 . . .
a m ) indicates the fraction of resource allocated to each entity.
The reward function is
[Equation]
where β i is a weight parameter for each entity.
Note that when β i = β for all i , the reward function becomes R ( x, a ) =
( (cid:80) i x i a i ) + βH ( a ) where H is the Shannon entropy.
This implies that maximizing the reward involves a tradeoff between focusing resources on high x i or distributing them uniformly across all i .
Note that the reward function is permutation invariant, but that when we allow a varying β i over the entities, the function deviates from being perfectly permutation invariant.
We use the range max i β i − min
i β i as a stand-in for (cid:15) .
Let m = 10 .
For each (cid:15) , we run two experiments.
The ﬁrst examines the performance of policies trained by LSPI using N real examples drawn i.i.d from the state-action space, for N = 20 . . .
A small Gaussian noise is added to each reward to make learning harder.
The second experiment uses only 20 real examples, but augments the training set (up to N ) through random permutation of the real examples.
The ﬁrst two ﬁgures in Fig.
1 show the results for (cid:15) = 0 .
8 and (cid:15) = 0 , respectively.
Performance improves with N , as predicted by the 1 / √ N term in our error bound.
Note that in the experiment using only 20 real examples, a performance gain is achieved by using permuted examples; this corresponds precisely to the multi-task gain predicted by the 1 / √ NT term.
When (cid:15) is large, there is a signiﬁcant gap between the results of the two experiments, as predicted by the (cid:15) -term in the error bound.
The last plot in
Fig.
1 shows this gap at N = 2000 when (cid:15) varies from 0 to 0 .
6. R EAL - WORD DATA
We consider two real-world resource allocation settings: ﬁnancial portfolio optimization and meta federated learning.
Financial portfolio optimization is discussed below while meta federated learning is in the Appendix.
Given historical prices for a universe of ﬁnancial assets, U , the goal of task t is to allocate investments across a subset of assets U t ⊆
U .
The multiple tasks t thus correspond to multiple portfolios of instruments.
Permutation invariance will be of use in this setting since, from a given universe of instruments (e.g. the 500 instruments in the S&P 500), an exponential number of tasks can be generated, each with its own portfolio.
Consider now one such task. At the beginning of time period n , the action a i,n represents the fraction of wealth the decision maker allocates to asset i .
The allocations evolve over the time period due to changes in asset prices.
Let w i,n denote the allocation of asset i at the end of time period n .
We model the state of an asset using its current allocation and a window of its H most recent prices.
In particular, let v i,n denote the close price of asset i over time period n , and let y i,n = v i,n /v i,n − 1 denote the ratio of close prices between adjacent time periods 1 .
Then, the allocation in asset i at the end of time period n is given by
[Equation]
and the state of asset i at the beginning of time period n is given by
[Equation]
The change in portfolio value over period n depends on the asset prices and transaction costs incurred in rebalancing the portfolio from ( w i,n − 1 ) i ∈U t to ( a i,n ) i ∈U t .
The reward over period n is deﬁned as the log rate of return:
[Equation]
where β can be evaluated using an iterative procedure (see Jiang et al.
Deﬁning the reward this way is appealing because maximizing average total reward over consecutive periods is equivalent to maximizing the total rate of return over the periods.
To leverage this, we approximate β (
[Equation] i,
[Equation]
≈
[Equation]
| w i
[Equation] n [Equation] 1 − a i,n | , where c is a commission rate to obtain a closed-form expression for R t ( x n , a n ) (see Jiang et al.
We optimize using direct policy gradient on minibatches of consecutive samples
[Equation]
where n b is the ﬁrst time index in the minibatch, B the size of a minibatch, and w t the IS weight for task t .
As in Jiang et al.
, we sample n b from a geometric distribution that prioritises recent samples and implement replay buffers for each task.
A benchmark trading strategy is equal constantly-rebalanced portfolio (CRP) that rebalances to maintain equal weights.
As we noted earlier, ideally one would prefer for the scoring function to depend only on the minibatch B t .
A deviation from Equal CRP can be viewed as learning to exploit price movements , and is thus here we use this as the goal of the policy.
Prioritised MTL thus prioritises tasks which deviate from Equal CRP.
Note that the policy deviates from CRP only when proﬁtable.
Let
[Equation]
be the scoring of tasks in Prioritised MTL using mean absolute deviation of the minibatch allocation from Equal CRP.
Figure 2 (left) shows a scatter plot of the maximum score seen every 50 steps and the change in episode rewards in a single-task learning experiment, and (right) of the minibatch score and the maximum gradient norm for the minibatch.
Higher scores imply higher variance in the episode rewards and hence more challenging and useful samples.
The correlation between scores and gradient norms shows that our approach is performing gradient-based prioritisation, (see Katharopoulos & Fleuret (2018); Loshchilov & Hutter (2015); Alain et al.
but in a computationally efﬁcient manner.
The details of the dataset and parameter settings can be found in the Appendix.
Figure 3 shows the performance of the learned policies tested on 10 tasks drawn from out-of-sample instruments.
The policy network with weights initialized close to zero behaves like an
Equal CRP policy.
As noted, any proﬁtable deviation from Equal CRP implies learning useful trading strategies. The plots show that the MTL policies perform well on instruments never seen during training, offering a remarkable beneﬁt for using RL in the design of trading policies. Fig. 4 shows the performance of prioritised multi-task learning (MTL) versus single-task learning (STL) (i.e. learning a policy for each task independently on the instruments in the task).
We also show results for MTL without prioritised sampling, i.e., with α = 0 .
We consider 5 tasks and 30 tasks.
The plots show that prioritised MTL performs signiﬁcantly better than STL in both convergence time and ﬁnal achieved performance.
The performance with 30 tasks is signiﬁcantly better than the performance with 5 tasks, showing that our approach leverages the samples of the additional tasks.
Fig. 5 illustrates the typical behavior of a multi-task learning (MTL) and a single-task learning (STL) policy on the test period for tasks where multi-task policy performed signiﬁcantly better.
The single-task policy kept constant equal allocations while the multi-task policy was able to learn more complex allocations.
In ﬁnancial data, strongly trending prices do not occur often and are inherently noisy.
Multi-task learning with permutation invariance helps with both challenges, allowing the algorithm to learn more complex patterns in a given training period.
C ONCLUSIONS
We introduce an approach for increasing the sample efﬁciency of reinforcement learning in a setting with widespread applicability within the class of sequential resource allocation problems.
This property is permutation invariance: resources are allocated to entities according to a score, and the order can change without modifying the optimal allocation.
Under this property, we show that a bound exists on the policy performance.
This bound motivates a highly effective algorithm for improving the policy through a multi-task approach.
Using prioritized task-sampling, the method not only improves the reward of the ﬁnal policy but also renders it more robust.
We illustrate the property and the method on two important problems: sequential ﬁnancial portfolio optimization and meta federated learning, where the latter is provided in the Appendix.
A A PPENDIX
Theorem 1 Let M =
(cid:104)X , A , R, P, γ (cid:105) be an MDP with reward function R and transition kernel P .
Denote its Bellman operator by
[Equation]
Given a policy π , deﬁne the Bellman difference operator between M t and M to be D πt V =
T πt V − T π V .
Apply the LSPI algorithm to M , by generating, at each iteration k , a path from M of size N , where n satisﬁes Lemma 4 in Antos et al.
Let V − 1 ∈ ˜ F be an arbitrary initial value function, V 0 , · · · , V K − 1 ( ˜ V 0 , · · · , ˜ V K − 1 ) be the sequence of value functions (truncated value functions) generated by the LSPI after K iterations, and π k be the greedy policy w.r.t.
the truncated value function ˜
V k − 1 .
Suppose also that
[Equation]
and (cid:107)D π k t ˜ V
k − 1 (cid:107) µ ≤ (cid:15) ∀ k.
Then, with probability 1 − δ (with respect to the random samples), we have
[Equation]
Proof: For convenience, we will simply remove the task subscript whenever we refer to variables associated with M .
Deﬁne
[Equation]
From the proof of Lemma 12 in Antos et al.
(2008), we get
[Equation]
By applying the above inequality, and taking the absolute value on both sides point-wise, we get
[Equation]
V π ∗ t t
[Equation]
| where we used the fact that | V π ∗ − V π 0 | ≤ (2 R max / (1 − γ )) 1 .
Next, we derive upper bounds for | V π ∗ t t − V π ∗ | and | V π K − V π K t | .
[Equation]
a) Observe that
[Equation]
The ﬁrst inequality follows from the fact that π ∗ is optimal with respect to V π ∗ .
The second inequality follows from the taylor expansion of the inverse term.
By closely following the same steps, we also get
[Equation]
By splitting into positive and negative components and applying the above bounds
[Equation]
we get
[Equation]
(b) Observe that
[Equation]
The ﬁrst inequality follows from the fact that π K is optimal with respect to ˜ V K − 1 .
The second inequality follows from the taylor expansion of the inverse term.
By closely following the same steps, we also get
[Equation]
By splitting into positive and negative components and applying the above bounds
[Equation]
we get | V π K − V π K t
[Equation]
By applying the upper bounds from (a) and (b), we get
[Equation]
where we introduced the positive coefﬁcients
[Equation]
and the operators
[Equation]
Let λ K = (cid:104) 2(1 − γ K +2 )
(1 − γ ) 2 (cid:105) p .
Note that the coefﬁcients α k , α , and β , sum to 1 , and the operators are positive linear operators that satisfy A k 1 = 1 and B π 1 = 1 .
Therefore, by taking the p th power on both sides, applying Jensen’s inequality twice, and then integrating both sides with respect to σ ( x ) , we get
[Equation]
From the deﬁnition of the coefﬁcients c σ,µ ( m ) , we get
[Equation]
Therefore, it follows that
[Equation]
where e = max 0 ≤ k<K (cid:107) e k (cid:107) pp,µ .
The terms involving B π satisfy
[Equation]
Putting all these together, and choosing p = 2 , we get
[Equation]
t
[Equation]
[Table]
The desired result can then be obtained by applying the same steps as in the proof of Theoremin Lazaric et al.
(cid:3)
A. F INANCIAL P ORTFOLIO O PTIMIZATION : A DDITIONAL D ETAILS
The dataset consists of daily prices for 68 instruments in the technology and communication sectors from 2009 to 2019.
We use 2009–2018 for training and 2019 for testing.
To validate that our approach learns common features across instruments, and thus can transfer , we reserve 18 instruments not seen during training for further testing.
The global asset universe U used for training containsinstruments.
We construct tasks by randomly choosing a portfolio of |U t | = 10 instruments for each task.
We create a permutation invariant policy network by applying the same sequence of operations to every instrument state.
That is, for each instrument, the ﬂattened input prices are passed through a common
RNN with 25 hidden units and tanh activation, this output is concatenated with the latest allocation fraction of the instrument, and passed through a common dense layer to produce a score.
Instrument scores are passed to a softmax function to produce allocations that sum to one.
The smoothing parameter for the scores γ = 0 .
2 , α = 0 .
5 for the task prioritisation parameter and β = 1 . 0 to fully compensate for the prioritized sampling bias.
A. M ETA F EDERATED L EARNING
Suppose we have a universe of federated learning clients U .
The goal of task t is to aggregate models in a federated learning experiment over a subset of clients U t ⊆
U .
At each step n , the action a i,n represents the weight assigned to the supervised learning model of client i in the averaging procedure.
Let v i,n denote the model of the client (i.e. the tensor of model parameters).
We model the state of the client as some function of its H most recent models x i,n = f ( v i,n − H +1 , . .
. , v i,n ) .
Assume that the aggregator has access to a small evaluation dataset that it can use to approximately assess the quality of models.
We deﬁne the reward at each step to be the accuracy of the aggregate model,
R t ( x n , a n ) =
L (cid:0)(cid:80)
i ∈U t
a i,n v i,n (cid:1) , where L ( v ) is a function that provides the accuracy of a model v on the evaluation dataset.
Therefore, by maximizing the total return over all time periods, we seek to maximize both the accuracy at the ﬁnal time step as well as the time to convergence.
We optimize the policy using Proximal Policy Optimization (PPO).
We use the MNIST digit recognition problem.
Each client observes 600 samples from the train dataset and trains a classiﬁer composed of one 5 x 5 convolutional layer (with 32 channels and ReLu activation) and a softmax output layer.
We use the same permutation invariant policy network architecture as before with 10 hidden units in the RNN.
We randomly select |U t | = 10 clients for each task.
We learn using an evaluation dataset comprised of 1000 random samples from the test dataset and test using all 10000 samples in the test dataset.
We ﬁx the number of federated learning iterations to 50 .
We explore the beneﬁt of MTL in identifying useful clients in scenarios with skewed data distribution.
We partition the dataset such that 8 of the clients in each task observe random digits between 0 toand the remaining 2 clients observe random digits between 6 to 9 .
Therefore, for each task, 20% of the clients possess 40% of the unique labels.
The state of each client are the accuracies of its H most recent models on the evaluation dataset.
Figure 6 shows the potential beneﬁts of multi-task learning when simulators are inaccurate.
In particular, we obtain two aggregation policies, one trained using single-task learning (STL), and another trained using multi-task learning (MTL), both trained using the same number of steps, and we observe their behavior during testing.
The plots show that multi-task learning is able to learn non-uniform averaging policies that improve the convergence and performance of federated learning runs.
More importantly, it can perform better than single-task learning even with the same number of samples.
This may be attributed to the wider variety of client conﬁgurations (and consequently experiences) in the multi-task approach.