B ILINEAR V ALUE N ETWORKS
A BSTRACT
The dominant framework for off-policy multi-goal reinforcement learning involves estimating goal conditioned Q-value function.
When learning to achieve multiple goals, data efﬁciency is intimately connected with generalization of the
Q-function to new goals.
The de-facto paradigm is to approximate Q ( s, a, g ) using monolithic neural networks.
To improve generalization of the Q-function, we propose a bilinear decomposition that represents the Q-value via a low-rank approximation in the form of a dot product between two vector ﬁelds.
The ﬁrst vector ﬁeld, f ( s, a ) , captures the environment’s local dynamics at the state s ; whereas the second component, φ ( s, g ) , captures the global relationship between the current state and the goal.
We show that our bilinear decomposition scheme substantially improves data efﬁciency, and has superior transfer to out-of-distribution goals compared to prior methods.
Empirical evidence is provided on the simulated
Fetch robot task-suite, and dexterous manipulation with a Shadow hand.
I NTRODUCTION
Learning a Q-value function (Watkins & Dayan, 1992) is among the core problems in reinforcement learning (RL) (Sutton & Barto, 2018).
It is often desired to learn a goal-conditioned Q-value function (Kaelbling, 1993) that estimates rewards for multiple goals.
In multi-goal RL, the Q-function takes the form of Q ( s, a, g ) which indicates the utility of taking action a at state s for approaching the goal g .
To enable the agent to work across a wide range of goals, it is usual to train Q ( s, a, g ) for a large set of goals.
Such training requires substantially more data than learning a Q-value function for a single goal (Schaul et al., 2015).
The data efﬁciency of learning Q ( s, a, g ) has a close relationship with the generalization of the
Q-function to new goals.
To understand why, consider the training process where goals are sampled from a pre-deﬁned training distribution.
The target of Q -learning is to reduce the error on the
Q -function for all goals in the training set.
At an intermediate point in the training process, the agent would have encountered a subset of training goals.
Suppose on these goals the error in approximating the Q -function is low.
If the learned Q -function generalizes to new goals sampled in subsequent training, less data would be required to correct the small errors in Q-function for these goals.
Conversely, poor generalization would necessitate collecting a large amount of data.
Currently popular methods (Schaul et al., 2015;
Andrychowicz et al., 2017) represent the Q ( s, a, g ) with a monolithic neural network, shown in Figure 1a, whose ability to generalize to new goals is solely derived from the neural network architecture.
Because such models jointly process the tuple ( s, a, g ) , the Q-function fails to generalize on new combinations of previously seen individual components.
E.g, given training tuples, ( s A , a A , g A ) and ( s A , a B , g B ) , the monolithic network might fail to generalize to a new data ( s A , a A , g B ) despite that s A , a A , and g B have been seen at least once during training.
We proposed bilinear value networks (BVN) to improve such generalization by introducing an inductive bias in the neural network architecture that disentangles the local dynamics from the question of how to reach the goal .
Concretely, we propose to represent the goal-conditioned Q-value
[Figure]
function as interaction between two concerns, φ ( s, g ) ( “where to go” ) and f ( s, a ) ( “what happens next” ), where each concern can be represented as a neural network.
The architecture of BVN is illustrated in Figure 1b.
φ ( s, g ) models how to change the state to get closer to the goal, while f ( s, a ) captures where the agent can move from the current state.
The Q-function is computed as the alignment between these concerns: φ ( s, g ) T f
( s, a ) .
As the φ -network only requires state-goal ( s, g ) and f -network simply needs state-action ( s, a ) , the learned Q-value function can generalize to the new data ( s A , a A , g B ) since both ( s A , a A ) and ( s A , g B ) are the seen data for f and φ , respectively.
Such decomposition is possible because the knowledge of local dynamics ( f ) is independent of where to go ( φ ) and thus f ( s, a ) be re-used across goals.
Another intuition about the proposed BVN decomposition is: BVN would predict a high Q-value when the vector representation of the desired future state, φ ( s, g ) , aligns with the vector representation of the future state, f ( s, a ) , after executing action a .
We illustrate this intuition with help of a toy example shown in Figure 1c.
Here the agent’s state s and the goal g are 2D positions, and the task is to get close to g shown as a pink cross.
As an analogy to “what happens next” , the black arrow originating at state s (shown as the blue circle) denotes the succeeding state after taking an action at state s , which is goal-agnostic.
Independent of action, the grey arrows can be viewed as the direction to the goal from the current state s (i.e., φ ( s, g ) or “where to go” ).
While this illustration was in 2D, we ﬁnd that the proposed bilinear value decomposition substantially improves data efﬁciency (Section 5.1) and generalization to new goals (Section 5.2) in complex and standard simulated benchmark tasks: (i) dexterous manipulation with a twenty degrees-of-freedom (DoF) shadow hand and (ii) object pick-place tasks using the Fetch manipulator.
P RELIMINARIES
We consider the multi-goal reinforcement learning problem (Kaelbling, 1993; Plappert et al., 2018), formulated as a goal-conditioned Markov Decision Process (MDP).
We assume inﬁnite horizon with the discount factor γ , and use S , A , G for the state, action, and goal spaces.
At the beginning of each episode, an initial state s 0 and a goal g is sampled according to the distribution ρ S and ρ G .
The goal g is ﬁxed throughout the episode.
At every time step t , the agent takes an action according to a t ∼ π ( s t , g ) , and receives the reward r t,g = R ( s t , a t , g ) while transitioning to the next state s t +1 .
The objective of multi-goal RL is to produce a policy π ∗ that maximizes the expected return J
[Equation]
where the reward r τ,g is often structured as the follows:
[Equation]
Note that our proposed parameterization scheme, BVN, does not rely on this reward structure in general.
All of the baselines use a discount factor γ = 0 .
2. O FF - POLICY RL WITH A CTOR - CRITIC M ETHODS
It is common to use actor-critic algorithms such as deep deterministic policy gradient (DDPG) (Silver et al., 2014), twin-delayed DDPG (TD3) (Fujimoto et al., 2018) and soft actor-critic (SAC) (Haarnoja et al., 2018), to solve multi-goal continuous state-action space tasks considered this paper.
In DDPG, the critic Q is learned by minimizing the temporal-difference (TD) loss:
[Equation]
where D is the the replay buffer (Mnih et al., 2015) containing state transition tuples ( s t , a t , r t , s t +1 , g ) .
The actor π is trained to maximize the score from the critic, using the deterministic policy gradient:
[Equation]
In multi-goal RL settings, we refer to the goal-conditioned critic as the universal value function approximator (UVFA)
(Schaul et al., 2015) because Q ( s, a, g ) approximates the expected utility for all goals ∈ G .
Because the actor relies on the critic for supervision, the ability for the critic to quickly learn and adapt to unseen goals is a key contributing factor to the overall sample efﬁciency of the learning procedure.
A critic that provides better value estimates using less sample and optimization gradient steps, will lead to faster policy improvement via the gradient in Equation 4.
For data efﬁciency it is common to use off-policy actor-critic methods with hindsight experience replay (HER) (Andrychowicz et al., 2017).
It is known that TD3 and SAC outperform DDPG on singletask continuous control problems.
However, in the multi-goal setting when used along with HER, we found DDPG to outperform both TD3 and SAC on the benchmark tasks we considered (see Section B).
We therefore use DDPG for majority of our experiments and report SAC/TD3 results in the appendix.
B ILINEAR V ALUE N ETWORKS
[Figure]
Bilinear value network (BVN) factorizes the value function approximator into a bilinear form from two vector representations into a scalar:
[Equation]
where f : S × A (cid:55)→ R d
and φ : S × G (cid:55)→
R d .
The ﬁrst component f
( s, a ) is a local ﬁeld that only concerns the immediate action at the current state s .
The second component is more global in the sense that φ ( s, g ) captures long-horizon relationship between the current state and the goal.
Closely related to our work is the lowrank bilinear decomposition, Q ( s, a, g ) =
f ( s, a ) (cid:62) φ ( g ) , proposed in Schaul et al.
Though Schaul et al.
(2015) showed this decomposition attains better data efﬁciency in simple environments (e.g., gridworlds), this decomposition is not expressive enough to model value functions in complex environments, which we will show
[Figure]
in Section 5.2.
Consequently, the majority of multi-goal RL algorithms use a monolithic network shown in Figure 1a to represent the value function.
To motivate the design of our factorization, we present a vector analysis perspective on previous value decomposition Schaul et al.
and our method in Figure 2, where each point on the 2D plane is a state s .
In Figure 2a, we illustrate the vector ﬁeld of low-rank bilinear UVFA (Schaul et al., 2015), where f : S × A (cid:55)→ R d and φ : G (cid:55)→ R d .
It can be seen that φ ( g ) is a constant across states since φ ( g ) does not depend on the state s in Figure 2a.
This makes learning more challenging due to restricted expressivity.
The expressivity of low-rank bilinear UVFA is restricted since the such a decomposition can only model linear relationships between states and goals.
Consequently, the burden of modeling different Q-values for different states falls completely on f .
In contrast, in
Figure 2b, the ﬁeld φ ( s, g ) produced by BVN is also a function of the state s .
This makes φ in BVN strictly more expressive than low-rank bilinear UVFA that do not depend on s .
Bilinear value networks are a drop-in replacement for the monolithic neural networks for approximating universal value function approximator.
Our implementation is not speciﬁc to a method and be used to learn value function using a variety of methods such as DDPG, SAC, TD3 or even methods that we do not make use of temporal differenceing.
3. A T OY E XAMPLE : V ISUALIZING f ( s, a ) A ND φ ( s, g )
To examine whether the learned bilinear value decomposition reﬂect our intuition mentioned in
Section 1, we trained a deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016) with hindsight experience replay (HER) (Andrychowicz et al., 2017) agent using BVN in a 2D maze shown in Figure 3 (a), where s ∈ R 2 , a ∈ R 2 , g ∈ R 2 , and the reward is the negative euclidean distance between states and goals.
The output dimensions of f and φ are both 16 .
We computed the vectors z ∗ f = f
( s, a ∗ ) and z φ = φ ( s, g ) densely on a grid of states ( x, y ) with respect to the ﬁxed goal g , where a ∗ =
π ( s, g ) is the optimal action predicted by the learned policy.
For visualization, we used principal component analysis (PCA) to project z φ to a 2-dimensional vector and is plotted as black arrows in Figure 3(c).
We cannot use PCA to plot z ∗ f on the 2D plane along with z phi because f ( s, a ∗ ) and φ ( s, g ) live in two different vector spaces.
Instead, we plot f ( s, a ∗ ) as a rotation from z φ denoted as ∠ ( f ( s, a ∗ ) , φ ( s, g )) .
Speciﬁcally, we ﬁrst plot the 2D projection of
Further, Figure 3(c) shows that the length of z phi correlates to the distance between the state ( s ) and the goal ( g ), providing further evidence in support of our hypothesis that φ ( s, g ) captures proximity to the goal.
If f ( s, a ) indeed captures the dynamics, then z ∗ f
(i.e., vector representation of future state after executing the optimal action a ∗ ) should be closely aligned to z phi to produce the maximal Qvalue.
To test if this is the case, we also plot in pink z randf denoting the vector representation of future state after executing a random action a rand .
We ﬁnd that in comparison to f ( s, a rand ) , f ( s, a ∗ ) better aligns with φ ( s, g ) validating our hypothesis.
Note that the optimal alignment corresponds to an angle of 90 ◦ because the maximum Q-value in this environment is 0 and not 1 .
As a result, f for the optimal action will be orthogonal to φ .
E XPERIMENTAL S ETUP
Tasks We evaluate our method in all the multi-goal RL benchmarks in Plappert et al.
Fetch robot and Shadow hand manipulation tasks, as shown in Figure 10.
For Fetch robot domain, the robot is required to move the object to a target position.
In Shadow hand domain, the agent controls the joints of the hand’s ﬁngers and manipulates the objects in hand to a target pose.
For all tasks, we set discount factor γ = 0 .
based on Plappert et al.
The details, including the reward functions, of these tasks can be found in the Section A.2.
Baselines We compare with two baselines, UVFA (Monlithic) (Andrychowicz et al., 2017; Schaul et al., 2015) and UVFA (Bilinear) proposed in Schaul et al.
UVFA (Monlithic) is the most common multi-goal value function architecture that parametrizes the Q-function by a monolithic neural network.
UVFA (Bilinear) is a bi-linear value function that decomposes the Q-function as
Q ( s, a, g ) =
f ( s, a ) (cid:62) φ ( g ) , where f and φ denote two neural networks respectively.
Our method uses a different bilinear decomposition and we refer to it as BVN (Ours) .
Implementation The baselines and our method are trained by deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016) with hindsight experience replay (HER) (Andrychowicz et al., 2017).
We use DDPG with HER as the base algorithm for our method and the baselines because it outperforms other recent Q-learning algorithms (Haarnoja et al., 2018; Fujimoto et al., 2018) on the task of learning a goal-conditioned Q-function.
Comparison of DDPG against SAC (Haarnoja et al., 2018) and TD3 (Fujimoto et al., 2018) is provided in Section B. For all baselines, the Q-function is implemented as a three layer neural network with 256 neurons per layer.
In our method, f and φ are both implemented as a 3-layer neural network with 176 neurons to match the total number of parameters of the baselines’ Q-function.
For BVN (Ours) and UVFA (Bilinear) , the output dimensions f , φ ( s, g ) , and φ ( g ) are set as 16 .
The rest of hyperparameters for training are left in Section A.3.
Evaluation metric We ran each experiment with 5 different random seeds and report the mean (solid or dashed line) and 95%-conﬁdence interval (shaded region) using bootstrapping method (DiCiccio & Efron, 1996).
Each training plot reports the average success rate over 15 testing rollouts as a function of training epochs, where the goal is sampled in the beginning of each testing rollout.
Each epoch consists of a ﬁxed amount of data (see Section A.3 for details).
R ESULTS
Our experiments answer whether BVN improves sample efﬁciency and generalizes better to unseen goals.
In Section 5.3, we provide ablation studies probing the importance of several design decisions.
5. S AMPLE E FFICIENCY AND A SYMPTOTIC P ERFORMANCE
Figure 4 compares the sample efﬁciency of our method, BVN (Ours) , against baselines on the Fetch tasks.
The results show that our method has higher data efﬁciency in four out of eight tasks and matches the baselines in other tasks.
Our method also attains higher asymptotic performance in
Bin-place .
Figure 5 shows that on the Shadow hand task suite, BVN improves the sample efﬁciency in six out of the eight tasks and is on-par with the baselines for the other two tasks.
These results suggest that BVN consistently improves sample efﬁciency of learning goal-conditioned policies.
[Figure] [Figure]
5. G ENERALIZATION TO U NSEEN G OALS
Our central hypothesis is that BVN increases data efﬁciency by generalizing better to new goals.
To test this, we split the training and evaluation goals in two ways in the Fetch robot domain: (1) left-to-right and (2) near-to-far.
In near-to-far scenario, all goals within a pre-specifed euclidean distance are used for training ( near ) and goals sampled outside the threshold distance are used for testing (i.e., far ).
The train-test split of goals is illustrated in Figure 6b by colors green and pink respectively.
For left-to-right scenario, we split the goal space into the left and right half-spaces (see Figure 6a).
During training, we disable HER to prevent the Q-function from receiving goals in the testing split due to relabeling.
We train the baselines and our method until they achieve similar success rates (see Section A.3 for details).
For testing, we ﬁnetune the learned Q-function and the policy using the weights from the best model over 3 random seeds from training time.
We ﬁnetune the Q-function and the policy using HER.
The ﬁne-tuning performance of the baselines and BVN is reported in Figure 6c.
We ﬁnd that BVN (shown in blue) quickly adapts to new goals and outperforms the monolithic UVFA (shown in grey).
This result suggests that BVN generalizes better than UVFA (Monlithic) to new goals.
Recall that in the BVN formulation, f ( s, a ) does not encode any goal speciﬁc information.
We therefore hypothesized that only ﬁnetuning φ ( s, g ) should be sufﬁcient for generalizing to new goals.
We call this version of ﬁnetuning as BVN (freeze f ) .
Results in Figure 6c support our hypothesis
[Figure]
[Figure]
as BVN (freeze f ) (orange) indeed outperforms UVFA (Monlithic) .
However, there is a signiﬁcant performance gap between BVN (ﬁnetune f ) and BVN (freeze f ) .
This result indicates that while
BVN provides an inductive bias to disentangle the local dynamics, f ( s, a ) , from the goal-speciﬁc component, φ ( s, g ) , such disentanglement is not fully learned.
This could be due to several reasons such as learning on limited data or because the disentangelment between f and φ is under-speciﬁed upto a scalar factor (i.e., λf and 1 λ φ yield the same result for a scalar λ ), etc.
Nonetheless, the results convincingly demonstrate that the proposed bilinear decomposition increased data efﬁciency and improves generalization.
The topic of achieving perfect disentanglement is outside the scope of current work and a worthwhile direction for future research.
5. A BLATION STUDIES
How does latent dimension affect our method?
Though the dimensions of the latent space (i.e., output dimension of f and φ ) are set as 16 across all the experiments above, we investigate the effect of the dimensionality of the latent space on the agent performance.
Figure 7 shows the learning curve for BVN with 3 , 4 , 8 and 16 dimensional latent spaces.
The performance consistently improves as latent dimensions increase, and even a BVN with just a 3-dimensional latent space still outperforms the monolithic UVFA baseline.
Hence bilinear value decomposition produces consistent gain with respect to the monolithic UVFA baseline regardless of the latent dimensions, and beneﬁts performance-wise from a larger latent space.
Dot product “ · ” vs (cid:96) 2 Metric (cid:107) · (cid:107)
As it is unclear if the dot-product is the best choice to model the interaction between two vector ﬁelds f and φ , we compare two alternative ways to decompose the value function using the same variable grouping as the BVN.
In the ﬁrst variant, we replace the dot-product with an (cid:96) 2 metric (cid:107) · (cid:107) 2 , and parameterize the Q function via Q ( s, a, g ) = −(cid:107)
f
( s, a ) − (cid:126)φ ( s, g ) (cid:107) 2 .
The negative sign is added because multi-goal RL tasks typically assume the reward is within the range [ − 1 , 0] .
We set the latent dimension to be three as it is the best one found through hyperparameter search (see Section A.4).
In the second variant we linearly combine the two latent vectors: Q ( s, a, g ) =
w i f i ( s, a )
+ v i φ i ( s, g ) ,
where x i y i implies a sum over the i indices implicitly.
This is the Einstein notation, or “einsum”.
The results shown in Figure 8 shows that both the bilinear
[Figure] [Figure]
and the metric decomposition attain good performance.
The (cid:96) 2 metric decomposition is marginally slower during the initial learning phase.
The linear combination works poorly, likely due to the restricted expressivity.
Alternate Bi-Linear Decomposition
Lastly, we investigate if there are other viable ways to decompose the Q-function based on grouping the inputs s, a, g .
An alternative hypothesis is that any grouping of the inputs arguments (i.e., s, a, g ) could improve the sample efﬁciency.
To test if this is the case, we compare our bilinear value decomposition scheme against two alternatives, f ( s, a ) (cid:62) φ ( a, g ) and f ( s, a, g ) (cid:62) φ ( g ) , as well as the scheme f
( s, a ) (cid:62) φ ( g ) shown in Section 5.1.
The results in Figure 9 show that neither matches the performance of the bilinear value network, which suggests correct grouping does play a crucial role.
It is important to process s and g together and duplicate the state input s on both branches.
[Figure]
[Equation]
R ELATED WORKS
A large number of prior works have explored ways to decompose value function to improve sample efﬁciency of value function approximation.
Classic works (Sutton & Barto, 2018; Littman et al., 2001; Singh et al., 2003) approximate the value function in a set of linear basis.
Successor features (Dayan, 1993; Kulkarni et al., 2016; Borsa et al., 2019) factorize the value function as the dot product between expectation of state-action feature occupancy and a vector of reward coefﬁcients to the features.r Laplacian reinforcement learning (Mahadevan & Maggioni, 2006; 2007) represents the value function using a set of Fourier basis.
The above previous works investigate value function decomposition in single task learning.
Instead, bilinear value networks is focused on decomposing a multi-goal value function by disentangling the representations for goals and actions.
Also, bilinear value networks use end-to-end training to learn the decomposition instead of separating the basis learning and the value approximation.
To summarize prior works and their choice of parameterization, we compile key publications in Table 1, ordered by the year of publication, in the appendix.
C ONCLUSION AND D ISCUSSION
Our empirical evaluation shows signiﬁcant sample efﬁciency gain when bilinear value networks is used with multi-goal RL tasks.
Moreover, bilinear value networks does not introduce extra hyperparamters for training, except for the output dimensions of f and φ .
Our ablations demonstrate that the beneﬁts of BVN result from faster adaptation of the value function to new goals.
An exciting direction for future research is to facilitate transfer across different robots’ morphology and goal spaces.
One could keep the state and goal spaces ﬁxed and train a single φ ( s, g ) that is shared between the different morphologies.
The difference between morphologies could be accommodated by different f (e.g., different manipulators for the same task).
Also, one can pre-train a shared f for the same robot and train φ for different tasks (i.e., goal spaces).
In addition, we further discuss the additional implications of our experimental results in Section C of appendix.
A CKNOWLEDGMENTS
We thank members of Improbable AI Lab for the helpful discussion and feedback.
We are grateful to MIT Supercloud and the Lincoln Laboratory Supercomputing Center for providing HPC resources.
The research in this paper was supported by the MIT-IBM Watson AI Lab, in part by the Army Research Ofﬁce and was accomplished under Grant Number W911NF-21-1-0328 and by the National Science Foundation AI Institute for Artiﬁcial Intelligence and Fundamental Interactions ( https://iaiﬁ.org/) under Cooperative Agreement PHY-2019786.
The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the Army Research Ofﬁce or the United States
Air Force or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
R EPRODUCIBILITY S TATEMENT
We provide detailed instructions for reproducing the results in this paper in the Appendix.
Please refer to Section Section A.
A A PPENDIX A. D ETAILS O N P REVIOUS L INEAR D ECOMPOSITION S CHEMES
We present the comparison of parameterizations in Table
[Table]
A. I MPLEMENTATION DETAILS OF TASKS
All experiments in Section 5.1 happen on the standard object and dexterous manipulation tasks from the gym robotics suite (Plappert et al., 2018).
We brieﬂy summarize the speciﬁcations of the tasks in the list below and illustrate these tasks in
Figure 10:
[Table]
In addition to (Plappert et al., 2018), we also use the tasks from Fetch extension 1 .
For left/right environments used in Section 5.2, the goal ( x, y, z ) with x < 0 are left goals, and x ≥
right goals.
In near/far environments, we set the distance thresholds as 0 .
2 for Slide, 0 .
2 for PickAndPlace, 0 .for Reach, and 0 .
15 for Push.
The goals with distance below the threshold are considered as near goals, otherwise far goals.
A. T RAINING HYPERPARAMETERS
Fetch We adapted the hyperparameters in (Plappert et al., 2018) for training in a single desktop.
In Plappert et al.
, the agent is trained by multiple workers collecting data in parallel, where each worker collects rollouts using the same policy with different random seeds.
The performances of the agent trained by our hyperparameters match that in (Plappert et al., 2018).
• num workers : 2 for Reach, 8 for Push, 16 for Pick & Place, and 20 for Slide.
• Batch size :
[Table]
• Warm up rollouts : We collected 100 initial rollouts for preﬁlling the replay buffer.
• Training frequency : We train the agent per 2 environment steps.
Each iteration of training uses the num workers batches for training.
The rest of hyperparameters remain the same as that used in (Plappert et al., 2018).
Shadow hand We use the hyperparameters used in (Plappert et al., 2018) and reuse the open source code 2 .
Adaptation experiments (Section 5.2)
We present the success rates after pretraining in Table 3.
Note that in the pretraining stage of each method and task, we trained the agents with 5 different random seeds and took the best-performing models as the initialization at the ﬁne-tuning time.
A. I MPLEMENTATION DETAILS
Q-function architecture The monolithic network baselines in this paper represents the Q-function as a multi-layer perceptron (MLP) network that takes in s , a , and g concatenated together as a single vector.
The bilinear UVFA (Schaul et al., 2015) uses an MLP f that takes the concatenated s and a as inputs, and an MLP φ using g as inputs.
Our biliniear value network on the other hand, concatenate [ s, a ] together as the input to f , and [ s, g ] as the inputs of φ .
Policy architecture We use an MLP as the policy network for all the methods.
For our method and its variants presented in Section 5.3 , we use the following nomenclature to denote them:
Dot product This is our proposed method.
We set the outputs’ dimension of f and φ as 16 .
2. 2-norm We parametrize Q-function as Q ( s, a, g ) = (cid:107) f
( s, a ) − φ ( s, g ) (cid:107) 2 .
We use set the dimension of the outputs from f and φ as 3 .
Figure 11 shows that d = 3 works the best for 2-norm.
concat We concatenate f
( s, a ) and φ ( s, g ) , feed the concatenated vectors to an MLP layer, and take the outputs of this layer as the Q-value Q ( s, a, g ) .
The dimensions of the outputs of f and φ are 16 .
[Figure]
Push
[Figure]
Pick & Place
UVFA d= d= d= d=
A. T RANSFERRING A CROSS T ASKS
In multi-goal RL tasks, we observe that most of tasks are identical in terms of high-level goals, but they have different environment dynamics.
For instance, the objective of ”Push” and ”Slide” are all to move the object to the destination.
They only differ in the friction coefﬁcient of the table where the agent cannot slide objects in ”Push” because of high friction.
Because we decouple the state-action and goals into two different functions, f ( s, a ) and φ ( s, g ) , one potential beneﬁt on multi-goal RL is reusing φ for transferring to tasks with the same (or similar) task objectives (i.e., reward speciﬁcation).
Reusing φ ( s, g ) exploits the learned reward-speciﬁc state-goal relations.
Then we simply need to learn a new f to recover the reward-agnostic relations in the new task.
To examine this hypothesis, we select three tasks with the same state and action spaces 3 , and measure the performance of transferring amongst these tasks.
In transferring an agent from a source task to a target task, we initialize UVFA (Monlithic) and BVN (no reset) agent’s policy and Q-function by the neural networks’ weights learned in the source task and ﬁne-tune them in the target task.
For
BVN (reset f) , we reinitialize f and ﬁne-tune φ when training them in the target task.
Figureshows the success rates of transferring from a source task to a target task.
For example, ”Push to
Slide” indicates transferring from ”Push” to ”Slide”.
It can be seen that BVN (reset f ) attains higher success rates at convergence than the baselines and BVN (no reset) in 4 out 6 tasks (i.e., ”Push to
Slide”, ”Pick & Place to Push”, ”Push to Pick & Place”, and ”Pick & Place to Slide”).
The limited efﬁciency gain could result from that φ is dependent on states s and thus φ cannot be effectively reused in new tasks.
Even though the target task’s reward function is the same as that in the source task, the relation of states and goals can be affected by the dynamics.
For instance, suppose we have a state where the object is near the goal.
In ”Pick & Place”, because the friction coefﬁcients of the table are high, the φ ( s, g ) should direct the agent to lift the object.
On the other hand, in ”Slide”, the φ ( s, g ) is expected to guide the agent to push the object to slide.
[Figure]
[Figure]
B C OMPARISON A GAINST SOTA
A LGORITHMS
Bilinear value networks achieves State-Of-The-Art (SOTA) performance in common multi-goal reinforcement learning tasks (Plappert et al., 2018).
In ﬁgure 13, we compare bilinear value networks against two strong baselines: soft actor critic (SAC, see Haarnoja et al. 2018) and twin-delayed deep deterministic policy gradient (TD3, see Fujimoto et al. 2018).
We reuse the hyperparameters reported in Fujimoto et al. 2018) and implement TD3 in our codebase.
As for SAC, we use the implementation in this codebase 4 .
Figure 13 shows the performance of each baseline algorithm and our method ( DDPG + BVN ).
We show that our DDPG + BVN outperforms the best baseline, and ﬁnd that DDPG works the best among all the MLP variants.
Our ﬁndings align with (Pitis et al., 2020) and the documentation in its released code 5 .
Pitis et al.
(2020) also found that DDPG works better than TD3 and SAC in multi-goal RL domains.
C A DDITIONAL DISCUSSION
Why does other parameter grouping schemes fail?
In Section 5.3, we showed that f ( s, a, g ) T φ ( g ) inputs grouping fails to improve sample efﬁciency.
One explanation to this result is that f ( s, a, g ) cannot prevent negative interference from new goals encountered during training since goals g couple with states-actions s, a .
As a result, the merit of bi-linear value function decomposition vanishes and even causes extra burden on learning because the number of model parameters is larger than UVFA (Monlithic) .
This preliminary ﬁnding shows that to beneﬁt from decomposition, the guideline is to only expose necessary inputs to each component.
Redundant inputs in either component could break the decomposition.
How is 2-norm compared with dot product?
Section 5.3 demonstrated that using 2-norm to reduce f ( s, a ) and φ ( s, g ) into Q-value matches the performance of our BVN with even lower dimensions of Z f and Z φ .
However, 2-norm relies on an assumption that the reward function must be non-positive or non-negative since norm is deﬁned as a non-negative value.
The (cid:107) f
( s, a ) − φ ( s, g ) (cid:107) can only ﬁt to either non-positive or non-negative Q-value by manipulating the sign of norm.
On the other hand, dot product is more expressive and free from the reward function assumption.
Can our method facilitate policy transfer?
Since our method decomposes the value function into a local (i.e., goal-agnostic) and a global (i.e., goal-speciﬁc) component, our BVN might facilitate policy transfer by resetting either one of components and ﬁne-tuning another one.
As multi-goal tasks considered in (Plappert et al., 2018) have the same objective in nature but different dynamics, one potential advantage of our method is to reuse φ ( s, g ) in the new task and re-train f ( s, a ) .
The results in Section A.5 showed that the agent transferred by BVN converges to the the performance of training from scratch in the target task slightly faster than the baselines.
One reason for the limited policy transferring performance is that φ ( s, g ) conditions on states s .
Consequently, environment dynamics shifts could affect the representations of φ .
We, nevertheless, showed that states variables are crucial in φ .
One exciting avenue is to disentangle the representations for keeping the rich expressivity as well as desired modularity
[Table]
tasks.