B ILINEAR V ALUE N ETWORKS
A BSTRACT
Universal value functions are a core component of off-policy multi-goal reinforcement learning.
The de-facto paradigm is to approximate Q ( s, a, g ) using monolithic neural networks which lack inductive biases to produce complex interactions between the state s and the goal g .
In this work, we propose a bilinear decomposition that represents the Q-value via a low-rank approximation in the form of a dot product between two vector ﬁelds.
The ﬁrst vector ﬁeld, f ( s, a ) , captures the environment’s local dynamics at the state s ; whereas the second component, φ ( s, g ) , captures the global relationship between the current state and the goal.
We show that our bilinear decomposition scheme improves sample efﬁciency over the original monolithic value approximators, and transfer better to unseen goals. We demonstrate signiﬁcant learning speed-up over a variety of tasks on a simulated robot arm, and the challenging task of dexterous manipulation with a Shadow hand.
I NTRODUCTION
Learning goal conditioned policies (Kaelbling, 1993; Jordan & Rumelhart, 1992; Wolpert et al., 1995;
Schaul et al., 2015; Agrawal et al., 2016; Nair et al., 2017; Pathak et al., 2018; Andrychowicz et al., 2017; Paster et al., 2021) is a central problem in decision making. Reinforcement learning (RL) is a popular approach for learning goal-conditioned policies. Learning a single policy for many goals using RL is challenging because the data requirement is substantially higher than single task learning because the agent needs to collect data for multiple goals.
To reduce the amount of data needed for learning, it is common to leverage off-policy methods to learn a goal-conditioned Qvalue function Q ( s, a, g ) (or, Universal Value Function Approximator (UVFA) (Schaul et al., 2015)), where s , a , and g denote states, actions, and goals, respectively. To efﬁciently learn Q ( s, a, g ) for a large set of goals, it is important for the UVFA to generalize to new goals, as suggested in Schaul et al. The number of combinations in the inputs (i.e., s, a and g ) grows signiﬁcantly than single-task learning problem since single-task problem does not require accounting for variations of goals.
The increased number of input patterns impedes the value function learning progress. The speed of value function learning inﬂuences the progress of policy learning, hence impacting the data collection process that is crucial for sample efﬁciency.
Under-performing data cannot effectively improve the agent’s performance. As a result, the ability to generalize value predictions to unseen combinations state-actions and goals is critical for multigoal RL. A generalizable Q-value function can make approximately accurate value predictions to new goals seen in the middle of training via the knowledge learned early during training.
A common approach to improve generalizability is using hindsight experience replay (HER) (Andrychowicz et al., 2017) to train the UVFA with additional data produced by relabeling, such that UVFA is exposed to more diverse combinations of goals.
An alternative strategy to achieve generalizability is disentangling the problem so that parts of the knowledge can be reused on new instances of the problem. For example, Barreto et al. (2016) decouple the representations of environment dynamics and rewards using successor features (Dayan, 1993), transferring the value [Figure]
function to new reward functions without re-training the dynamics representations.
In the context of multi-goal RL, the goal-conditioned value function can be disentangled as the interaction between two concerns “where to go” and “what happens next” .
“Where to go” models how should the state be changed to get closer to the goal, while “what happens next” captures where the agent can move at the current state.
Thus, one plausible way to factorize the value function is separating the local knowledge of the dynamics, and the global topology of the domain.
The knowledge of “what happens next” is independent of “where to go” and thus can be re-used across new goals.
We illustrate this intuition in Figure 1c.
As an analogy to “what I can do” , the black arrow denotes the succeeding state after taking an action at state s , which is goal-agnostic.
Independent of action, the grey arrow represents “where to go” that can be viewed as a direction to the goal g from the current state s .
Adhering to this intuition, we thus treat value predictions as the interactions between two vector ﬁelds f ( s, a ) and φ ( s, g ) .
The value predictions can be interpreted as the degree of alignment between two vector ﬁeld f and φ .
The maximum value implicitly emerges when “what I can do” ( f ( s, a ) ) corresponding to taking an action a aligns “where to go” ( φ ( s, g ) ).
Our principal contribution is bilinear value network (BVN), a parameterization scheme, shown in Figure 1b, that increases the sample efﬁciency of the value function by decomposing the goal-conditioned value function into a bilinear map from two distinct vector ﬁelds into a scalar.
The ﬁrst vector ﬁeld, a neural network φ ( s, g ) captures the long-range structure between the current state and a distant goal, whereas the second vector ﬁeld, network f ( s, a ) is local , which represents taking immediate action at the current state.
We build upon state-of-the-art off-policy learning algorithms such as DDPG (Lillicrap et al., 2016) and TD3 (Fujimoto et al., 2018) where the policy is supervised entirely by the value function on offpolicy interaction data. We show empirically that bilinear value decomposition enables the value function to learn signiﬁcantly faster than the commonly used monolithic architecture shown in Figure 1a (Section 5.1).
Further, BVN also exhibits superior speed when adapting to new goals (Section 5.2). Results are presented on two standard benchmark task suites in simulation: (i) dexterous manipulation with a twenty degrees-of-freedom (DOF) shadow hand and (ii) object pick-place tasks using the Fetch manipulator.
P RELIMINARIES
We consider the multi-goal reinforcement learning problem (Kaelbling, 1993; Plappert et al., 2018), formulated as a goal-conditioned Markov Decision Process (MDP).
We assume inﬁnite horizon with the discount factor γ , and use S , A , G for the state, action, and goal spaces.
At the beginning of each episode, an initial state s 0 and a goal g is sampled according to the distribution ρ S and ρ G .
The goal g is ﬁxed throughout the episode.
At every time step t , the agent takes an action according to a t ∼ π ( s
The objective of multi-goal RL is to produce a policy π ∗ that maximizes the expected return J
[Equation]
where the reward r τ,g is often structured as the follows:
[Equation]
Note that our proposed parameterization scheme, BVN, does not rely on this reward structure in general. All of the baselines use a discount factor γ = 0 .
2. O FF - POLICY RL WITH A CTOR - CRITIC M ETHODS
The multi-goal continuous control tasks considered in this paper are typically solved via actor-critic algorithms such as deep deterministic policy gradient (DDPG) (Silver et al., 2014), twin-delayed
DDPG (TD3) (Fujimoto et al., 2018) and soft actor-critic (SAC) (Haarnoja et al., 2018), to name a few.
In DDPG, the critic Q is learned by minimizing the temporal-difference (TD) loss:
[Equation]
where D is the the replay buffer (Mnih et al., 2015) containing state transition tuples ( s t , a t , r t , s t +1 , g ) .
The actor π is trained to maximize the score from the critic, using the deterministic policy gradient:
[Equation]
In multi-goal RL settings, we refer to the goal-conditioned critic as the universal value function approximator (UVFA) (Schaul et al., 2015) because Q ( s, a, g ) approximates the expected utility for all goals ∈ G .
Because the actor relies on the critic for supervision, the ability for the critic to quickly learn and adapt to unseen goals is a key contributing factor to the overall sample efﬁciency of the learning procedure.
A critic that provides better value estimates using less sample and optimization gradient steps, will lead to faster policy improvement via the gradient in Equation 4.
We refer the reader to the original papers for the algorithmic details on SAC and TD3 because we found empirically that DDPG works better than TD3 and SAC on these robot manipulation tasks (see Section B).
B ILINEAR V ALUE N ETWORKS
Bilinear value network (BVN) factorizes the value function approximator into a bilinear form from two vector representations into a scalar:
[Equation]
where f : S × A (cid:55)→ R d
and φ : S × G (cid:55)→ R d .
The ﬁrst component f ( s, a ) is a local ﬁeld that only concerns the immediate action at the current state s .
The second component is more global in the sense that φ ( s, g ) captures long-horizon relationship between the current state and the goal.
[Equation]
[Figure] [Equation] [Figure]
[Equation]
A close bilinear decomposition was explored in Schaul et al.
(2015), though local-global disentanglement was not the motivation of Schaul et al. However, such a bilinear decomposition was not widely used afterwards, despite being potentially beneﬁcial for generalization.
The majority of multi-goal RL algorithms employ a monolithic network shown in Figure 1a to model a value function.
We argue that the critical missing piece for bilinear decomposition is a proper inputs factorization. To motivate the design of our factorization, we present a vector analysis perspective on previous value decomposition Schaul et al. and our method in Figure 2, where each point on the 2D plane is a state s .
In Figure 2a, we illustrate the vector ﬁeld of low-rank bilinear UVFA (Schaul et al., 2015), where f : S × A (cid:55)→ R d and φ : G (cid:55)→ R d . It can be seen that φ ( g ) is a constant across states since φ ( g ) does not depend on the state s .
This makes learning more challenging due to restricted expressivity.
The expressivity of low-rank bilinear UVFA is restricted since the joint relation between state s and goal g only occurs in the dot product at the last layer. Consequently, the burden of modeling different Q-values for different states falls completely on f . In contrast, in Figure 2b, the ﬁeld φ ( s, g ) produced by BVN is also a function of the state s .
This makes φ in BVN strictly more expressive than low-rank bilinear UVFA that do not depend on s . Notably, bilinear value networks is a drop-in replacement for the monolithic universal value function approximator.
Our implementation is not speciﬁc to DDPG, SAC or TD3, but can be applied to any algorithms that computes a goal-conditioned value function. Bilinear value network can be trained by any value function learning objective such as TD objective introduced in Section 2.1.
3. A T OY E XAMPLE : V ISUALIZING f ( s, a ) A ND φ ( s, g )
To examine whether the learned bilinear value decomposition reﬂect our intuition mentioned in Section 1, we trained a deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016) with hindsight experience replay (HER) (Andrychowicz et al., 2017) agent using BVN in a 2D maze where s ∈ R 2 , a ∈ R 2 , g ∈ R 2 , and the reward is the negative euclidean distance between states and goals. The output dimensions of f and φ are both 16 . We collected the output vectors of f ( s, a ∗ ) and φ ( s, g ) with a grid of states ( x, y ) , a ﬁxed goal g , and a ∗ =
π ( s, g ) .
For visualization, we use principal component analysis (PCA) to project φ ( s, g ) to a 2-dimensional vector space, and plot them along with f ( s, a ∗ ) in Figure 3 (right). We plot f ( s, a ∗ ) as φ ( s, g ) with ∠ ( f ( s, a ∗ ) , φ ( s, g )) rotation. In Figure 3 (right), we found that the lengths of the vectors reﬂect the distances to the goal at a state, which validates our hypothesis that φ captures the global topology to the goal (i.e., proximity to the goal). Moreover, we further compare the the angle ∠ f,φ between that of the optimal action (from the policy π ) f
( s, a ∗ ) and a sub-optimal, random action f ( s, a rand ) . We show that on this simple domain, the f ( s, a ∗ ) better aligns with φ ( s, g ) , validating our intuition that the policy ﬁnds the best-aligned f ( s, a ) Note that there is an offset of 90 degrees in the relative angle between f and φ , that exists. This is because in this domain, the maximum value the agent can get is not 1 , but zero.
As a result, f for the optimal action will not be perfectly aligned with φ .
E XPERIMENTAL S ETUP
Tasks We evaluate our method in all the multi-goal RL benchmarks in Plappert et al.
Fetch robot and Shadow hand manipulation tasks, as shown in Figure 9.
For Fetch robot domain, the robot is required to move the object to a target position.
In Shadow hand domain, the agent controls the joints of the hand’s ﬁngers and manipulates the objects in hand to a target pose.
For all tasks, we set discount factor γ = 0 .
based on Plappert et al.
The details, including the reward functions, of these tasks can be found in the Section A.2.
Baselines We compare with two baselines, UVFA (Monlithic) (Andrychowicz et al., 2017; Schaul et al., 2015) and UVFA (Bilinear) proposed in Schaul et al.
UVFA (Monlithic) is the most common multi-goal value function architecture that parametrizes the Q-function by a monolithic neural network.
UVFA (Bilinear) is a bi-linear value function that decomposes the Q-function as Q ( s, a, g ) =
f ( s, a ) (cid:62) φ ( g ) , where f and φ denote two neural networks respectively.
Our method uses a different bilinear decomposition and we refer to it as BVN (Ours) .
Implementation The baselines and our method are trained by deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016) with hindsight experience replay (HER) (Andrychowicz et al., 2017).
We use DDPG with HER as the base algorithm for our method and the baselines because it outperforms other recent Q-learning algorithms (Haarnoja et al., 2018; Fujimoto et al., 2018) on the task of learning a goal-conditioned Q-function. Comparison of DDPG against SAC (Haarnoja et al., 2018) and TD3 (Fujimoto et al., 2018) is provided in Section B. For all baselines, the Q-function is implemented as a three layer neural network with 256 neurons per layer.
In our method, f and φ are both implemented as a 3-layer neural network with 176 neurons to match the total number of parameters of the baselines’ Q-function.
For BVN (Ours) and UVFA (Bilinear) , the output dimensions f , φ ( s, g ) , and φ ( g ) are set as 16 .
The rest of hyperparameters for training are left in Section A.3.
Evaluation metric We ran each experiment with 5 different random seeds and report the mean (solid or dashed line) and 95%-conﬁdence interval (shaded region) using bootstrapping method (DiCiccio & Efron, 1996).
Each training plot reports the average success rate over 15 testing rollouts as a function of training epochs, where the goal is sampled in the beginning of each testing rollout.
Each epoch consists of a ﬁxed amount of data (see Section A.3 for details).  R ESULTS Our experiments answer whether BVN improves sample efﬁciency and generalizes better to unseen goals. In Section 5.3, we also provide ablation studies that investigate the importance of several design decisions.
5. S AMPLE E FFICIENCY AND A SYMPTOTIC P ERFORMANCE
Figure 4 compares the sample efﬁciency of our method, BVN (Ours) , against baselines on the Fetch tasks.
The results show that our method has higher data efﬁciency in 4 out of 8 tasks and matches the baselines in other tasks.
Our method also attains higher asymptotic performance in Bin-place .
Figure 5 shows that in Shadow hand , our method improves the sample efﬁciency in 6 out of 8 tasks and is on par with the baselines for the other two tasks.
These results suggest that BVN consistently improves sample efﬁciency of learning goal-conditioned policies.
5. G ENERALIZATION TO U NSEEN G OALS
Since learning values of multiple goals is one of the challenges in multi-goal RL, we hypothesized that the improvements in data efﬁciency result from the fast adaptation of Q-function to new stategoal pairs encountered during the training time. Fast adaptation implies that the Q-function learned in the early stage of training requires fewer data to learn the values of new state-goal pairs appearing in the later stage. The resulting better critic improves the policy’s performance faster, thus improving data-efﬁciency.
To examine this hypothesis, we split the goal distributions at pre-training and ﬁnetuning time.
We consider two scenarios in the Fetch robot domain: (1) left-to-right and (2) nearto-far.
In near-to-far scenario, we specify a threshold on the goals’ distance to the agent’s initial [Figure] [Figure] state, and sample goals with Euclidean distance under and above the threshold for pre-training and ﬁne-tuning respectively (see Figure 6b).
For left-to-right scenario, we split the goal space by an axis on the table (see Figure 6a) into the left and right half-spaces.
At pre-training time, we disable HER in order to prevent the Q-function from receiving goals in ﬁne-tuning goals by relabeling.
We pretrain the baselines and our method until they have similar success rates (see Section A.3 for details).
During ﬁne-tuning, we initialize the Q-function and the policy with the best pre-trained models over 3 random seeds of each task at the ﬁne-tuning time, and train the Q-function and the policy with
HER.
We compared the ﬁne-tuning performance of the baselines and BVN in Figure 6c. As f network in BVN is goal-agnostic, we also compared the performance of BVN with freezing f at the ﬁne-tuning time. It can be seen that both BVN (ﬁnetune f ) and BVN (freeze f ) outperform UVFA (Monlithic) .
Also, it is observed that BVN (freeze f ) causes BVN performance to plateau, indicating both f and φ are pivotal for approximating the Q-values.
5. A BLATION STUDIES
How does latent dimension affect our method? We investigate the effect of the dimensionality of the latent space on the agent performance. Figure 7a shows the learning curve for BVN with 3 , 4 , 8 and 16 dimensional latent spaces.
The performance consistently improves as latent dimensions increase, and even a BVN with just a 3-dimensional latent space still outperforms the monolithic [Figure] [Figure] UVFA baseline.
Hence bilinear value decomposition produces consistent gain with respect to the monolithic UVFA baseline regardless of the latent dimensions
[Figure]
and beneﬁts performance-wise from a larger latent space [Figure] f [Figure]
· ~ g ~ f ~ gw i f i + v i g i [Figure] Dot product “ · ” vs (cid:96) 2 Metric (cid:107) · (cid:107)
We compare two alternative ways to decompose the value function using the same variable grouping as the BVN.
In the ﬁrst variant, we replace the dot-product with an (cid:96) 2 metric (cid:107)·(cid:107) 2 , and parameterize the Q function via Q ( s, a, g ) = −(cid:107)
f
( s, a ) − (cid:126)φ ( s, g ) (cid:107) 2 .
The negative sign is added because multi-goal RL tasks typically assume the reward is within the range [ − 1 , 0] .
In the second variant we simply combine the two latent vectors linearly. The Q-function is formulated as a linear combination Q ( s, a, g ) = w i f i ( s, a ) + v i φ i ( s, g )
We set the latent dimension of the (cid:96) 2 -norm variant to 3 as it is the best one found through hyperparameter search (see Section A.4).
The results shown in Figure 7b shows that both the bilinear and the metric decomposition attain good performance. The (cid:96) 2 metric decomposition is marginally slower [Table] during the initial learning phase.
The linear combination works poorly, likely due to the restricted expressivity. Does other inputs arguments grouping strategy work as well? Lastly, we investigate if there are other viable ways to decompose the Q-function based on grouping the inputs s, a, g . An alternative hypothesis is that any grouping of the inputs arguments (i
[Table] e., s, a, g ) could improve the sample efﬁciency.
To test if this is the case
[Table]
we compare our bilinear value decomposition scheme against two alternatives: f ( s, a ) (cid:62) φ ( a, g ) and f ( s, a, g ) T φ ( g ) .
The results in Figure 8 show that neither matches the performance of the bilinear value network, which suggests correct grouping does play a crucial role.
It is important to process s and g together and duplicate the state input s on both branches.
[Figure]
D ISCUSSION
A large number of prior works have explored ways to improve value function approximation by decomposing the function approximator into linear superposition of basis functions.
Classic works (Sutton & Barto, 2018; Littman et al., 2001; Singh et al., 2003) approximate the value function in a set of linear basis.
Horde (Sutton et al., 2011) uses a collection of value functions, each learned for a different task, as a representation of the state.
Successor features (Dayan, 1993; Kulkarni et al., 2016; Borsa et al., 2019) factorize the value function as the dot product between state-action feature expectation and a vector of reward coefﬁcients to the features.
Laplacian reinforcement learning (Mahadevan & Maggioni, 2006; 2007) cast learning a set of basis value functions as a geometric problem, where one generalizes Fourier basis functions into higher dimension and non-euclidean manifolds via the diffusion operator ∇ 2 .
In the original universal value function approximator (UVFA, see Schaul et al. 2015), the authors included a linearized UVFA Q ( s, a, g )
=
f ( s, a ) (cid:62) w g where w g is a set of mixing coefﬁcients that depends on the goal.
As we will demonstrate in this work, this na¨ıve linear decomposition reduces the approximator’s expressivity and lowers the agent’s performance.
Therefore the more thoughtful approach is needed.
To summarize prior works and their choice of parameterization, we compile key publications in Table 1, ordered by the year of publication.
A more comprehensive list is available in Table 2 in the appendix.
Bilinear value networks take its name from the bilinear interaction between the two vector representations that resembles a bilinear form.
Such exploitation of linearity goes back as early as 1989, with Ullman & Basri exploiting the linear basis of the special orthogonal group SO (2) for pattern recognition, a scheme further extended to bilinearity by Freeman & Tenenbaum 1997; Tenenbaum & Freeman 2000.
In research that naturally span multiple modalities such as visual question answering (VQA), the bilinear form is used as a means to “align” the visual representations with those of language (Fukui et al., 2016; Yu et al., 2017; Hudson & Manning, 2018). When the input is just images, techniques such as bilinear-pooling induce a similar two-stream architecture (Kong & Fowlkes, 2017; Lin et al., 2018). Self-attention layers used in generative models (Zhang et al., 2019) and transformers Vaswani et al. (2017) are also bilinear forms, and in the latter case involve non-trivial processing through multiple layers.
Moreover, multiplicative interactions are known to improve generalizability of neural networks and increase the expressivity of the network (Jayakumar et al., 2020).
When combined with sinusoidal activation functions, even a shallow, multiplicative network can express an exponential number of basis functions (Fathony et al., 2020).
C ONCLUSION
Our empirical evaluation shows signiﬁcant sample efﬁciency gain when bilinear value networks is used with multi-goal RL tasks.
Our ablation study on adaptation to unseen goals demonstrates that such gains likely result from faster adaptation of the value function.
Exciting future work is to leverage the bilinear decomposition to facilitate transferring across different robots’ morphology and goal spaces.
One could keep the state and goal spaces ﬁxed and train a single φ ( s, g ) that is shared between the different morphologies. The difference between morphologies could be accommodated by different f (e.g., different manipulators for the same task).
Also, one can pre-train a shared f for the same robot and train φ for different tasks (i.e., goal spaces).
In addition, we further discuss the additional implications of our experimental results in Section C of appendix.
A CKNOWLEDGMENTS
This work is supported in part by the Army Research Ofﬁce under Grant W911NF-21-1-0328.
National Science Foundation under Cooperative Agreement PHY-2019786 (The NSF AI Institute for
Artiﬁcial Intelligence and Fundamental Interactions, https://iaiﬁ.org/), and the MIT-IBM Watson AI
Lab and Research Collaboration Agreement No.W1771646. The authors also acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing high performance computing resources. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the
Army Research Ofﬁce or the U.S. Government.
The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein
R EPRODUCIBILITY S TATEMENT
We provide detailed instructions for reproducing the results in this paper in the Appendix.
Please refer to Section Section A.4.
A A PPENDIX A. D ETAILS O N P REVIOUS L INEAR D ECOMPOSITION S CHEMES
We present the comparison of parameterizations in Table
[Table] A. I MPLEMENTATION DETAILS OF TASKS
All experiments in Section 5.1 happen on the standard object and dexterous manipulation tasks from the gym robotics suite (Plappert et al., 2018).
We brieﬂy summarize the speciﬁcations of the tasks in the list below and illustrate these tasks in
Figure 9:
[Table]
In addition to (Plappert et al., 2018), we also use the tasks from Fetch extension 2 .
For left/right environments used in Section 5.2, the goal ( x, y, z ) with x < 0 are left goals, and x ≥
right goals.
In near/far environments, we set the distance thresholds as 0 .
2 for Slide, 0 .
2 for PickAndPlace, 0 .for Reach, and 0 .
15 for Push.
The goals with distance below the threshold are considered as near goals, otherwise far goals.
A. T RAINING HYPERPARAMETERS
Fetch We adapted the hyperparameters in (Plappert et al., 2018) for training in a single desktop. In Plappert et al.
, the agent is trained by multiple workers collecting data in parallel, where each worker collects rollouts using the same policy with different random seeds.
The performances of the agent trained by our hyperparameters match that in (Plappert et al., 2018).
• num workers : 2 for Reach, 8 for Push, 16 for Pick & Place, and 20 for Slide.
• Batch size : [Table] • Warm up rollouts : We collected 100 initial rollouts for preﬁlling the replay buffer.
• Training frequency : We train the agent per 2 environment steps.
Each iteration of training uses the num workers batches for training.
The rest of hyperparameters remain the same as that used in (Plappert et al., 2018).
Shadow hand We use the hyperparameters used in (Plappert et al., 2018) and reuse the open source code 3 .
Adaptation experiments (Section 5.2)
We present the success rates after pretraining in Table 4.
Note that in the pretraining stage of each method and task, we trained the agents with 5 different random seeds and took the best-performing models as the initialization at the ﬁne-tuning time.
A. I MPLEMENTATION DETAILS
Q-function architecture The monolithic network baselines in this paper represents the Q-function as a multi-layer perceptron (MLP) network that takes in s , a , and g concatenated together as a single vector. The bilinear UVFA (Schaul et al., 2015) uses an MLP f that takes the concatenated s and a as inputs, and an MLP φ using g as inputs. Our biliniear value network on the other hand, concatenate [ s, a ] together as the input to f , and [ s, g ] as the inputs of φ . Policy architecture We use an MLP as the policy network for all the methods.
For our method and its variants presented in Section 5.3 , we use the following nomenclature to denote them:
Dot product This is our proposed method.
We set the outputs’ dimension of f and φ as 16 .
2. 2-norm We parametrize Q-function as Q ( s, a, g ) = (cid:107) f
( s, a ) − φ ( s, g ) (cid:107) 2 .
We use set the dimension of the outputs from f and φ as 3 .
Figure 10 shows that d = 3 works the best for 2-norm.
concat We concatenate f
( s, a ) and φ ( s, g ) , feed the concatenated vectors to an MLP layer, and take the outputs of this layer as the Q-value Q ( s, a, g ) .
The dimensions of the outputs of f and φ are 16 . [Figure] Push [Figure]
Pick & Place
UVFA d= d= d= d=
A. T RANSFERRING A CROSS T ASKS
In multi-goal RL tasks, we observe that most of tasks are identical in terms of high-level goals, but they have different environment dynamics.
For instance, the objective of ”Push” and ”Slide” are all to move the object to the destination.
They only differ in the friction coefﬁcient of the table where the agent cannot slide objects in ”Push” because of high friction.
Because we decouple the state-action and goals into two different functions, f ( s, a ) and φ ( s, g ) , one potential beneﬁt on multi-goal RL is reusing φ for transferring to tasks with the same (or similar) task objectives (i.e., reward speciﬁcation).
Reusing φ ( s, g ) exploits the learned reward-speciﬁc state-goal relations.
Then we simply need to learn a new f to recover the reward-agnostic relations in the new task.
To examine this hypothesis, we select three tasks with the same state and action spaces 4 , and measure the performance of transferring amongst these tasks.
In transferring an agent from a source task to a target task, we initialize UVFA (Monlithic) and BVN (no reset) agent’s policy and Q-function by the neural networks’ weights learned in the source task and ﬁne-tune them in the target task.
For BVN (reset f) , we reinitialize f and ﬁne-tune φ when training them in the target task.
Figureshows the success rates of transferring from a source task to a target task.
For example, ”Push to Slide” indicates transferring from ”Push” to ”Slide”.
It can be seen that BVN (reset f ) attains higher success rates at convergence than the baselines and BVN (no reset) in 4 out 6 tasks (i.e., ”Push to
Slide”, ”Pick & Place to Push”, ”Push to Pick & Place”, and ”Pick & Place to Slide”).
The limited efﬁciency gain could result from that φ is dependent on states s and thus φ cannot be effectively reused in new tasks.
Even though the target task’s reward function is the same as that in the source task, the relation of states and goals can be affected by the dynamics.
For instance, suppose we have a state where the object is near the goal.
In ”Pick & Place”, because the friction coefﬁcients of the table are high, the φ ( s, g ) should direct the agent to lift the object.
On the other hand, in ”Slide”, the φ ( s, g ) is expected to guide the agent to push the object to slide. [Figure] [Figure]
B C OMPARISON A GAINST SOTA
A LGORITHMS
Bilinear value networks achieves State-Of-The-Art (SOTA) performance in common multi-goal reinforcement learning tasks (Plappert et al., 2018). In ﬁgure 12, we compare bilinear value networks against two strong baselines: soft actor critic (SAC, see Haarnoja et al. 2018) and twin-delayed deep deterministic policy gradient (TD3, see Fujimoto et al. 2018).
We reuse the hyperparameters reported in Fujimoto et al. 2018) and implement TD3 in our codebase.
As for SAC, we use the implementation in this codebase 5 .
Figure 12 shows the performance of each baseline algorithm and our method ( DDPG + BVN ).
We show that our DDPG + BVN outperforms the best baseline, and ﬁnd that DDPG works the best among all the MLP variants.
Our ﬁndings align with (Pitis et al., 2020) and the documentation in its released code 6 .
Pitis et al.
(2020) also found that DDPG works better than TD3 and SAC in multi-goal RL domains.
C A DDITIONAL DISCUSSION
Why does other parameter grouping schemes fail?
In Section 5.3, we showed that f ( s, a, g ) T φ ( g ) inputs grouping fails to improve sample efﬁciency.
One explanation to this result is that f ( s, a, g ) cannot prevent negative interference from new goals encountered during training since goals g couple with states-actions s, a .
As a result, the merit of bi-linear value function decomposition vanishes and even causes extra burden on learning because the number of model parameters is larger than UVFA (Monlithic) .
This preliminary ﬁnding shows that to beneﬁt from decomposition, the guideline is to only expose necessary inputs to each component.
Redundant inputs in either component could break the decomposition.
How is 2-norm compared with dot product?
Section 5.3 demonstrated that using 2-norm to reduce f ( s, a ) and φ ( s, g ) into Q-value matches the performance of our BVN with even lower dimensions of Z f and Z φ .
However, 2-norm relies on an assumption that the reward function must be non-positive or non-negative since norm is deﬁned as a non-negative value.
The (cid:107) f
( s, a ) − φ ( s, g ) (cid:107) can only ﬁt to either non-positive or non-negative Q-value by manipulating the sign of norm.
On the other hand, dot product is more expressive and free from the reward function assumption.
Can our method facilitate policy transfer?
Since our method decomposes the value function into a local (i.e., goal-agnostic) and a global (i.e., goal-speciﬁc) component, our BVN might facilitate policy transfer by resetting either one of components and ﬁne-tuning another one.
As multi-goal tasks considered in (Plappert et al., 2018) have the same objective in nature but different dynamics, one potential advantage of our method is to reuse φ ( s, g ) in the new task and re-train f ( s, a ) .
The results in Section A.5 showed that the agent transferred by BVN converges to the the performance of training from scratch in the target task slightly faster than the baselines.
One reason for the limited policy transferring performance is that φ ( s, g ) conditions on states s .
Consequently, environment dynamics shifts could affect the representations of φ .
We, nevertheless, showed that states variables are crucial in φ .
One exciting avenue is to disentangle the representations for keeping the rich expressivity as well as desired modularity
[Table]