B I - LINEAR V ALUE N ETWORKS FOR M ULTI - GOAL R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Universal value functions are used to score the long-term utility of actions to achieve a goal from the current state.
In contrast to prior methods that learn a monolithic function to approximate the value, we propose a bi-linear decomposed value function.
The ﬁrst component, akin to a global plan models how the state should be changed to reach the goal.
The second component, akin to a local controller selects the optimal action to actualize the desired change in state.
We simultaneously learn both the components.
Such decomposition enables both the global and local components to make efﬁcient use of interaction data and independently generalize.
The consequence is superior overall generalization and performance of our system on a wide range of challenging goal conditioned tasks in comparison to the current state-of-the-art.
I NTRODUCTION
[Figure]
Constructing agents capable of accomplishing a diverse set of goals is a long standing challenge in reinforcement learning (Kaelbling, 1993).
A major issue is the high sample complexity resulting from interference caused by learning a single policy or value function for a multitude of goals.
This issue can be partially mitigated by making efﬁcient reuse of past interaction data, such as employing off-policy learning algorithms, along with hindsight experience re-labeling (HER, see Andrychowicz et al. 2017).
A complementary approach improves data efﬁciency by encoding inductive biases in the neural network, such as the graph structure (Li et al., 2019).
Such a structure enables sharing information across goals and thereby mitigates interference and results in better generalization.
In multi-goal reinforcement learning, a single, goal-conditioned universal value function (UVFA Schaul et al. 2015) approximates the long-term return for a variety of goals.
One possibility is to model the UVFA as a monolithic neural network, Q θ ( s, a, g ) , where state s , action a and goal g (or the embedding of which) are simply concatenated together as input.
In such form, there is no explicit inductive bias for the network to discover shared structures between goals that could otherwise greatly improve the performance when the agent is tasked with a new goal (Sutton et al., 2011).
This ability to transfer to new goals when learning online, as the ability for the policy to perform given limited data has a signiﬁcant compounding effect through the active collection of interactive data. Separating the local knowledge of the dynamics, and the global topology of the domain is a natural way to disentangle the problem.
Inspired by this dichotomy, we decompose goal-conditioned value approximation into two disentangled concerns “where to go” and “what happens next” .
“Where to go” models how should the state be changed to get closer to the goal, while “what I can do” captures where the agent can move at the current state.
The knowledge of “what I can do” is independent of “where to go” and thus can be re-used across new goals.
We illustrate this intuition in ﬁgure 1a.
As an analogy to “what I can do” , the red arrow denotes the succeeding state after taking an action at state s , which is goal-agnostic.
Independent of action, the blue arrow represents “where to go” that can be viewed as a direction to the goal g from the current state s .
Adhering to this intuition, we thus treat value predictions as the interactions between two vectors.
Consider “what I can do” and “where to go” as two vector ﬁelds (cid:126)f ( s, a ) and (cid:126)ϕ ( s, g ) in ﬁgure 1b.
It is natural to deﬁne the value predictions as the level of alignment between two vector ﬁeld (cid:126)f and (cid:126)ϕ .
The maximum value implicitly emerges when “what I can do” ( (cid:126)f ( s, a ) ) corresponding to taking an action a aligns “where to go” ( (cid:126)ϕ ( s, g ) ).
In fact, our intuition resembles bilinear mapping in vector analysis, that parametrizes a scalar function by two vector spaces.
Our principal contribution is bi-linear value network (BVN), a parameterization scheme that decomposes the goal-conditioned value function into a bilinear map between two distinct vector representations.
The ﬁrst component φ ( s, g ) captures the long-range topology between the current state and a distant goal, whereas the second component f ( s, a ) is akin to a local model that represents taking an immediate action at the current state.
To understand how this parameterization affects learning dynamics, we evaluate the agent on goals that never appeared during training.
Surprisingly, we found that the bi-linear decomposition allows the critic to generalize to unseen goals (see section 4.3), which in turn improves the overall sample complexity and policy performance.
We evaluate our approach on the challenging task of dexterous manipulation with a simulated shadow hand and other object manipulation tasks.
Our method, BVN, achieves state-of-the-art sample efﬁciency and asymptotic performance on the majority of tasks.
Apart from comparing with prior works, we also produce detailed analysis on the inﬂuence of the dimensionality of the two latent spaces, and how different constructions of the binary mapping affects performance.
P RELIMINARIES 2. M ULTI - GOAL R EINFORCEMENT L EARNING
We consider a special case of the Markov decision process (MDP), the multi-goal reinforcement learning (RL) problem (Kaelbling, 1993).
At the beginning of an episode, an initial state s 0 and a behavior goal g b is sampled according to the distribution ρ S and ρ G .
The behavior goal g b is ﬁxed throughout the episode during roll-out by the policy π .
For each time step t , the agent takes action a t =
π ( s t , g b ) , receives reward r t,g b
= R ( s t , a t , g b ) , and moves to the next state s t
+1 .
We consider an inﬁnite horizon MDP problem with the discount factor γ .
The goal of multi-goal RL is to produce a policy π ∗ that maximizes the expected return J [Equation] where
[Equation]
where the discount factor γ is set as 0 .
98 in common multi-goal RL benchmarks (Plappert et al., 2018), and the reward r τ,g b is typically deﬁned as
[Equation]
:= (cid:26) 0 , goal reached − 1 , otherwise (2)
Note that our method, BVN, is not dependent on this reward structure.
2. O FF - POLICY RL WITH A CTOR - CRITIC M ETHODS
As we consider multi-goal continuous control tasks, we follow Andrychowicz et al. (2017) and Plappert et al. and use deterministic policy gradient algorithm (DDPG) (Silver et al., 2014) to learn a π ∗ .
DDPG is an actor-critic method.
Note that a more advanced actor-critic method (Fujimoto et al., 2018) can be used, but we empirically found that DDPG is better than that in the tasks we consider.
In actor-critic methods the value function, referred to as the critic, is responsible for generating the learning signal of π .
The critic is trained by minimizing the loss function
[Equation]
[Equation] where Z denotes a replay buffer (Mnih et al., 2015) that stores multiple state transitions tuples ( s t , a t , r t , s t +1 , g b ) .
The actor π is trained to maximize the critic outputs by the gradient
[Equation]
In multi-goal RL settings, the value function Q is often referred as universal value function approximator (Schaul et al., 2015) since Q encompasses the values of all goals in the MDP.
As we are particularly interested in sample efﬁciency of policy learning, we highlight the importance of the critic for the sample efﬁciency of training the actor.
A critic that provides better estimate of values for previously unseen states (i.e., better generalization of the critic) will lead to faster policy improvement via optimization in Equation 4.
B I - LINEAR V ALUE D ECOMPOSITION FOR M ULTI - GOAL RL
The key insight behind the bilinear value network architecture, as illustrated in ﬁgure 2, is that we can often decompose value estimate into the alignment between the concern of “what I can do” and “where I want to go.”
When these two concerns are implicitly aligned, we consider the action to be good for reaching that goal.
Whereas if these two are misaligned, the action is considered bad.
These two concerns interact with each other when we try to ﬁnd the solution to what I can do, but they are also disentangled because they take in different inputs.
“What I can do,” for example, only concerns the current state of affair and the possible set of actions.
When a new goal appears the agent would not have to reconsider the relationship between its current state and the available actions, therefore lowers the burden to learn.
It does however need to update what it knows about getting to that goal from where it is, which is inescapable.
[Figure]
To bring this insight into reality, we design a drop-in replacement of the standard universal value function approximator , as illustrated in ﬁgure 3, which we refer to as the bilinear value network .
For context we present the standard architecture of a universal value function approximator (UVFA) on the left (see ﬁgure 3a).
The UVFA is a single, monolithic neural network which is entangled in the way it processes the inputs.
It ﬁrst concatenates the state, action and the goal together into a single vector, and does not impose any inductive bias into the way these inputs interact.
With bilinear value networks (see ﬁgure 3b), we partition the three input vectors into two overlapping groups.
In the ﬁrst group that we consider to be local , the neural network (cid:126)f ( s, a ) is only concerned with the current state and the immediate actions the agent can take.
This is the component that is relatively easier to learn, because it does not have to integrate over long horizons.
The second group is more global .
The neural network (cid:126)φ ( s, g ) captures the long-horizon relationship between the current state and the goal which can be quite far apart.
To bring the information that are processed separately by these two components, we borrow the concept of bilinear map from vector analysis, and deﬁne the Q function as a dot product between the two distinct vector representations
[Equation]
A consequence of this treatment is that the value becomes the result of an alignment between the two vector ﬁelds:
[Equation]
s, a ) (cid:62) (cid:126) φ ( s, g ) . In ﬁgure 2, we show that passing the current state s into the global stream φ ( s, g ) is important, because to ﬁnd the solution to the optimal action, the global ﬁeld needs to vary depending on the current state.
Figure 2a shows a goal stream that does not take into the current state – the vector ﬁeld becomes a constant ∀ s ∈ S , making it impossible to accommodate the complex behavior needed to succeed from the entire state space.
An important issue to keep in mind is that Equation 6 is unidentiﬁable in the statistical sense because we can insert an arbitrary vector λ into the vector space and produce the same value estimate, as long as λ contains no zeros.
In statistics, identiﬁability is the condition under which we can recover the true parameter of the model given inﬁnite samples.
[Equation]
What this means is that under end-to-end training, the particular semantics of the two vector ﬁelds becomes particularly irrelevant.
It is the grouping of the input variables and the bilinear structure that is critical.
It is also important to note that the bilinear value networks is implemented as a drop-in replacement of the universal value function approximator, as opposed to any additional algorithmic step.
The training procedure we adopt in this paper follows those of deep deterministic policy gradient and hindsight experience replay, but the bilinear value networks is equally applicable to other off-policy learning algorithms under goal-conditioned scenario that uses either a value function, or a critic. [Equation] [Figure] [Equation]
[Figure]
E XPERIMENT
Our experiments are aimed to answer the following two questions: (1) Does our value function parametrization improve sample efﬁciency?
(2) Does the sample efﬁciency gain result from better value function generalizability?
Moreover, we investigated the inﬂuence of each of our design choices in section 4.4.
4. S ETUP
Tasks We evaluate our method in all the multi-goal RL benchmarks in Plappert et al. Fetch robot and Shadow hand manipulation tasks, as shown in ﬁgure 4.
For Fetch robot domain, the robot has to interact with the object and move the object to the target positions.
In Shadow hand domain, the agent controls the joints of the hand’s ﬁngers and manipulates the objects in hand to a target pose.
For all tasks, we set discount factor γ = 0 .
based on Plappert et al.
The details,including the reward functions, of these tasks can be found in the the section A.1.
Baselines and our method We compare with two baselines: UVFA (Monlithic) (Andrychowicz et al., 2017; Schaul et al., 2015) and UVFA (Bilinear) (Schaul et al., 2015). UVFA (Monlithic) and
UVFA (Bilinear) are all proposed in the original UVFA paper Schaul et al.
UVFA (Monlithic) is the most common multi-goal value function architecture that parametrizes the Q-function by a monolithic neural network.
UVFA (Bilinear) is a bi-linear value function parameterization similar to our method.
UVFA (Bilinear) decomposes the Q-function into Q ( s, a, g ) =
f ( s, a ) T ϕ ( g ) , where f and ϕ denote two neural networks respectively.
In the following sections, we will refer our method to as BVN (Ours) .
Implementation The baselines and our method are trained by deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016) with hindsight experience replay (HER) (Andrychowicz et al., 2017).
For all the baselines, we implement the Q-function as a 3-layer neural network with 256 neurons per layer.
In our method, f and ϕ are both implemented as a 3-layer neural network with 176 neurons to match the total number of parameters of the baselines’ Q-function.
Following the implementation in (Andrychowicz et al., 2017), we train each agent in parallel with multiple workers (varies across tasks).
For BVN (Ours) and UVFA (Bilinear) , the output dimensions f , ϕ ( s, g ) , and ϕ ( g ) are set as 16 .
The rest of hyperparameters for training are left in section A.2.
Evaluation metric We ran each experiment with 5 different random seeds and reported the mean (solid or dashed line) and 95% -conﬁdence interval on the learning curves.
The learning curves indicate the average success rate over 15 testing rollouts in a varying number of training epoch.
One epoch corresponds to the number of workers times two rollouts.
4. S AMPLE E FFICIENCY AND A SYMPTOTIC P ERFORMANCE
To examine if our bi-linear value function parametrization improves sample efﬁciency, we evaluate each method in Fetch and Shadow hand domains.
For Fetch domain, ﬁgure 5 shows that our method exhibits higher sample efﬁciency in 4 out of 8 tasks and matches the baselines in the rest of tasks.
Remarkably, our method even attains higher asymptotic performance in Bin-place .
Figure 5 shows that in Shadow hand , our method substantially improves the sample efﬁciency in 6 out of 8 tasks and is on par with the baselines for the rest of tasks.
These results suggest that our bi-linear value network consistently advances the sample efﬁciency.
One explanation is that better Q-function’s generalizability across goals helps the agent quickly learn a useful critic that will improve the policy’s performance.
As a result, the exploration will be more focused on high-return regime.
The collected high-return experience in turn improves the policy.
4. G ENERALIZATION TO U NSEEN G OALS
To verify our hypothesis that generalizability improves sample efﬁciency, we examine bi-linear value network ’s performance on unseen goals.
A Q-function that generalizes well should result in a policy that obtains high rewards on unseen goals since the policy’s training signals is solely from the Qfunction in actor-critic methods.
To quantify the generalizability of the Q-function we split the goal distributions at training and testing time.
We consider two scenarios in the Fetch robot domain: (1) near-to-far and (2) left-to-right.
In near-to-far case, we specify a threshold on the goals’ distance to the agent’s initial state, and sample goals with Euclidean distance under and above the threshold for training and testing respectively.
For left-to-right scenario, we split the goal space by an axis on the table (see ﬁgure 7) into the left and right half-spaces.
The goals from the left space are sampled at training time, whereas others at test time.
We present the rest of implementation details on these scenarios in section A.3.
Figure 8 shows both the training and testing success rate of the policies trained by each method.
It can be seen that our method outperforms the baselines in terms of testing performance, showing that our method generalizes [Figure] better to unseen goals.
Moreover, it is shown that our method’s gap between training and testing success rate are smaller than that of the baselines in Push and Slide in both near-to-far and left-to- right scenarios.
Notably, ﬁgure 8 shows that UVFA (Bilinear) underperforms the UVFA (Monlithic) in most tasks, suggesting that taking states as inputs at both branches (i.e., f and ϕ ) is crucial to the success of the bi-linear decomposed value function.
4. A BLATION STUDIES
We investigate the inﬂuence of each component in our method to further our understanding of BVN.
First, as the output dimension of f and ϕ could affect the expressivity of our bi-linear Q-function, we are interested in what extent this dimension inﬂuences the performance.
Section C shows that larger dimension helps the performance and even the 3-dimensional space shows the sample efﬁciency gain.
Also, we investigate if dot product is a good way to combine f and ϕ by comparing various ways to combine them.
Section D shows that dot-product and 2-norm perform similarly, while concating f and ϕ largely undermines the performance.
Lastly, we investigate if there is other viable ways to decompose the Q-function based on grouping the inputs s, a, g .
One hypothesis is that any inputs arguments (i.e., s, a, g ) grouping can enhance sample efﬁciency as long as the parametrization shares some variables in f and ϕ .
Thus we compare with two alternative inputs grouping scheme: f
( s, a ) T ϕ ( a, g ) and f ( s, a, g ) T ϕ ( g ) .
Figure 9 demonstrates that both groupings cannot match the performance of our proposed f ( s, a ) T ϕ ( s, g ) , suggesting that not only the decomposition is important but a carefully designed grouping scheme is also crucial and states s might be necessary to be shared in two branches..
4. V ISUALIZATION OF THE LEARNED f ( s, a ) AND ϕ ( s, g )
To examine whether the learned bilinear value decomposition reﬂects our intuition mentioned in section 1, we trained an DDPG+HER agent with our bilinear value network in a 2D maze where s ∈ R 2 , a ∈ R 2 , g ∈ R 2 , and the reward is the negative euclidean distance between states and goals.
The dimensions of Z f and Z ϕ are both 16 . We collected the output vectors of f ( s, a ∗ ) and ϕ ( s, g ) with a grid of states ( ( x, y ) ), a ﬁxed goal g , and a ∗ =
π ( s, g ) .
For visualization, we use principal component analysis (PCA) to project ϕ ( s, g ) to a 2-dimensional vector space, and plot them along with f ( s, a ∗ ) in ﬁgure 10(right). We plot f ( s, a ∗ ) as ϕ ( s, g ) with ∠ ( f ( s, a ∗ ) , ϕ ( s, g )) rotation.
In ﬁgure 10(right), we found that the lengths of the vectors reﬂect the distances to the goal at a state, which validates our hypothesis that ϕ captures the global topology to the goal (i.e., proximity to the higher goal). Moreover, we further compare the the angle ∠ f,ϕ between that of the optimal action (from the policy π ) (cid:126)f ( s, a ∗ ) and a sub-optimal, random action (cid:126)f ( s, a rand ) . We show that on this simple domain, the (cid:126)f ( s, a ∗ ) better aligns with (cid:126)φ ( s, g ) validating our intuition that the policy ﬁnds the bestaligned (cid:126)f ( s, a ) Note that there is an offset of 90 degrees in the relative angle between (cid:126)f and (cid:126)ϕ , that exists.
This is because in this domain, the maximum value the agent can get is not but zero. As a result, (cid:126) f for the optimal action will not be perfectly aligned with (cid:126) φ .  R ELATED W ORKS
Our work is closely related to the literature on value function approximation in the sense that both seek for a lower dimensional space to represent the value function in a more compact way.
Most early works in RL approximate the value function by the linear combination of basis functions (Sutton & Barto, 2018)
(e.g., polynomial basis, radial basis, fourier basis (Konidaris et al., 2011)).
Instead of arbitrarily picking basis functions, proto-value function framework (Mahadevan, 2007; Mahadevan & Maggioni, 2006) compose the value function in the linear combination of task-independent basis vectors constructed by the spectral of the state transition function of the environment.
The works on successor representations (Dayan, 1993; Kulkarni et al., 2016; Borsa et al., 2018; Barreto et al., 2017; 2018) seek for a value function representation that help the value function transfer to new tasks.
This line of works bi-linearly compose the value function by a rewardagnostic and reward-speciﬁc parts, so the reward-agnostic part can be reused in a new task where the environment’s transition function is the same.
The idea of successor representations was introduced to multi-goal RL in (Schaul et al., 2015) for decomposing the goal-conditioned value function.
Its difference to prior works on successor representations is that the reward-speciﬁc component is a function conditioned on goals.
Despite the appealing performance shown in toy gridworld, the follow-up works in multi-goal RL rarely use the decomposed architecture proposed in (Schaul et al., 2015).
Andrychowicz et al.
Pong* et al.
all parameterize the Q-function as a monolithic neural network.
Section 4.2 empirically shows that the bilinear decomposition proposed in (Schaul et al., 2015) cannot extend its advantages to more complex tasks.
As a result, positioned in the literature of multi-goal value function approximation, our principal contribution lies in solving the bottleneck that hinders the advantages of value function decomposition in multi-goal RL. In addition to devising the function approximator parameterization, recent works attempted to improve RL by learning the Q-function in a hierarchical manner. Edwards et al. (2020) estimate the
Q-value conditioned on the current state and the next state, and deriving the action to the optimal next state by an inverse dynamics model. Mirowski et al. (2018) trains a monolithic value network using the goals prepossessed by different networks for each environment. Nachum et al.
Levy et al.
(2019) learn a hierarchy of policies and multi-goal value functions, and take the highlevel policies’ outputs as the low-level policy/value function’s goals inputs.
Orthogonal to this line of works, we focus on improving the value function’s parameterization, which can be a drop-in replacement in the above related works.
In the abstract sense, the spirit of our method is to model the goal-agnostic (state-action) and goalspeciﬁc (state-goal) information separately in different components.
A proper decomposition is known to make function approximation easier due to modularity.
Rusu et al.
(2016) demonstrates the modularity in a network facilitates continual learning in new tasks, as the previously learned subnetworks can be reused in new tasks Wang et al.
(2018) learns a modular policy where each network controls a joint of a robot.
This modular policy is shown to be able to better transfer to new robot morphology.
Galashov et al.
(2019) train a specialized policy and a reusable policy restricted from the reward information for encouraging the reusability in other tasks.
Different from these works, our work imposes modularity in the parametrization of value functions and focuses on improving genearlizability and sample efﬁciency of value functions.
D ISCUSSION
Our experimental results demonstrated that our BVN signiﬁcantly improves sample efﬁciency in multi-goal RL tasks.
Also, we show that this sample efﬁciency gain could result from better generalizability of the value function.
Beyond the aspect on the sample efﬁciency and generalizability in
RL, an exciting future work is leveraging the decomposition to facilitate transferring across different robots’ morphology and goal spaces.
One can pre-train a shared ϕ for the same state (e.g., camera) and goal space, and train f for robots with different action spaces (e.g., different manipulator for the same task).
Also, one can pre-train a shared f for the same robot, and train ϕ for different tasks (i.e., goal spaces).
In addition, we further discuss the additional implications of our experimental results in section E of appendix.
R EPRODUCIBILITY S TATEMENT
Please refer to section A.3.
A A PPENDIX A. I MPLEMENTATION DETAILS OF TASKS
The implementation details of all the tasks used in section 4.2 are referred to (Plappert et al., 2018). We brieﬂy summarize the task speciﬁcations below • FetchReach: s ∈ R 10 , a ∈ R 3 , g ∈ R•
FetchPush: s ∈ R 25 , a ∈ R 3 , g ∈ R• FetchPickAndPlace: s ∈ R 25 , a ∈ R 4 , g ∈ R• FetchSlide: s ∈ R 25 , a ∈ R 3 , g ∈ R• DrawerClose: s ∈ R 25 , a ∈ R 3 , g ∈
R•
DrawerOpen: s ∈ R 25 , a ∈ R 3 , g ∈ R•
ShadowHandBlockRotateZ: s ∈ R 61 , a ∈ R 20 , g ∈ R•
ShadowHandBlockRotateParallel: s ∈ R 61 , a ∈ R 20 , g ∈ R• ShadowHandBlockRotateXYZ: s ∈ R 61 , a ∈ R 20 , g ∈ R• ShadowHandBlockRotate: s ∈ R 61 , a ∈ R 20 , g ∈ R•
ShadowHandEggRotate: s ∈ R 61 , a ∈ R 20 , g ∈ R• ShadowHandEgg: s ∈ R 61 , , a ∈ R 20 , g ∈ R• ShadowHandPen: s ∈ R 61 , , a ∈ R 20 , g ∈ R• ShadowHandPenRotate: s ∈ R 61 , , a ∈ R 20 , g ∈ R
In addition to (Plappert et al., 2018), we also use the tasks from Fetch extension 1 .
For left/right environments used in section 4.3, the goal ( x, y, z ) with x < 0 are left goals, and x ≥
right goals.
In near/far environments, we set the distance thresholds as 0 .
2 for Slide, 0 .
2 for PickAndPlace, 0 .for Reach, and 0 .
15 for Push.
The goals with distance below the threshold are considered as near goals, otherwise far goals.
A. T RAINING HYPERPARAMETERS
Fetch We adapted the hyperparameters in (Plappert et al., 2018) for training in a single desktop.
The performances of the agent trained by our hyperparameters match that in (Plappert et al., 2018).
• num workers : 2 for Reach, 8 for Push, 16 for Pick & Place, and 20 for Slide.
• Batch size :• Warm up rollouts : We collected 100 initial rollouts for preﬁlling the replay buffer.
• Training frequency : We train the agent per 2 environment steps.
Each iteration of training uses the num workers batches for training.
The rest of hyperparameters remain the same as that used in (Plappert et al., 2018).
Shadow hand We use the hyperparameters used in (Plappert et al., 2018) and reuse the open source code 2 .
A. I MPLEMENTATION DETAILS
For the baselines, we concatenate s, a, g into a vector and feed it into the network.
For our method, we concatenate ( s, a ) as inputs of f , and ( s, g ) as inputs of ϕ .
Dot product This is our proposed method.
We set the outputs’ dimension of f and ϕ as 16 .
2-norm We parametrize Q-function as Q ( s, a, g )
=
(cid:107) f
( s, a ) − ϕ
( s, g ) (cid:107) 2 .
We use set the dimension of the outputs from f and ϕ as 3 .
Figure 11 shows that d = 3 works the best for 2-norm.
concat We concatenate f
( s, a ) and ϕ ( s, g ) , feed the concatenated vectors to a multi-layer perceptron (MLP) layer, and take the outputs of this layer as the Q-value Q ( s, a, g ) .
The dimensions of the outputs of f and ϕ are 16 .
A. T RANSFERRING A CROSS T ASKS
In multi-goal RL tasks, we observe that most of tasks are identical in terms of high-level goals, but they have different environment dynamics.
For instance, the objective of ”Push” and ”Slide” are all to move the object to the destination.
They only differ in the friction coefﬁcient of the table where the agent cannot slide objects in ”Push” because of high friction.
Because we decouple the state-action and goals into two different functions, f ( s, a ) and ϕ ( s, g ) , one potential beneﬁt on multi-goal RL is reusing ϕ for transferring to tasks with the same (or similar) task objectives (i.e., reward speciﬁcation).
Reusing ϕ
( s, g ) exploits the learned reward-speciﬁc state-goal relations.
Then we simply need to learn a new f to recover the reward-agnostic relations in the new task.
To examine this hypothesis, we select three tasks with the same state and action spaces 3 , and measure the performance of transferring amongst these tasks.
In transferring an agent from a source task to a target task, we initialize UVFA (Monlithic) agent’s policy and Q-function by the neural networks’ weights learned in the source task and ﬁne-tune them in the target task.
Different from UVFA (Monlithic) , we reinitialize f and ﬁne-tune ϕ in UVFA (Bilinear) and BVN (Ours) when training them in the target task.
Figure 12 shows the success rates of transferring from a source task to a target task.
For example, ”Push to Slide” indicates transferring from ”Push” to ”Slide”.
It can be seen that BVN (ours, w reset f ) attains higher success rates at convergence than the baselines and BVN (ours, w/o reset) in 4 out 6 tasks (i.e., ”Push to Slide”, ”Pick & Place to Push”, ”Push to Pick & Place”, and ”Pick & Place to Slide”), despite slight improvements.
The limited efﬁciency gain could result from that ϕ is dependent on states s and thus ϕ cannot be effectively reused.
Even though the target task’s reward function is the same as that in the source task, the relation of states and goals can be affected by the dynamics.
For instance, suppose we have a state where the object is near the goal.
In ”Pick & Place”, because the friction coefﬁcients of the table are high, the ϕ ( s, g ) should direct the agent to lift the object.
On the other hand, in ”Slide”, the ϕ ( s, g ) is expected to guide the agent to push the object to slide. B C OMPARISON WITH OTHER ALGORITHMS
To show that our bilinear value network achieves the state-of-the-art performance in common multigoal RL tasks (Plappert et al., 2018), we further compare our method with twin-delayed deep deterministic policy gradient (TD3) (Fujimoto et al., 2018) and soft actor critic (SAC) ((Haarnoja et al., 2018)).
For TD3 speciﬁc hyperparameters, we reuse the hyperparameters reported in (Fujimoto et al., 2018) and implement TD3 in our codebase.
As for SAC, we use the implementation in this codebase 4 .
Figure 13 shows the performance of each baseline algorithm and our method ( DDPG + BVN (Ours) ).
We show that our DDPG + BVN (Ours) outperforms the best baseline, and ﬁnd that DDPG works the best among all the UVFA (Monolithic) variants.
Our ﬁndings align with (Pitis et al., 2020) and the documentation in its released code 5 .
Pitis et al.
(2020) also found that DDPG works better than TD3 and SAC in multi-goal RL domains.
C H OW DOES LATENT DIMENSION AFFECT OUR METHOD ?
We made several variants of our method with different latent dimension of the outputs of f and ϕ and present their learning curves in ﬁgure 14.
It can be seen that the performance consistently improves when we increase the latent dimension.
Though the performance drops when the dimension decreases, our method with dimension 3 still outperforms the UVFA (Monlithic) baseline.
These results suggest that our method beneﬁts from increasing latent dimension but is insensitive to the choice of the dimension.
D I S COMBINING f AND ϕ BY DOT PRODUCT CRUCIAL ?
We compared two variants that reduces f and ϕ by 2-norm and concatenation.
In 2-norm variant, we parametrize the Q-function as Q ( s, a, g ) =
−|| f ( s, a ) − ϕ
( s, g )
|| 2 .
The negative sign is added because multi-goal RL tasks typically assume the reward is within the range [ − 1 , 0] .
For concatenation variant, the Q-function is formulated as Q ( s, a, g ) =
MLP ([ f ( s, a ) , ϕ ( s, g )] , 1) , where we feed the concatenated f ( s, a ) and ϕ ( s, g ) to a multi-layer perceptron (MLP) network that outputs a one-dimensional value.
We set the latent dimension of the 2-norm variant as the best one found in our hyperparameter search (see section A.3).
The results shown in ﬁgure 15 demonstrate that
Bi-linear-full (dot) signiﬁcantly outperforms concat variant, and Bi-linear-full (2-norm) approximately matches the performance of Bi-linear-full (dot) .
This result suggests separating the f and ϕ embedding is the key to the performance gain, and concatenating them at the output diminishes the gain.
E A DDITIONAL DISCUSSION
Why does other parameter grouping schemes fail?
In section 4.4, we showed that f ( s, a, g ) T ϕ ( g ) inputs grouping fails to improve sample efﬁciency.
One explanation to this result is that f ( s, a, g ) cannot prevent negative interference from new goals encountered during training since goals g couple with states-actions s, a .
As a result, the merit of bi-linear value function decomposition vanishes and even causes extra burden on learning because the number of model parameters is larger than UVFA (Monlithic) .
This preliminary ﬁnding shows that to beneﬁt from decomposition, the guideline is to only expose necessary inputs to each component.
Redundant inputs in either component could break the decomposition.
How is 2-norm compared with dot product?
Section 4.4 demonstrated that using 2-norm to reduce f ( s, a ) and ϕ ( s, g ) into Q-value matches the performance of our BVN with even lower dimensions of Z f and Z ϕ .
However, 2-norm relies on an assumption that the reward function must be non-positive or non-negative since norm is deﬁned as a non-negative value.
The (cid:107) f
( s, a ) − ϕ
( s, g ) (cid:107) can only ﬁt to either non-positive or non-negative Q-value by manipulating the sign of norm.
On the other hand, dot product is more expressive and free from the reward function assumption.
Can our method facilitate policy transfer?
Since our method decomposes the value function into a local (i.e., goal-agnostic) and a global (i.e., goal-speciﬁc) component, our BVN might facilitate policy transfer by resetting either one of components and ﬁne-tuning another one.
As multi-goal tasks considered in (Plappert et al., 2018) have the same objective in nature but different dynamics, one potential advantage of our method is to reuse ϕ ( s, g ) in the new task and re-train f ( s, a ) .
The results in section A.4 showed that the agent transferred by BVN converges to the the performance of training from scratch in the target task slightly faster than the baselines.
One reason for the limited policy transferring performance is that ϕ ( s, g ) conditions on states s .
Consequently, environment dynamics shifts could affect the representations of ϕ .
We, nevertheless, showed that states variables are crucial in ϕ .
One exciting avenue is to disentangle the representations for keeping the rich expressivity as well as desired modularity for transferring across tasks.