L EARNING FROM D EMONSTRATION WITH W EAKLY S UPERVISED D ISENTANGLEMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Robotic manipulation tasks, such as wiping with a soft sponge, require control from multiple rich sensory modalities.
Human-robot interaction, aimed at teaching robots, is difﬁcult in this setting as there is potential for mismatch between human and machine comprehension of the rich data streams.
We treat the task of interpretable learning from demonstration as an optimisation problem over a probabilistic generative model.
To account for the high-dimensionality of the data, a high-capacity neural network is chosen to represent the model.
The latent variables in this model are explicitly aligned with high-level notions and concepts that are manifested in a set of demonstrations.
We show that such alignment is best achieved through the use of labels from the end user, in an appropriately restricted vocabulary, in contrast to the conventional approach of the designer picking a prior over the latent variables.
Our approach is evaluated in the context of a tabletop robot manipulation task performed by a PR2 robot – that of dabbing liquids with a sponge (forcefully pressing a sponge and moving it along a surface).
The robot provides visual information, arm joint positions and arm joint efforts.
We have made videos of the task and data available - see supplementary materials at: https://sites.google.com/view/weak-label-lfd.
I NTRODUCTION
Learning from Demonstration (LfD)
(Argall et al., 2009) is a commonly used paradigm where a potentially imperfect demonstrator desires to teach a robot how to perform a particular task in its environment.
Most often this is achieved through a combination of kinaesthetic teaching and supervised learning—imitation learning (Ross et al., 2011).
However, such approaches do not allow for elaborations and corrections from the demonstrator to be seamlessly incorporated.
As a result new demonstrations are required when either the demonstrator changes the task speciﬁcation or the agent changes its context—typical scenarios in the context of interactive task learning (Laird et al., 2017).
Such problems mainly arise because the demonstrator and the agent reason about the world by using notions and mechanisms at different levels of abstraction .
A modiﬁable LfD setup requires establishing a mapping from the high-level notions humans use—e.g. spatial concepts, different ways of applying force—to the low-level perceptive and control signals robot agents utilise—e.g. joint angles, efforts and camera images.
With this in place, any constraints or elaborations from the human operator, must be mapped to a behaviour on the agent’s side that is consistent with the semantics of the operator’s desires.
Concretely, we need to be able to ground (Vogt, 2002; Harnad, 1990) the speciﬁcations and symbols used by the operator in the actions and observations of the agent.
Often the actions and the observations of a robot agent can be high-dimensional—high DoF kinematic chains, high image resolution, etc.—making the symbol grounding problem non-trivial.
However, the concepts we need to be able to ground lie on a much-lower-dimensional manifold, embedded in the high-dimensional data space (Fefferman et al., 2016).
For example, the concept of pressing softly against a surface manifests itself in a data stream associated with the 7 DoF real-valued space of joint efforts, spread across multiple time steps.
However, the essence of what differentiates one type of soft press from another nearby concept can be summarised conceptually using a lower dimensional abstract space.
The focus of this work is ﬁnding a nonlinear mapping (represented as a high-capacity neural model) between such a low-dimensional manifold and the high-dimensional ambient space of cross-modal data.
Moreover, we show that, apart from ﬁnding such a mapping, we can also shape and reﬁne the low-dimensional
[Figure]
manifold by imposing speciﬁc biases and structures on the neural model’s architecture and training regime.
In this paper, we propose a framework which allows human operators to teach a PR2 robot about different spatial and force-related aspects of a robotic manipulation task, using tabletop dabbing as our main example.
This task serves as a concrete representative task that incorporates key issues speciﬁc to robotics (e.g. continuous actions, forceful interactions, and discrete categorization of these).
Numerous other applications require the very same capability.
Our main contributions are:
• A learning method which incorporates information from multiple high-dimensional modalities—vision, joint angles, joint efforts—in order to instill a disentangled lowdimensional manifold (Locatello et al., 2019).
By using weak expert labels during the optimisation process, the manifold eventually aligns with the human demonstrators’ ‘ common sense ’ notions in a natural and controlled way without the need of separate post-hoc interpretation.
• We release a dataset of subjective concepts grounded in multi-modal demonstrations.
Using this, we evaluate whether discrete latent variables or continuous latent variables, both shaped by the discrete user labels, better capture the demonstrated continuous notions.
I MAGE - CONDITIONED T RAJECTORY & L ABEL M ODEL
In the task we use to demonstrate our ideas, which is representative of a broad class of robotic manipulation tasks, we want to control where and how a robot presses against a table-top surface through the use of a user speciﬁcation deﬁned by a set of coarse labels—e.g. “press softly and slowly behind the cube in the image” .
In this context, let, x denote a K × T dimensional trajectory for K robot joints and a ﬁxed time length T , y denote a set of discrete labels semantically grouped in
N label groups G = { g 1 , . .
. , g N } (equivalent to multi-label classiﬁcation problem)
and i denote an
RGB image 1 .
The labels y describe qualitative properties of x and x with respect to i —e.g. left dab, right dab, hard dab, soft dab, etc.
We aim to model the distribution of demonstrated robotarm trajectories x and corresponding user labels y , conditioned on a visual environment context i .
This problem is equivalent to that of structured output representation (Walker et al., 2016; Sohn et al., 2015; Bhattacharyya et al., 2018)—ﬁnding a one-to-many mapping from i to { x , y } (one image can be part of the generation of many robot trajectories and labels).
For this we use a conditional generative model, whose latent variables c
= { c s , c e , c u } can accommodate the aforementioned mapping—see Figure 2.
The meaning behind the different types of latent variables— c s , c e and c u —is elaborated in section 3.
We want the variability in the trajectory and label data to be concisely captured by c .
We choose by design the dimensionality of the latent variables to be much lower than the dimen- sionality of the data.
Therefore, c is forced to represent abstract properties of the robot behaviour which still carry enough information for the behaviour to be usefully generated—absolute notions like speed, effort, length, and relative spatial notions which are grounded with respect to the visual context.
Another way to think of c is as a continuous version of y which can therefore incorporate nuances of the same label.
In the existing literature discrete labels are usually represented as discrete latent variables—e.g. digit classes (Kingma et al., 2014).
x
y i c
s c e c u
However, the discrete labels humans use are a rather crude approximation to the underlying continuous concepts—e.g. we may have the notion for a soft and a softer dab even though both would be alternatively labelled as soft .
For this reason we use continuous latent variables, shaped by discrete labels, to represent these subjective concepts.
The joint distribution over x and y , conditioned on i , is modelled according to Eq.
A prior of choice—isotropic Gaussian, Uniform,
Categorical—is placed over the latent variables c which are independent of the image.
[Equation] [Equation]
We choose to represent the distribution over x and y as Gaussian and Multinomial distributions respectively.
The parameters of these distributions are deﬁned as nonlinear functions (of the image context and latent variables) which are represented as the weights θ of a neural network p θ — µ θ ( · ) is a vector of means, σ θ ( · ) is a vector of log variances and π θ ( · ) is a vector of probabilities.
p θ
[Equation]
The parameters of p θ are optimised by maximising the variational lower bound (VLB) of the data distribution—see Eq. 4.
We additionally optimise a recognition network q φ , parametrised by φ , which acts as an approximation to the posterior of c , also modelled as a Gaussian.
The recognition network q φ is used to ﬁnd good candidates of c , for given x and i , thus making the optimisation of θ , tractable (Kingma & Welling, 2013).
The posterior is conditioned on the environment context as well, since i and c are conditionally dependent, once x is observed (Husmeier, 2005).
Intuitively, we want some of the latent variables to represent relative spatial concepts—e.g. left of the cube in the image.
For such concepts, under the same robot trajectory x we should get different latent codes, given different contexts i .
The omission of y from q φ means that we might not be able to fully close the gap between the data distribution and its VLB, leading to a less efﬁcient optimisation procedure (due to higher variance).
That is mainly owing to the fact that we use weak supervision—only for some demonstrations do we have labels y .
The derivation of and commentary on the optimised VLB are provided in the Supplementary Materials.
[Equation]
W EAK L ABELLING AND U SER S PECIFICATION
Interpretability through Weak Labels Even though the latent variables are constrained to be useful both for the tasks of trajectory and label generation, nothing forces them to carry the intuitive absolute and relative notions described in Section 2 in a factorised fashion—e.g. c 1 corresponds to the effort applied when pressing against the table, c 2 corresponds to how quickly the robot presses against the table, etc.
Under an optimised model, those notions can be captured in c but with no guarantee that they will be represented in an interpretable way— i.e. c needs to be disentangled .
We achieve disentanglement in the latent space by establishing a one-to-one correspondence between a subset of the latent axes— c s , c e —and the concept groups in G (Hristov et al., 2018).
c s is opti- mised to represent spatial notions, c e —effort-related notions.
The rest of the latent dimensions— c u —encode features in the data which don’t align with the semantics of the labels and their concept groups but are still necessary for good trajectory reconstruction (Figure 2).
Under these assumptions the label likelihood from Eq.
3 becomes:
[Equation]
Labelling is weak/restricted since each demonstration is given just a single label from a single concept group.
This is meant to represent a more natural LfD scenario, where the main task of the expert is still to perform a number of demonstrations, rather than to exhaustively annotate each demonstration with relevant labels from all groups.
For example, a demonstrator might say “this is how you press softly” and proceed to perform the demonstration (as is common, say, in an instructional video).
The demonstrated behaviour might also be slow and short but the demonstrator may not have been explicitly thinking about these notions, hence may not have labelled accordingly.
The missing label values are incorporated with an indicator function in Eq.
Condition on User Speciﬁcation Apart from being used in the optimisation objective, the weak labels are also used in an additional conditioning step at test time.
The generative process we have consists of ﬁrst sampling values for c from its prior.
These are then passed together with i through p θ in order to get distributions over x and y , from which we can sample.
However, we are really interested in being able to incorporate the provided information about the labels y into the sampling of semantically aligned parts of c .
Speciﬁcally, we are interested in being able to generate robot behaviours which are consistent with chosen values for y (what we call a user speciﬁcation ).
Postoptimising the θ and φ parameters, we choose to approximate the posterior over c for each label l in each concept group j with a set of Gaussian distributions N ( µ jl , Σ jl ) .
We use Maximum
Likelihood Estimation for µ jl and Σ jl over a set of samples from c .
The samples are weighted with the corresponding label likelihood p θ ( y j = l | π j ( c j )) .
As a result, the process of generating a trajectory x ∼ p θ ( x | i , c ) can be additionally conditioned on an optional user speciﬁcation y =
{ y 1 , . . .
, y |G| } , through c , in a single-label fashion (Eq. 7) or a compositional fashion (Eq. 8).
Pseudo-code for the generation of trajectories for both types of conditioning is provided in the
Supplementary Materials.
[Equation]
M ETHODOLOGY
In terms of a concrete model architecture, our model is a Conditional Variational Autoencoder (CVAE)
(Sohn et al., 2015) with an encoding network q φ and a decoding network p θ .
The parameters of both networks are optimised jointly using methods for amortised variational inference and a stochastic gradient variational bayes estimation (Kingma & Welling, 2013).
Encoding & Decoding Networks Due to the diversity of modalities that we want to use, q φ is implemented as a combination of 2D convolutions (for the image input), 1D convolutions (for the trajectoy input) and an MLP that brings the the output of the previous two modules to the common concept manifold c .
For the decoding network we implement a Temporal Convolutional Network (TCN) (Bai et al., 2018) which operates over the concatenation h of a single concept embedding c and a single image encoding i .
However, the TCN takes a sequence of length T and transforms it to another sequence of length T , by passing it through a series of dilated convolutions, while h is equivalent to a sequence of length 1 .
Thus, we tile the concatenated vector T times and to each instance of that vector h
i , i ∈ { 1 , . . .
, T } , we attach a time dimension t i = iT .
This broadcasting technique has previously shown to achieve better disentanglement in the latent space both on visual
(Watters et al., 2019) and time-dependent data
(Noseworthy et al., 2019).
We only set up the mean µ θ of the predicted distribution over x to be a non-linear function of [ h ; t ] while σ is ﬁxed, for better numerical stability during training.
This allows us to use the L 2 distance—Mean Squared
Error (MSE)—between the inferred trajectories µ θ , acting as a reconstruction, and the ground truth ones.
Label Predictors For each concept group g j we take values from the corresponding latent axis c j and pass it through a single linear layer, with a softmax activation function, to predict a probability vector π j .
Maximising the label likelihood is realised as a Softmax-Cross Entropy (SCE) term in the loss function.
Optimising this term gives us better guarantee that the some of the latent axes will be semantically aligned with the notions in the label groups.
The fact that some labels might be missing is accounted for with an indication function which calculates ∇ θ, ∇ φ with respect to the
SCE, only if y j is present.
Weighted Loss
The full loss function that we optimise is presented in Eq.
It is composed of three main terms—the trajectory MSE, equivallent to the negative trajectory log likelihood, the label
SCE, equivallent to the negative weak label log likelihood and the KL divergence between the ﬁt amortised posterior over c and its prior.
The concrete values of the three coefﬁcients— α, β, γ —are discussed in the Supplementary Materials.
min
[Equation]
+ γ |G| (cid:88)
i 1 { y i
(cid:54) = ∅} H ( c i w Ti , y i ) (cid:124) (cid:123)(cid:122) (cid:125) SCE (9)
E XPERIMENTS
The models we experiment with include a Variational Autoencoder (Kingma & Welling, 2013) with a Gaussian prior (VAE), an Adversarial Autoencoder (Makhzani et al., 2015) with a Uniform prior (AAE) and a VAE with mixed continuous and categorical latent variables (GS).
For the last model, the continuous variables have a Gaussian prior and the categorical variables utilise the GumbelSoftmax reparametrisation (Jang et al., 2017).
For each of the three models we train them while utilising the weak labels—VAE-weak, AAE-weak, GS-weak—and with no label information—VAE,
AAE, GS.
Whether or not labels are used during training is controlled by setting γ = 0 in Eq. 9.
The setup used in the experiments consists of a PR2 robot, an HTC Vive controller and headset, and a tabletop with a single object on it—a red cube.
Multiple pressing motions on the surface of the table are demonstrated by teleoperating the end effector of the robot.
The pose of the end effector, with respect to the robot’s torso, is mapped to be the same as the pose of the controller (held by the demonstrator) with respect of the headset (behind the demonstrator in Figure 4).
Data A total of 100 demonstrations were captured from the PR2, consisting of:
• Initial frame of each demonstration: 128x128-pixel Kinect2 image (RGB); • 7-DoF joint angle positions; • 7-DoF joint efforts; • 5 groups of discrete weak labels; spatial (where)
[Table] [Table]
Each demonstration has a single label attached to it.
In total we have 4 spatial symbols—where in the image, with respect to the red cube, do we dab—and 6 force-related symbols—how do we dab (Table 1).
Figure 3 provides a graphical depiction demonstrating the differences between the different force-relate groupings qualitatively.
All trajectories are standardised to be of ﬁxed length—the length of the longest trajectory ( timesteps)—by padding them with the last value for each channel/joint.
Additionally, both the joint positions and efforts are augmented with Gaussian noise or by randomly sliding them leftwards (simulating an earlier start) and padding accordingly.
The size of the total dataset after augmentation
[Figure]
is 1000 demonstrations which are split according to a 90-10 training-validation split.
The size of the latent space is chosen to be | c | = 8 .
In the context of the problem formulation in Section 2, c s = { c 0 , c 1 } , c e = { c 2 , c 3 , c 4 } and c u = { c 5 , c 6 , c 7 } .
Conditioning on a Single Label We use the
Gaussian distributions for each label l in each concept group j — N ( µ jl , Σ jl ) —to sample and generate trajectories that are consistent with the meaning of a particular label.
For a set oftest images, which have not been seen during training, we draw 1000 samples of c per image for each of the 10 labels.
For each label we then judge whether the generated trajectories match the semantics of the corresponding label using a series of manually-designed heuristics, described in the Supplementary Materials.
Average accuracy across all images and all samples is reported for each label.
For the labels of a spatial nature, for each 7-dimensional (
DoFs) robot state x i , x = { x 1 , . . .
, x T } , we compute the corresponding end-effector pose p i through the forward kinematics K of the robot.
Using the camera model of the Kinect2 sensor, each pose p i is projected in the image we condition on.
We report the Mean Absolute Error (MAE) in normalised pixel coordinates between the dab location and the landmark location (in pixel coordinates) for the misclassiﬁed trajectories.
This gives us a sense of how far from the desired ones the generated samples are.
The dab location is inferred as the point in the end-effector pose trajectory with lowest z coordinate.
For example, if we have conditioned on the left label but some of the sampled trajectories result in a dab to the right of the cube in the image, we report how far from the true classiﬁcation of left was the robot when it touched the table.
Conditioning on a Composition of Labels Since the concepts we encode in the latent space of the model should be independent of each other they can be composed together—e.g. a trajectory can be simultaneously hard , long and fast .
We examine how well is this conceptual independence preserved in the latent space by generating trajectories conditioned on all possible combinations of labels regarding effort, speed and length of press—8 possible combinations in total.
We still use the Gaussian distributions for each label in each concept group but only take the values along the relevant latent axis—Eq. 8.
To fully close the loop, the trajectories which we sample from the model could further be executed on the physical robot through a hybrid position/force controller (Reibert, 1981).
However, such evaluation is beyond the scope of the paper.
R ESULTS & D ISCUSSION
Our experiments demonstrate that the models which utilise weak user labels within our proposed setup can more reliably generate behaviours consistent with a given spatial or force-related user speciﬁcation, in contrast to baseline models which do not beneﬁt from such information.
Moreover, we ﬁnd that continuous latent variables better capture the independent continuous notions which underlie the discrete user labels in comparison to discrete latent variables.
Below we present quantitative results from the experiments with further qualitative results in the Supplementary Materials.
When conditioning on a single force label—Table 3—the best-performing models are VAE-weak and AAE-weak.
In the worse-performing cases—VAE, GS—it would appear that the models suffer from partial mode collapse.
For instance, in Table 3, penultimate row, the VAE can not represent soft trajectories, most samples are hard .
Interestingly AAE and AAE-weak perform similarly well.
We attribute that to the fact the Uniform prior used does not have a discrete number of modes to which the optimised posterior might potentially collapse.
Simultaneously, regardless of their prior, none of the weak models suffer from mode collapse, since the label information helps for the corresponding notions to be more distinguishable in the latent space.
All models have high accuracy, when conditioned on a spatial label—Table 2.
However, GS-weak, VAE-weak and AAEweak appear to be closer to the true conditioned-on concept when a bad trajectory is generated, as measured by the lower Mean Absolute Error of the image-
-effector positions.
[Table] [Table]
[Figure]
[Table]
[Figure]
While single-label conditioning is possible under all models, it is only VAE-weak, AAE-weak and
GS-weak which provide a reliable mechanism for composing labels from different independent concept groups.
That is possible due to the explicit alignment of each concept and its labels with a single latent axis.
Trajectory generation is done after conditioning on compositions of labels from the three independent concept groups designating effort—see Table 1 (right column).
Speciﬁcally, the difference between the sampled latent values for hard long slow and hard long fast , for example, are only along the axis used to predict slow from fast — c 4 in this case.
If the generated trajectories match their speciﬁcation (high measured accuracy) then indeed the perturbed axis is aligned with the notion behind the perturbed labels.
Results from this experiment are presented in Figure 5.
Evidently, the better-performing models are the ones with continuous latent variables.
While the VAE-weak and AAE-weak perform equally well for most label combinations, and do manage to encode independent concepts as such (along speciﬁc latent axes), the model with the mix of continuous and categorical latent variables (GS-weak) does not match their results.
Taking the second row in Table 3 into account, this means that the categorical model (which has more constrained capacity than the VAE-weak and AAE-weak) can capture the data distribution (shows good single-label-conditioning reconstructions) but in an entangled fashion.
Namely, going from slow to fast corresponds to perturbing more than a single latent axis simultaneously.
This further supports our initial hypothesis that while discrete, the weak labels only provide a mechanism for shaping the latent representation of what are otherwise continuous phenomena.  P OURING E XPERIMENT
An additional pouring task is demonstrated where the manner of execution varies spatially and in terms of how it is executed-see Table 4 and Figure 6. 20 demonstrations are captured for each label. [Figure] [Figure]
cup behind
[Table]
sideways fully
[Table]
partially
When compared to the dabbing demonstration, this task presents a more signiﬁcant challenge in identifying the switching latent dynamics for pouring in each cup (compared to the single red cube in the previous task, acting as a reference point for resolving spatial prepositions).
As an initial exploration step towards analysing that, we train the VAE and VAE-weak models on the data for pouring in the red cup and blue cup .
The ﬁrst latent dimension of the VAE-weak model is optimised to represent the conceptual variation of pouring in one of the two cups.
Post-training, the values of the latent vectors c for both models are ﬁxed and trajectory samples are drawn as the models are conditioned on four different test scenes, representing different cup conﬁgurations.
Figure 7 demonstrates that, averaged across the different conﬁgurations, the VAE-weak model has greater variations in the sampled trajectories for all 7 joints of the robot arm when compared to the VAE one.
On the contrary, for different images, the VAE-weak model samples trajectories which are more different from each other, hence the greater average variation.
This suggests that unless utilised, the input image modality might be ignored—the VAE does not have any auxiliary tasks to force it to use the scene image input, which can result in it rather auto-encoding the trajectory data.
In contrast, predicting where the robot pours in the scene is both a function of the joint angle trajectory and the image inputs—utilised through the classiﬁcation loss of VAE-weak.
Interpolation experiments for the VAE-weak model are presented in Figure 8 for all 4 test cup conﬁgurations.
The ﬁrst latent dimension of the model— c 0 , optimised to predict in which cup to pour—is linearly interpolated in the range [ − 2 , 2] .
In all four rows we can see the end effector projections in the image plane, through the robot kinematics and camera model, showing a gradual shift from initially pouring in the red cup to eventually pouring in the blue cup .
The color evolution of each end effector trajectory from cyan to magenta signiﬁes time progression.
Additional plots, of the x/y/z/roll/pitch/yaw proﬁles of the end effector for each sample are presented in Appendix E.
C_0=-2.
C_0=-1.
C_0=0.
C_0=1.
C_0=2.
C_0=-2.
C_0=-1.
C_0=0.
C_0=1.
C_0=2.
C_0=-2.
C_0=-1.
C_0=0.
C_0=1.
C_0=2.
C_0=-2.
C_0=-1.
C_0=0.
C_0=1. C_0=2.  R ELATED W ORK Methods which utilise high-parameter neural models to learn disentangled low-dimensional representation achieve that by either tuning the optimised loss function (Higgins et al., 2017; Chen et al., 2018), imposing crafted priors over the latent space (Chen et al., 2016) or using weak supervision (Hristov et al., 2019; Rasmus et al., 2015).
While being able to produce interpretable manifolds, most of these approaches focus on modelling visual data and respectively visually-manifested concepts, with minor exceptions—e.g. modelling motion capture data (Ainsworth et al., 2018).
In contrast, we solve a problem involving multiple heterogeneous modalities—vision, forces, joint angles.
Moreover, in addition to separating factors of variation in the data, to achieve latent space interpretability, we also deal with the problem of entanglement across modalities—i.e. which factors relate to which data modalities.
Dynamic Movement Primitives (Schaal, 2006) and Probabilistic Movement Primitives (Paraschos et al., 2013) are commonly used to model dynamical systems as ‘point-mass attractors with a nonlinear forcing term’.
However, the resulting control parameters can be high-dimensional and unintuitive for end-user manipulation.
Performing dimensionality reduction (Colom´e et al., 2014) or imposing hierarchical priors (Rueckert et al., 2015) are both ideas seen in the literature as a means of making the high-dimensional parameter space more meaningful to the human demonstrator.
These have the advantage of yielding a clearer idea of how changes in the optimised parameters result in deterministic changes in the generated behaviours.
However, most of these approaches limit themselves to end-effector trajectories, or – at most – making use of visual input (Kober et al., 2008).
Noseworthy et al.
(2019) explore disentangling task parameters from manner of execution parameters, in the context of pouring.
They utilise an adversarial training regime, which facilitates better separation of the two types of parameters in the latent space.
However, it is assumed that the task parameters are known a priori .
Interpretation of the learned codes is achieved post-training by perturbing latent axis values and qualitatively inspecting generated trajectories (standard evaluation technique for models which are trained unsupervised).
We argue that through the use of weak supervision through discrete labels we have ﬁner control over the ‘meaning’ of latent dimensions.
C ONCLUSION
We recontextualise the problem of interpretable multi-modal LfD as a problem of formulating a conditional probabilistic model.
In the example task of table-top dabbing, we utilise weak discrete labels from the demonstrator to represent abstract notions, manifested in a captured multi-modal robot dataset.
Through the use of high-capacity neural models and methods from deep variational inference, we show how to align some of the latent variables of the formulated probabilistic model with high-level notions implicit in the demonstrations.
A V ARIATIONAL L OWER B OUND D ERIVATION
For completeness we include a derivation of the Variational Lower Bound of the data likelihood
The VLB is used as an optimisation objective—Eq. 9.
Jensen’s inequality becomes an equality when the amortised posterior q ( c | x , i ) matches exactly the true posterior p ( c | x , i , y ) .
The posterior we optimise is not conditioned on y , potentially meaning that we might never close the gap between the log of the data distribution and the VLB, measured by D KL ( q ( c | x , i )
||
p ( c | x , i , y )) .
However, maximising the VLB still goes in the direction of maximising the data distribution. log
[Equation]
[Jensen]
[Equation]
B L ABEL C ONDITIONING & E VALUATION H EURISTICS
Algorithms 1 and 2 provide pseudo-code for the trajectory generation procedures described in Section 3.
The main difference can be summarised as following—when the generative process is conditioned on a single label l , all 8 dimensions of the latent samples are drawn from a single 8-dimensional Gaussian distribution associated with that label and its concept group—line 2 in Algorithm 1.
However, when the process is conditioned on a composition of labels from different groups—a user speciﬁcation—the sampling procedure differs as c is iteratively built.
For each latent axis c j , if a label is speciﬁed for the concept group aligned with c j , only the values along the j -th axis are taken of a sample from the corresponding 8-dimensional Gaussian.
If no label is given, c j is sampled from a Standard Normal distribution N (0 , 1) (or the respective prior of choice.
Same applies for latent dimensions which have not been aligned with any concept groups, since | c | > |G| —lines 2 to 9 in Algorithm 2.
and j < = |G| then  sample w ∼ N ( w | µ jl , Σ jl ) , such that, l = y j ; only take values along the j -th dimension: append w [ j ] to c ; else sample w ∼ N ( w | 0 , 1) ; append w to c ; 10 concatenate c and i : h ←
c ; i
The hand-designed heuristics used for reporting accuracy on the the generated trajectories have concrete semantics and values which have been chosen after closely inspecting the captured dataset, the given user labels and the robot conﬁguration during the demonstrations. :
• spatial labels— left , right , front , behind —the location of the red object in the scene is extracted using standard computer vision techniques based on color segmentation.
The position of the cube is inferred as the center of mass of all red pixels detected.
From the overall end-effector trajectory, derived through the robot forward kinematics, the pressing location is chosen as the end-effector position with the lowest z coordinate.
That position is then projected into the image plane in order to determine where with respect to the detected object position did the robot touch the table.
• soft vs hard trajectories - the generated trajectories have normalised values in the range [ − 1 , 1] .
A soft trajectory is considered as such if speciﬁc joint efforts are below a particular threshold.
More speciﬁcally: – shoulder lift joint efforts are below 0.5.
– upper arm roll joint effort are above -0.5.
• short vs long trajectories - the generated trajectories have length of T = 240 .
A short trajectory is considered as such if efforts in the range of [max effort-0.2, max effort] are maintained for more than 50 time steps.
Otherwise the trajectory is deemed long .
The heuristic operates over the following joints: – shoulder lift joint efforts – upper arm roll joint efforts • slow vs fast trajectories - a slow trajectory is considered as such if the time it takes to reach maximal joint effort is more than 80 time steps.
Otherwise the trajectory is deemed fast .
The heuristic operates over the following joints: – shoulder lift joint efforts – upper arm roll joint efforts
For resolving robot joint names, please consult Figure 9 below:
C A RCHITECTURE D ETAILS
The model architectures are implemented in the PyTorch framework 3 and is visually presented in
Figure 10.
The image encoder network takes as input a single RGB 128x128 pixel image.
The trajectory encoder takes a single 14-dimensional, 240 time-step-long trajectory.
Their outputs both feed into an MLP network which, after a series of nonlinear transformations, outputs parameters of a distribution over the latent space c .
Through the reparametrisation trick Kingma & Welling (2013) we sample values for c .
Through a residual connection, the output of the image encoder is concatenated to the sampled latent values.
The resultant vector is tiled 240 times, extended with a time dimension, and fed into a TCN decoder network to produce reconstructions for the original 14-dimensional input trajectories.
[Table]
learningrate = 0 .
001 , β 1 = 0 .
9 , β 2 = 0 .
999 , eps = 1 e − 08 , weightdecayrate = 0 , amsgrad = False ).
For all experiments, the values (unless when set to 0) for the three coefﬁcients from Equation 9 are:
• α = 1 , β = 0 .
1 , γ =
The values are chosen empirically in a manner such that all the loss terms have similar magnitude and thus none of them overwhelms the gradient updates while training the full model.
D Q UALITATIVE E VALUATION FOR D ABBING
Figures 11 to 15 present the qualitative results from perturbing in a controlled way the latent space optimised under the VAE-weak model.
For ﬁgure the ﬁrst row of plots represents the generated joint angle positions for each of the 7 joints, for each of the 5 drawn samples, the second row—the generated joint efforts—and the third row the corresponding end-effector positions, from the forward robot kinematics, projected in the image plane.
As we can see, perturbing c 0 and c 1 corresponds to clear and consistent movement of the end effector (through the generated joint position proﬁles) while there is not much change in the generated effort proﬁles—Figures 11 and 12.
Simultaneously, we observe the opposite effect in other 3 Figures.
Perturbations in c 2 correspond solely changes in the output efforts in the shoulder lift and upper arm rolls joints—Figure 13 (i) and (j).
Perturbations in c 3 correspond to changes in length for which the joint efforts are exerted but does not change their magnitude or where they are applied on the table—Figure 14.
And ﬁnally, perturbations in ccorrespond to changes on how quickly the maximum efforts are reached - most noticeable in the changing slope of the shoulder lift and upper arm rolls joints—Figure 15 (i) and (j).
As a result, we can conclude that the latent dimensions, which we aligned with the semantically grouped weak labels, have indeed captured the notions and concepts which underlie the labels.
[Figure]
Location
[Figure] [Figure]
[Figure]
C3=2.
C3=-2.0C3=-1.0C3=0.0C3=1.
C3=2.0StartEndDab
Location
[Figure]
E Q UALITATIVE E VALUATION FOR P OURING
Figures 16 to 19 present the qualitative results from perturbing in a controlled way the latent space optimised under the VAE-weak model for the pouring task across different scene conﬁgurations.
Each ﬁgure represents the image projections of the end-effector, together with the associated x/y/z and roll/pitch/yaw proﬁles.
Each sampled pour can either be a behind pour—dip in the roll proﬁle—or a sideways pour—bump across the roll and yaw proﬁles (which can sometimes wrap around from 180 degrees to -180 degrees.
[Figure]
[Figure]
[Figure]
[Figure]