Implicit Behavioral Cloning
Abstract
We find that across a wide range of robot policy learning scenarios, treating supervised policy learning with an implicit model generally performs better, on average, than commonly used explicit models.
We present extensive experiments on this finding, and we provide both intuitive insight and theoretical arguments distinguishing the properties of implicit models compared to their explicit counterparts, particularly with respect to approximating complex, potentially discontinuous and multi-valued (set-valued) functions.
On robotic policy learning tasks we show that implicit behavioral cloning policies with energy-based models (EBM) often outperform common explicit (Mean Square Error, or Mixture Density) counterparts, including on tasks with high-dimensional action spaces and visual image inputs.
We find these policies provide competitive results or outperform state-of-the-art offline reinforcement learning methods on the challenging human-expert tasks from the D4RL benchmark suite, despite using no reward information.
In the real world, robots with implicit policies can learn complex and remarkably subtle behaviors on contact-rich tasks from human demonstrations, including tasks with high combinatorial complexity and tasks requiring 1mm precision.
Introduction
Behavioral cloning (BC)
[1] remains one of the simplest machine learning methods to acquire robotic skills in the real world.
BC casts the imitation of expert demonstrations as a supervised learning problem, and despite valid concerns (both empirical and theoretical) about its shortcomings (e g., compounding errors [2, 3]), in practice it enables some of the most compelling results of real robots generalizing complex behaviors to new unstructured scenarios [4, 5, 6].
Although considerable research has been devoted to developing new imitation learning methods [7, 8, 9] to address BC’s known limitations, here we investigate a fundamental design decision that has largely been overlooked: the form of the policy itself.
Like many other supervised learning methods, BC policies are often represented by explicit continuous feed-forward models (e g., deep networks) of the form ˆ a = F θ ( o ) that map directly from input observations o to output actions a ∈A .
But what if F θ is the wrong choice?
In this work, we propose to reformulate BC using implicit models – specifically, the composition of argmin with a continuous energy function E θ (see Sec.
2 for definition) to represent the policy π θ : ˆ a =argmin a ∈A
E θ
( o , a ) instead of ˆ a = F θ ( o ) .
In this formulation, imitation is cast as a conditional energy-based modeling (EBM) problem [10] (Fig. 1), and at inference time (given o ) performs implicit regression by optimizing for the optimal action ˆ a via sampling or gradient descent [11, 12].
This approach provides a direct connection with critic-only Q-learning methods [13], in which no explicitly-represented actor is required, and actions are selected implicitly from the Q function.
Critically, however, our approach uses only supervised learning, with no rewards required, and isolates the explicit versus implicit policy choice without any confounding factors such as Bellman updates.
This enables a unique case study that investigates the choice between implicit vs. explicit policies, and can inform the broader reinforcement learning literature.
Forked from https://docs.google.com/drawings/d/1SAUv3dd5K31ZupawJggATf4aFoKVKxMQJ2qGx-Frp8M/edit?resourcekey=0-XbrfXUSfSBgZMSdpe8bHVA
[Figure]
Our experiments show that this simple change can lead to remarkable improvements in performance across a wide range of contact-rich tasks: from bi-manually scooping piles of small objects into bowls with spatulas, to precisely pushing blocks into fixtures with tight 1mm tolerances, to sorting mixed collections of blocks by their colors.
Results show that implicit models for BC exhibit the capacity to learn long-horizon, closed-loop visuomotor tasks better than their explicit counterparts – and surprisingly, give rise to a new class of BC baselines that are competitive with state-of-the-art offline RL algorithms on standard simulated benchmarks [14].
To shed light on these results, we provide observations on the intuitive properties of implicit models, and present theoretical justification that we believe are highly relevant to part of their success: their ability to represent not only multi-modal distributions, but also discontinuous functions.
Contributions.
Our primary contributions are as follows.
(i)
We present Implicit Behavioral Cloning (IBC), which is a novel, simple method for imitation learning in which behavioral cloning is cast as a conditional energy-based modeling (EBM) problem, and inference is performed via stochastic optimization. (ii)
We validate IBC in real-world robot experiments, where on several end-to-end, image-input -only, human-demonstrated contact-rich pushing tasks, IBC performs significantly better than our best explicit
BC models.
(iii)
We present extensive simulation experiments comparing IBC to both comparable explicit
BC models and also SOTA (state-of-the-art) offline RL methods on the human-expert tasks from the standard D4RL benchmark.
Averaged across D4RL human-expert tasks IBC outperforms our best explicit
BC models and CQL [15], and through simple reward-based filtering [16, 17], IBC policies achieve new
SOTA performance, outperforming even S4RL [18].
(v) We analyze the nature of implicit models in simple 1D-1D examples, and we highlight aspects of implicit models that we believe are not known to the generative modeling community, including their behavior (a) at discontinuities and (b) in extrapolation.
(vi)
We provide theoretical insight into implicit models, including proofs of their (i) representational abilities (Thm. 1) and (ii) approximation abilities (Thm. 2), which are shown to be distinct from continuous explicit models in their ability to handle discontinuities and set -valued functions.
Paper Organization.
After a brief background (Sec. 2), to build intuition on the nature of implicit models, we present their empirical properties (Sec. 3).
We then present our main results with policy learning (Sec. 4), both in simulated tasks and in the real world.
Inspired by these results, we provide theoretical insight (Sec. 5), followed by related work (Sec. 6) and conclusions (Sec. 7).
Background: Implicit Model Training and Inference
We define an implicit model as any composition (argmin y ◦
E θ ( x , y )) , in which inference is performed using some general-purpose function approximator E : R m + n → R 1 to solve the optimization problem ˆ y = argmin
y E
θ ( x , y ) .
We use techniques from the energy-based model (EBM) literature to train such a model.
Given a dataset of samples { x i , y i } , and regression bounds y min , y max ∈ R m , training consists of generating a set of negative counter-examples { ˜ y ji } N neg.
j =1 for each sample x i in a batch, and employing an InfoNCE-style [19] loss function.
This loss equates to the negative log likelihood of p θ ( y | x )= exp( − E θ ( x , y ))
Z ( x ,θ ) , and the counter-examples are used to estimate Z ( x i ,θ ) :
[Equation]
With a trained energy model E θ ( x , y ) , implicit inference can be performed with stochastic optimization to solve ˆ y =argmin
y E
θ ( x , y ) .
To demonstrate a breadth of approaches, we present results with three different EBM training and inference methods discussed below, however a comprehensive comparison of all
EBM variants is outside the scope of this paper; see [20] for a comprehensive reference.
We use either a) a x x x y y
y
x
x x y y y
Explicit
[Figure]
Implicit
ReLU-MLP trained as EBM 2:512:512:15k steps shown density is: ̂ y = f θ ( x ) ̂
y = argmin
y
E θ ( x , y ) normalized y ( E θ ( x , y ))
( a ) ( b ) ( c ) ( d ) ( e ) ( f ) derivative-free (sampling-based) optimization procedure, b) an auto-regressive variant of the derivative-free optimizer which performs coordinate descent, or c) gradient-based Langevin sampling [11, 12] with gradient penalty [21] loss during training – see the Appendix for descriptions and comparisons of these choices.
Intriguing Properties of Implicit vs. Explicit Models
Consider an explicit model y
=
f θ ( x ) , and an implicit model argmin y E θ ( x , y ) where both f θ ( · ) and E θ ( · ) are represented by almost-identical network architectures.
Comparing these models, we examine: (i) how do they perform near discontinuities?
, (ii) how do they fit multi-valued functions?, and (iii) how do they extrapolate?
For both f θ and E θ we use almost-identical ReLU-activation fully-connected Multi-Layer Perceptrons (MLPs), with the only difference being the additional input of y in the latter.
Explicit “MSE” models are trained with Mean Square Error (MSE), explicit “MDN” models are Mixture Density Networks (MDN) [22], and implicit “EBM” models are trained with L InfoNCE and optimized with derivative-free optimization.
Figs. 2, 3 show models trained on a number of R 1 → R 1 functions (Fig. 2) and multi-valued functions (Fig. 3).
For each of these we examine regions of discontinuities, multi-modalities, and/or extrapolation.
Discontinuities.
Implicit models are able to approximate discontinuities sharply without introducing intermediate artifacts (Fig. 2a), whereas explicit models (Fig. 2d), because they fit a continuous function to the data, take every intermediate value between training samples.
As the frequency of discontinuities increases, the implicit model predictions remain sharp at discontinuities, while also respecting local continuities, and with piece-wise linear extrapolations up to some decision boundary between training examples (Fig.
2a-c).
The explicit model interpolates across each discontinuity (Fig. 2d-f).
Once the training data is uncorrelated (i.e. random noise) and without regularization (Fig. 2c, Fig. 2f) , implicit models exhibit a nearest-neighbors-like behavior, though with non-zero ∂y∂x segments around each sample.
Extrapolation.
For extrapolation outside the convex hull of the training data (Fig.
2a-f), even with discontinuous or multi-valued functions, implicit models often perform piecewise linear extrapolation of the piecewise linear portion of the model nearest to the edge of the training data domain.
Recent work [23] has shown that explicit models tend to perform linear extrapolation, but the analysis assumes the ground truth function is continuous.
Multi-valued functions.
Instead of using argmin to identify a single optimal value, argmin may return a set of values, which may either be interpreted probabilistically as sampling likely values from the distribution, or in optimization as the set of minimizers ( argmin is set-valued).
Fig.
3 compares a ReLU-MLP trained as a Mixture Density Network (MDN) vs an EBM across three example multi-valued functions.
Visual Generalization Of particular relevance to learning visuomotor policies, we also find striking differences in extrapolation ability with converting high-dimensional image inputs into continuous outputs.
[Figure]
Fig.
4 shows how on a simple visual coordinate regression task, which is a notoriously hard problem for convolutional networks [24], an MSE-trained Conv-MLP model [25] with CoordConv [24] struggles to extrapolate outside the convex hull of the training data.
This is consistent with findings in [5, 26].
A
Conv-MLP trained via late fusion (Fig. 4b) as an EBM, on the other hand, extrapolates well with only a few training data samples, achieving 1 to 2 orders of magnitude lower test-set error in the low-data regime (Fig. 4d).
This is additional evidence that distinguishes implicit models from explicit models in a distinct way from multi-modality, which is absent in this experiment
Policy Learning Results
[Table]
We evaluate implicit models for learning BC policies across a variety of robotic task domains (Fig. 5).
The goals of our experiments are three-fold: (i) to compare the performance of otherwise-identical policies when represented as either implicit or explicit models, (ii) to test how well our models (both implicit and explicit) compare with author-reported baselines on a standard set of tasks [Table] and (iii) to demonstrate that implicit models can be used to learn effective policies from human demonstrations with visual observations on a real robot.
The following results and discussions are organized by task domain – each evaluating a unique set of desired properties for policy learning (Table 1)
[Table]
All tasks are characterized by discontinuities and require generalization (e.g., extrapolation) to some degree.
[Table]
Nearest
[Table]
D4RL [14] is a recent benchmark for offline reinforcement learning.
We evaluate our implicit (EBM) and explicit (MSE) policies across the subset of tasks for which offline datasets of human demonstrations are provided, which is arguably is the hardest set of tasks.
Surprisingly, we find that our implementations of both implicit and explicit policies significantly outperform the BC baselines reported on the benchmark, and provide competitive results with state-of-the-art offline reinforcement learning results reported thus far, including CQL [15] and S4RL [18].
By adding perhaps the simplest way to use reward information, if we prioritize sampling to be only the top 50% of demonstrations sorted by their returns (similar to
Reward-Weighted Regression (RWR) [16])
, this intriguingly generally improves implicit policies, in some cases to new state-of-the-art performance, while less so for explicit models.
This suggests that implicit BC policies value data quality higher than explicit BC policies do.
A simple Nearest-Neighbor baseline (see
Appendix) performs better than one might expect on these tasks, but on average not as well as implicit BC.
While many of the D4RL tasks have complex high-dimensional action spaces (up to 30-D), they do not emphasize the full spectrum of task attributes (Table 1) we are interested in.
The following tasks isolate other attributes or introduce new ones, such as highly stochastic dynamics (i.e., single-point-of-contact block pushing), complex multi-object interactions (many small particles), and combinatorial complexity.
N-D Particle Integrator is a simple environment with linear dynamics but where a discontinuous oracle policy is used to generate training demonstrations: once within the vicinity of goal-conditioned location (Fig. 5, shown for N = 2 ), the policy must switch to the second goal.
The benefit of studying this environment is two-fold: (i) it has none of the complicating attributes in Table 1 and so allows us to study discontinuities in isolation, and (ii) we can define this simple environment to be in N dimensions.
Varying N from 1 to 32 dimensions, but holding the number of demonstrations constant, we find we are able to train 95% successful implicit policies up to 16 dimensions, whereas explicit (MSE) policies can only do 8 dimensions with the same success rate.
The Nearest-Neighbor baseline, meanwhile, cannot generalize, and only performs well on the 1D task (see Appendix for more analysis).
Method
SingleTarget, MultiTarget, SingleTarget, states states pixels
EBM 100 ±99.0 ± 0.
100 ± 0 MDN 100 ±99.7 ± 0.
10.0 ± 4.3 MSE 98.3 ± 0.
89.7 ± 4.
87.0 ± 4.1 Nearest-Neighbor 4.0 ± 0.
0.0 ± 0.
4.3 ± 1.
Simulated Pushing consists of a simulated 6DoF robot xArm6 in PyBullet [27] equipped with a small cylindrical end effector.
The task is to push a block into the target goal zone, marked by a green square labeled on the tabletop.
We investigate 2 variants: (a) pushing a single block to a single target zone, or (b) also pushing the block to a second goal zone (multistage).
We evaluate implicit (EBM) and explicit (MSE and MDN [28, 29]) policies on both variants, trained from a dataset of 2,000 demonstrations using a scripted policy that readjusts its pushing direction if the block slips from the end effector.
Results in
Table 3 show that all learning methods perform well on the single-target task, while MSE struggles with the slightly longer task horizon.
For the image-based task, the MDN significantly struggles compared to
MSE and EBM.
The failures of the Nearest-Neighbor baseline, with only 0-4% success rate, show that generalization is required for this task.
Planar Sweeping [30] is a 2D environment that consists of an agent (in the form of a blue stick) where the task is to push a pile of 50 - 100 randomly positioned particles into a green goal zone.
The agent hasdegrees of freedom (2 for position, 1 for orientation).
We train implicit (EBM) and explicit (MSE) policies from 50 teleoperated human demonstrations, and test on episodes with unseen particle configurations.
For the image-based inputs, we also test two types of encoders with different forms of dimensionality reduction: spatial soft(arg)max and average pooling over dense features (see Appendix for architecture descriptions).
For the state-based inputs, since the number of particles vary between episodes, we flatten the poses of the particles and 0-pad the vector to match the size of the vector at maximum particle cardinality.
[Figure]
[Table]
[Figure]
The results in Table 4 (averaged over 3 training runs with different seeds) suggest that image-based
EBMs outperform the best MSE architectures by 7%.
Interestingly, image-based EBMs seem to synergize well with spatial soft(arg)max for dimensionality reduction, as opposed to pooling, which works best for MSE explicit policies.
In both cases, state observations as inputs do not perform well compared with image pixel inputs.
This is likely because the particles have symmetries in image space, but not when observed as a vector of poses.
Simulated Bi-Manual Sweeping consists of two robot
KUKA IIWA arms equipped with spatula-like end effectors.
The task is to scoop up randomly configured particles from a 0 .
4 m 2 workspace and transport them into two bowls, which should be filled up equally.
Successfully scooping particles and transporting them requires precise coordination between the two arms (e.g., such that the particles do not drop while being transported to the bowls).
The action space is 12DoF (6DoF Cartesian per arm), and each episode consists of 700 steps recorded at 10Hz.
Perspective RGB images from a simulated camera are used as visual input, along with current end effector poses as state input.
The task is characterized by many mode changes and discontinuities (transitioning from scooping to lifting, from lifting to transporting, and deciding which bowl to transport to).
EBM and MSE policies on the task use the best corresponding image encoder from the planar sweeping task.
As shown in Table 5, our results show that EBM outperforms MSE by 14%.
[Figure]
Real Robot Manipulation , using a cylindrical end-effector on an xArm6 robot (Fig. 9a), we evaluate implicit BC and explicit BC policies on 4 real-world manipulation pushing tasks: 1) pushing a red block and a green block into assigned target fixtures, 2) pushing the red and green blocks into either target fixture, in either order, 3) precise pushing and insertion of a blue block into a tight (1mm tolerance) target fixture, and 4) sortation of 4 blue blocks and 4 yellow into different targets.
The observation input is only raw perspective RGB images at 5Hz, with task horizons up to 60 seconds, and teleoperated demonstrations.
[Table]
Across all four tasks, we observe significantly higher performance for the implicit policies compared to the explicit baseline.
This is especially apparent on the pushing-and-oriented-insertion task ( Insert Blue ), which requires highly discontinuous behavior in order to subtly nudge enough, but not too far, the block into place (Fig. 9c).
On this task we see the implicit BC policy has an order of magnitude higher success rate than the explicit BC policy.
The sorting task in particular ( Sort-Blue-From-Yellow , Fig.
9d) is our attempt to push the generalization abilities of our models, and we see a 2.4x higher success rate for the implicit policy.
Note these experimental results are averaged over 3 different models, for each task, for each policy type.
The red/green pushing tasks, including multi-modal variant (Fig. 9b), also show notably higher success rates for the implicit policies.
These real-world results are best appreciated in our video.
Theoretical Insight: Universal Approximation with Implicit Models
In previous sections, we have empirically demonstrated the ability of implicit models to handle discontinuities (Section 3), and we hypothesized this is one of the reasons for the strong performance of implicit
BC policies (Section 4).
Two theoretical questions we now ask are: (i) is there a provable notion for what class of functions can be represented by implicit models given some analytical E ( · ) , and (ii) given that energy functions learned from data may always be expected to have non-zero error of approximating any function, are there inference risks with large behaviour shifts resulting from a combination of argmin and spurious peaks in E ( · ) ?
Recent work [31] has shown that a large class of functions (namely, functions defined by finitely many polynomial inequalities) can be approximated implicitly by argmin y g
( x , y ) using
SOS polynomials to represent g ( · ) .
Here we show that for implicit models with g θ represented by any continuous function approximator (such as a deep ReLU-MLP network)
, argmin y g θ ( x , y ) can represent a larger set of functions including multi-valued functions and discontinuous functions (Thm. 1), to arbitrary accuracy (Thm. 2).
These results are stated formally in the following; proofs are in the Appendix.
Theorem 1.
For any set-valued function F ( x ): x ∈ R m → P ( R n ) \{∅} where the graph of F is closed, there exists a continuous function g ( x , y ): R m + n → R 1 , such that argmin y g ( x , y ) = F ( x ) for all x .
[Figure]
Theorem 2.
For any set-valued function F ( x ): x ∈ R m → P ( R n ) \{∅} , there exists a function g ( · ) that can be approximated by some continuous function approximator g θ ( · ) with arbitrarily small bounded error (cid:15) , such that ˆ y = argmin
y g θ ( x , y ) provides the guarantee that the distance from ( x , ˆ y ) to the graph of F is less than (cid:15) .
Of practical note, explicit functions ( F ( x ) in Thms.
1 and 2) with arbitrarily small or large Lipschitz constants can be approximated by an implicit function with bounded Lipschitz constant (see Appendix for more discussion).
This means that implicit functions can approximate steep or discontinuous explicit functions without large gradients in the function approximator that may cause generalization issues.
This is not the case for explicit continuous function approximators, which must match the large gradient of the approximated function.
In both their multi-valued nature and discontinuity-handling, the approximation capabilities of implicit models are distinctly superior to explicit models.
See Fig.
10 for visual intuition, and more discussion in the Appendix.
Related Work
Energy-Based Models, Implicit Learning.
Reviews of energy-based models can be found in LeCun et al. [10] and Song & Kingma [20].
Du & Mordatch [12] proposed Langevin MCMC [11] sampling for training and implicit inference, and argued for several strengths of implicit generation, including compositionality and empirical results such as out-of-distribution generalization and long-horizon sequential prediction.
A general framework for energy-based learning of behaviors is also presented in [32].
In applications, energy based models have recently shown state-of-the-art results across a number of domains, including various computer vision tasks [33, 34], as well as generative modeling tasks such as image and text generation [12, 35, 36].
Many other works have investigated using the notion of implicit functions in learning, including works that investigate implicit layers [37, 38, 39, 40].
There is also a surge of interest in geometry representation learning in implicit representations [41, 42, 43, 44].
In robotics, implicit models have been developed for modeling discontinuous contact dynamics [45].
Energy-Based Models in Policy Learning .
In reinforcement learning, [46] uses an EBM formulation as the policy representation.
Other recent work [47] uses EBMs in a model-based planning framework, or uses EBMs in imitation learning [48] but with an on-policy algorithm.
A trend as well in recent RL works has been to utilize an EBM as part of an overall algorithm, i.e. [49, 50].)
Policy Learning via Imitation Learning.
In addition to behavioral cloning (BC) [1], the machine learning and robotics communities have explored many additional approaches in imitation learning [51, 52, 53], often in ways that need additional information.
One route is by collecting on-policy data of the learned policy, and potentially either labeling with rewards to perform on-policy reinforcement learning (RL) [54, 55, 56] or labeling actions by an expert [2].
Distribution-matching algorithms like GAIL [7] require no labeling, but may require millions of on-policy environment interactions.
While algorithms like ValueDice [57] implement distribution matching in a sample-efficient off-policy setting, they have not been proven on image-observations or high degree-of-freedom action spaces.
Another route to using more information beyond BC is for the off-policy data to be labeled with rewards, which is the focus of the offline RL community [14].
All of these directions are good ideas.
A perhaps not fully appreciated finding, however, is that in some cases even the simplest forms of BC can yield surprisingly good results.
On offline RL benchmarks, prior works’ implementations of BC already show reasonably competitive results with offline
RL algorithms [14, 58].
In real-world robotics research, BC has been widely used in policy learning [4, 28, 5, 26].
Perhaps the success of BC comes from its simplicity : it has the lowest data collection requirements (no reward labels or on-policy data required), can be data-efficient [5, 26], and it is arguably the simplest to implement and easiest to tune (with fewer hyperparameters than RL-based methods).
Approximation of Discontinuous Functions.
The foundational results of Cybenko [59] and others in
Universal Approximation of neural networks have had foundational impact in guiding machine learning research and applications.
Various approaches have been developed in the function approximation literature and elsewhere to approximate discontinuous functions [60, 61, 62, 63], which typically do not use neural networks.
Also motivated by applications to modeling phenomena for robots, [64] develops theory of approximating discontinuous functions with neural networks, but the method requires a-priori knowledge of the discontinuity’s location.
Our work builds on the well-known and well-applied results in continuous neural networks, but through composition with argmin provides a notion of universal approximation even for discontinuous, set-valued functions.
Conclusion
In this paper we showed that reformulating supervised imitation learning as a conditional energy-based modeling problem, with inference-time implicit regression, often greatly outperforms traditional explicit policy baselines.
This includes on tasks with high-dimensional action spaces (up to 30-dimensional in the D4RL human-expert tasks), visual observations , and in the real world .
In terms of limitations, a primary comparison with explicit models is that they typically require more compute, both in training and inference (see Appendix for comparisons).
However, we have both shown that we can run implicit policies for real-time vision-based control in the real world, and training time is modest compared to offline RL algorithms.
To further motivate the use of implicit models, we presented an intuitive analysis of energy-based model characteristics, highlighting a number of potential benefits that, to the best of our knowledge, are not discussed in the literature, including their ability to accurately model discontinuities.
Lastly, to ground our results theoretically we developed a notion of universal approximation for implicit models which is distinct from that of explicit models.