Can Foundation Models Perform Zero-Shot Task Speciﬁcation For Robot Manipulation?
Anonymous Authors
Abstract
Task speciﬁcation is at the core of programming autonomous robots.
A low-effort modality for task speciﬁcation is critical for engagement of non-expert end-users and ultimate adoption of personalized robot agents.
A widely studied approach to task speciﬁcation is through goals, using either compact state vectors or goal images from the same robot scene.
The former is hard to interpret for non-experts and necessitates detailed state estimation and scene understanding.
The latter requires the generation of desired goal image, which often requires a human to complete the task, defeating the purpose of having autonomous robots.
In this work, we explore alternate and more general forms of goal speciﬁcation that are expected to be easier for humans to specify and use such as images obtained from the internet, hand sketches that provide a visual description of the desired task, or simple language descriptions.
As a preliminary step towards this, we investigate the capabilities of large scale pre-trained models (foundation models) for zero-shot goal speciﬁcation, and ﬁnd promising results in a collection of simulated robot manipulation tasks and real-world datasets.
1. Introduction
Robots are gradually entering our homes to help automate aspects of our everyday life.
End users of such technologies are likely to have personalized requirements and preferences.
A low-effort and intuitive modality of communication is needed to allow non-expert users to program robots and customize them to user needs.
This requires robots to be equipped with a form of “common sense” that is grounded in human-centric experiences and understanding of the world.
Recent advances in computer vision (Krizhevsky et al., 2012) and natural language processing (NLP) (Devlin et al., 2019; Radford et al., 2019) have enabled machine learning models to make sense of images and text using large-scale internet datasets, that comprise in part, data generated by humans in human-centric environments.
Of particular interest are “foundation models” (Bommasani et al., 2021) – deep neural network models trained on massive internet datasets – that have powered impressive advances in downstream vision and NLP tasks.
In this backdrop, our work explores if such foundation models can also enable task speciﬁcation for robotics and advance embodied intelligence.
Traditional approaches to program robots, both at training and deployment times, involve the use of reward functions, task demonstrations, or goal speciﬁcation.
Among these, goal speciﬁcation is particularly appealing from an end user viewpoint since goals are easier to provide compared to full demonstrations and also require minimal expertise compared to scripting detailed reward functions that are prone to exploitation (Amodei et al., 2016).
Existing goal-conditioned approaches use either compact state space goal vectors or RGB images from the same robot scene.
The former is often hard to interpret and requires detailed state estimation, which is difﬁcult outside lab settings.
The latter forces a human expert to perform the task to generate the necessary goal image, thereby defeating the purpose of an autonomous robot.
We study two alternate forms of goal conditioning to overcome the aforementioned difﬁculties.
The ﬁrst approach involves goals or instructions in natural language, such as “open the microwave” .
To be successful, the robot must understand grounding and semantics – it must be capable of understanding what a microwave looks like and also be capable of differentiating between open and closed microwaves.
The second approach involves using goal images from a different scene, such as images from the internet.
For example, to instruct a robot to open a microwave, we can provide the robot an image of an open microwave obtained from the internet.
To be successful, the robot must have domain adaptation capabilities and form correspondences between the state of the microwave in the goal image and current scene.
The goal of our work is to study if foundation models are useful for such types of task speciﬁcation, particularly in the zero-shot regime where they have been very successful in traditional vision and NLP tasks.
Our Contributions:
In this work, (1) We introduce a framework for studying foundation models for zero-shot task speciﬁcation (ZeST).
See Figure 1 for an overview.
(2) We evaluate the effectiveness of ZeST for enabling zero-shot policy execution through a set of goal selection tasks; and (3) We evaluate ZeST for policy learning in ofﬂine reinforcement learning.
We ﬁnd that ZeST is quite effective in zero-shot goal-selection and provides a 14-fold increase in performance over a random guessing baseline.
In ofﬂine RL, we ﬁnd that using ZeST scores as a proxy for the reward function enables the learning of policies that perform better than a behavior cloning baseline.
2. Background and Related Work
We consider environments that take the form of a High-Dimensional Markov Decision Process (MDPs) , described by the tuple M = (cid:104)S , X , A , P, R, d 0 , γ (cid:105) .
Informally, this setting has been widely studied and was characterized more formally by Du et al.
(2019) as “Block MDPs”.
Here,
S denotes a compact state space that is not directly observable by the agent, X denotes the highdimensional observation space that contains sufﬁcient information to uniquely recover the underlying state, and A is the action space.
For example, the observation space can be (multi-view) camera images while the state space can be object poses.
The underlying transition dynamics and reward function are described in the state space as P ( s (cid:48) | s, a ) and R ( s ) .
Additionally, d 0 is the starting state distribution and γ ∈ [0 , 1) is the discount factor.
Since the observation can be mapped uniquely to the state, an equivalent MDP can be constructed in the observation space as well.
A trajectory τ = { ( X 0 , a 0 ) , ( X 1 , a 1 ) , ..., ( X H , a H ) } is a sequence of observation-action pairs of length H .
A policy, π ( a t | X t ) maps from observations to a probability distribution over actions.
The objective is to learn a policy that maximizes the long term reward, i.e. max π E (cid:2)(cid:80) ∞ t =
γ t R ( s t ) (cid:3) where the expectation is under s 0 ∼ d 0 , a t ∼ π ( ·| X t ) and s t +1 ∼ P ( ·| s t , a t ) .
Goal-Conditioned Policies
In the standard RL formulation, an agent learns a policy for a single task (reward function).
Goal-conditioning allows the agent to perform multiple tasks by conditioning on different goals from a goal space G ∈ G .
The policy π ( a t | X t , G ) and reward function
R ( s t , G ) can be made a function of the goal, thereby enabling multi-task learning.
We can interpret goal-conditioning as constructing a broader MDP with an augmented state and observation spaces
S ∪ G and X ∪ G .
The idea of goal-conditioning has been extensively studied.
Prior work has considered goals either in the compact state space (Kaelbling, 1993; Andrychowicz et al., 2017; Fu et al., 2018; Gupta et al., 2019), or in high-dimensional image space, but from the same scene as the agent (Nair et al., 2018; Nair and Finn, 2019; Singh et al., 2019; Johns, 2021).
Compact state vectors are hard to interpret, depend on the scene, and require detailed state estimation.
Image based goals are human-interpretable, but requires another agent (typically a human) to perform the task ﬁrst to generate the image for conditioning.
In this work, we propose the use of off-domain images and/or language instructions to reduce the task speciﬁcation burden on users.
Our method can also be adapted to work with existing goal-conditioned learning algorithms.
Foundation Models and Applications
A major advancement in modern deep learning is the emergence of representation and transfer learning.
Models trained on generic datasets from the internet are capable of learning representations that transfer successfully to a plethora of downstream applications.
This observation has been widespread since the seminal works of AlexNet (Krizhevsky et al., 2012) and ResNet (He et al., 2016).
Recent advances in self-supervised learning have led to further advancements in visual (He et al., 2020), language (Devlin et al., 2019;
Radford et al., 2019), and multi-modal (Radford et al., 2021) representation models that have widely impacted downstream applications in vision (Tian et al., 2020; Conde and Turgutlu, 2021; Patashnik et al., 2021; Frans et al., 2021) and NLP (Lin et al., 2021;
Bugliarello et al., 2021; Izacard and Grave, 2020), and for this reason have been touted as “foundation models” (Bommasani et al., 2021).
However, the use of foundation models for control and embodied intelligence is relatively new and under-explored.
CLIPort (Shridhar et al., 2021) combines CLIP embedding with transporter network to learn language-conditioned robot manipulation policies from demonstrations.
The concurrent works of Khandelwal et al.
(2021) and Parisi et al.
(2022), study the use CLIP and other self-supervised representation networks as a perception module for control tasks and observe they outperform traditional ImageNet-pretrained backbones.
3. ZeST: Using Foundation Models for Task Speciﬁcation
As outlined in Section 2, current approaches to goal-conditioning are limited due to requirement of state-vector based goals or goal images from the same scene as agent.
In this work, we introduce ZeST, a framework for studying more generic and intuitive forms of goal-speciﬁcation like off-domain images (e.g. those found on the internet) and language instructions.
To succeed with this type of goal speciﬁcation, the agent must capture “common sense” that is grounded in humancentric experiences and understanding of the world.
Motivated by the advancement in foundation models (Bommasani et al., 2021) that were trained using massive amounts of data from the internet, which contain in part data generated by humans in human-centric environments, we speciﬁcally study if foundation models can enable these generic forms of goal-conditioning.
In this work, we also focus on zero-shot goal-speciﬁcation, where the agent must be capable of performing tasks when simply presented with a user-speciﬁed (off-domain) goal.
We do not assume any human annotations or supervised pairings between prior experiences of the agent and goal speciﬁcations, which is representative of real deployment scenarios.
Moreover, foundation models have demonstrated impressive zero-shot learning results (Radford et al., 2021; Brown et al., 2020).
At its core, the ZeST framework provides a way to measure the “similarity” between the image observation of the agent and a user-speciﬁed goal, with high similarity implying the goal speciﬁcation is satisﬁed.
The most similar image in the replay buffer of the agent can be used for goal-conditioning.
The similarity can also be used as a proxy for reward signal in case of policy learning.
While the approach of measuring similarity between agent’s observation and goal has been explored in prior work, for instance by using a learned classiﬁer (Pinto and Gupta, 2016; Fu et al., 2018;
Eysenbach et al., 2021), they have all required goal images from the same scene as the agent.
The overall ZeST framework is described in Figure 1, and consists of three main components: (1) embedding network; (2) feature selector; (3) feature distance function.
Each of the three modules can be instantiated according to properties and constraints of the downstream task.
For the embedding network , we use pre-trained foundation models for embedding both the goal speciﬁcation and agent observation.
While any foundation model can be used, in our experimental evaluation, we focus on ImageNet-supervised ResNet50 (He et al., 2016), ImageNet-trained
MoCo (He et al., 2020), and CLIP (Radford et al., 2021).
Since CLIP is a multi-modal embedding network, it also provides a mechanism for specifying tasks through language descriptions.
Details of the three candidate embedding networks can be found in the Appendix.
The feature selection module further featurizes the embedding.
In this work, we consider two simple featurizations: 1) raw features that use the raw embeddings themselves, and 2) delta features that use the difference in embeddings between the desired and initial conﬁgurations, and analogously for the agent, the difference in embeddings between current observation and initial observations.
The raw features are useful to directly compare the speciﬁed goal with the current observation.
With delta features, the user could specify both the goal state and an initial/current state in order to indicate a desired change of an object’s state.
For example, a user would provide the images of both a closed microwave and open microwave, which serves to better specify the task by describing the high-level action of opening the microwave.
For the similarity metric , we explore the use of cosine similarity and L2 distance, both of which have been used in prior work involving representation learning.
In summary, the similarity between a pair of observations ( X t , X 0 ) and goal-speciﬁcation ( G 0 , G f ) is written as:
[Equation]
where ψ ( · ) is the embedding function (foundation model) and α is the similarity metric (cosine similarity, L2 distance etc.).
Figure 2 illustrates an instantiation of ZeST with delta features.
We emphasize here that ZeST provides an abstraction to study generic forms of goal conditioning, and the use of various foundation models for the same.
While we explore a number of design choices as outlined above and in the Appendix, there are many additional possibilities that all conform to our broad framework.
An exhaustive study of all such combinations is beyond the scope of any single work, and we hope that our work inspires further work in the study of goal-conditioning and foundation models for embodied intelligence.
4. Experimental Tasks and Evaluation Metrics
In this section, we outline three different tasks that span both policy deployment and policy learning.
For each task, we also outline different variations and evaluation metrics.
4.1. Goal Selection Task
Given a dataset of experience D , off-domain goal-speciﬁcation ( G 0 , G f ) , and initial/context observation X 0 , the task for the agent is to select an observation X f ∈ D that satisﬁes the goalspeciﬁcation in context of X 0 .
The dataset of experience, D = { τ 1 , τ 2 , . . .
τ N } , in general consists of observation-action trajectories.
We remain agnostic to the source of dataset, which could have been obtained through teleoperation or autonomous execution various policies.
The task for the agent is to ﬁnd the image in the dataset that has maximum similarity to the goal-speciﬁcation in the context of initial image X 0 .
Let µ denote the number of observations in the dataset that satisfy the goal speciﬁcation.
We evaluate the performance according to two metrics:
Top-N Success Rate: We rank each observation in the dataset according to the similarity measure, and compute the fraction of top N ≤ µ ranked samples that satisﬁes the goal speci- ﬁcation.
This metric allows us to study if a high-similarity can be used to retrieve observations close to the goal, and ultimately enable downstream goal-conditioned policy execution.
Dataset Total Variation: While top-N success rate allows us to evaluate model prediction for high-similarity observations, it is agnostic to how the model behaves for low-similarity observations (i.e. non-goal states).
Towards obtaining a broad evaluation metric for similarity scores across the entire dataset, we consider the dataset total variation measure.
Let φ i denote the similarity score of observation X i , and let I ( X i ) be an indicator function to denote whether X i satisﬁes the goal speciﬁcation.
Then, the dataset total variation is given by:
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (
[Equation]
In our experiments, we average the results over multiple choices of X 0 and goal-speciﬁcation ( G 0 , G f ) to quantify the overall performance of the similarity measure.
4.2. High-Level Action Selection Task
The agent is presented with a dataset of experience D = { τ 1 , τ 2 , . . .
τ N } , where each trajectory consists of an observation-action sequence.
The agent is also presented with goal speciﬁcation ( G 0 , G f ) .
Let ∆ g
=
ψ ( G f ) − ψ ( G 0 ) be the goal-feature and ∆ a ki =
ψ
( I ki ) − ψ
( I k 0 ) be the highlevel action feature for timestep i in
trajectory k
( I k 0 is the initial image in trajectory k ).
The task for the agent is to ﬁnd a pair of images in the dataset that has maximum similarity to the high-level action (i.e. transformation) in ∆ g .
The action selection task is useful when a user need to specify an action but do not have access to images of the particular object of interest.
For example, they could use images of “opening a cabinet door” to specify “opening microwave”, since both share the same high-level action semantics of “opening”.
For this task, we identify semantically equivalent actions among trajectories and evaluate model performance with Top-N success rate.
4.3. Ofﬂine Reinforcement Learning
The task is speciﬁed by G 0 and G f , and a robot must manipulate an object in its own environment that matches the object in G 0 and G f .
A set of sub-optimal trajectories are provided for learning consistent with the ofﬂine RL setting.
Behavioral cloning directly learns a supervised model that maps state to action, and thus simply mimics the behavior in the dataset without any improvement.
We propose to evaluate ZeST by using an ofﬂine RL algorithm to learn the policy using the ZeST similarity scores as a proxy for the reward function.
If we observe an improvement over the behavior cloning baseline, then we may conclude that ZeST scores function as a reasonable reward proxy for policy learning.
5. Experimental Results and Discussion
To test the effectiveness of ZeST, we evaluate performance of various instantiations of ZeST for enabling policy execution and learning.
In our experiments, we consider the following design choices for components outlined in Section 3.
Embedding Foundation Models: ImageNet-supervised ResNet50 (He et al., 2016), ImageNettrained MoCo (He et al., 2020), and CLIP (Radford et al., 2021).
We consider these models primarily due to wide adoption in computer vision, with CLIP being a natural choice for encoding both visual and language goals due to its multi-modal training.
Features: Raw embedding and delta feature variants as speciﬁed in Section Feature
: Cosine similarity and L distance
. 5.1. How does ZeST perform in the Goal Selection Task? How do various design choices impact performance?
To evaluate the performance of ZeST, we ﬁrst consider an experience dataset in the Franka Kitchen domain (Gupta et al., 2019).
The dataset consists of ﬁve different manipulation tasks: turning the top/bottom burner knob on, opening the microwave, opening the hinge door, and opening the sliding door.
The experience dataset is generated by rolling out randomized expert policies that were trained using a policy gradient algorithm (Rajeswaran et al., 2017, 2018).
Uniform noise of varying levels were added to the actions to create a diverse dataset that contains both successful and failed trajectories.
We evaluate the performance of ZeST using different forms of goalspeciﬁcation: (1) same-scene images – i.e. goal images from the Franka Kitchen scene itself; (2) off-domain images from the internet; (3) hand-sketches and drawings; (4) and instructions in natural language.
The internet image and drawings dataset are part of the Visual Task Dataset (Cui, 2021), and Figure 4 provides a few representative examples from this dataset.
We ﬁrst evaluate the choice of features and feature distance function.
To do so, we compute the dataset total variation (Eq. 3), averaged across different embedding models.
The results are summarized in Figure 5.
We ﬁnd that when goals are speciﬁed through images, the use of delta features and cosine similarity leads to the best results.
In case of language goals, we ﬁnd that using delta features and L2 distance leads to the lowest total variation.
Based on these results, for the remainder of the paper, we primarily focus on the best performing design choices.
We consider two different variations for evaluation: (1) Narrow dataset (ND), in which the experience dataset contains trajectories related to only a single manipulation task.
For example, the narrow dataset would only contain (suboptimal) trajectories where the Franka robot attempts to manipulate the microwave.
(2) Diverse dataset (DD), which is the entire experience dataset comprising of a diverse set of trajectories related to all the ﬁve manipulation tasks.
The ND scenario is simpler and serves to evaluate if ZeST can perform goal-selection in the context of a single manipulation task.
This can also be useful for policy learning similar to prior works that utilize a goal classiﬁer as a reward function (Fu et al., 2018;
Eysenbach et al., 2021).
The DD scenario is harder and is also representative of goal-selection for policy execution in open-ended environments.
The experimental results are presented in Figures 6 and 7.
In terms of dataset total variation, we ﬁnd that CLIP has the lowest total variation among the three embedding models across all the goal-speciﬁcation modalities.
We hypothesize that this is due to CLIP being trained on a larger data corpus (400 million images) compared to ResNet50 and MoCo trained on ImageNet (15 million images).
CLIP has also shown promising zero-shot results in downstream tasks, and our observations in the goal selection tasks are consistent with these general trends.
We also observe that using same scene images leads to better results than using internet images or drawings, which is along expected lines due to smaller domain gap.
The Top-25 success rates for both the goal selection and high-level action selection tasks are plotted in Figure 7, along with the performance of a random selection mechanism, which can be used to judge the level of difﬁculty of the task.
We ﬁnd that ZeST with all the embedding models vastly outperform a random selection scheme suggesting that ZeST is capable of computing useful similarity scores.
We again observe that CLIP is the best embedding model, resulting in a near 14-fold improvement over a random selection baseline in the harder DD setting.
Further details about this experiment can be found in the Appendix.
5.2. How does ZeST perform in real-world video datasets?
In this experiment, we examine whether our proposed approach can generalize to real world video data and how different embedding models compare.
To this end, we adopt a challenging real-world video dataset, SomethingSomething-V2 (Goyal et al., 2017), that consists of short clips of everyday human actions.
These videos are recorded with hand-held devices and therefore do not have a stable view of the scene.
Both the motion and human hand in the view can cause occlusion.
We extract a subset of these video clips that all are opening something as the dataset for experimental evaluation.
In addition to real video frames, we further test how synthetic images from a simulated environment interact with real world data, to test cross-domain task speciﬁcation.
Using objects from 3D Warehouse dataset, we generate object trajectories in simulation that matches with the opening something tasks from the SomethingSomething-V2 dataset.
Similar to the evaluation in Franka kitchen domain, we consider narrow (ND) and diverse (DD) datasets for evaluation.
In real2real modality of goal-speciﬁcation, we use the frames from real trajectories to specify the goal while in the sim2real modality, we use rendered object frames to specify goal.
As shown in Figure 8, averaging over all three embedding models, ZeST instantiations are able to outperform random guessing.
Please see the Appendix for additional details and visualizations.
Figure 9 presents the results for individual embedding models in the real2real setting.
Interestingly, we observe that CLIP is only marginally better than random guessing while ResNet and MoCo are substantially better.
Overall, we observe that performing goal-selection and action-selection using real-world video datasets remains challenging.
An interesting direction for future work would involve the learning of foundation models using internet-scale video datasets, which we hypothesize would lead to improved results.
5.3. Can ZeST enable ofﬂine reinforcement learning?
In this experiment, we test how ZeST signals could be used to enable policy learning in ofﬂine
RL (Levine et al., 2020).
Ofﬂine RL is the setting where an agent is presented with a dataset of sub-optimal trajectories, and the agent must learn a competent policy using this dataset without additional environment interactions.
We study if the similarity computed by ZeST can be used as a proxy for the reward function in ofﬂine RL.
To do so, we use the Franka Kitchen domain outlined in Section 5.1, and use internet images for goal-speciﬁcation.
For our experimental evaluation, we compare: (1) behavior cloning (BC) which performs straightforward supervised learning to predic- tion actions in the dataset conditioned on observations; and (2) decision transformer (DT) (Chen et al., 2021) – a state of the art ofﬂine RL algorithm that performs reward conditioned behavior cloning using a sequence model.
This comparison involves two closely related algorithms with the major difference being reward conditioning.
In the case of DT, we use our ZeST similarity as the reward function.
Table 1 shows the learned policies’ average performance across task.
When goal images with high similarity are selected, DT achieves 90.77% of expert performance.
This suggests that ZeST similarities serve as a reasonable proxy, and has the potential to enable ofﬂine RL.
[Table]
6. Conclusion
In this work we introduce a framework for studying foundation models for zero-shot task speciﬁcation (ZeST).
We evaluate the effectiveness of ZeST for enabling zero-shot policy execution through a set of goal selection tasks; and we evaluate ZeST for enabling policy learning in ofﬂine reinforcement learning.
Leveraging existing pre-trained models, we demonstrate the effectiveness of ZeST on goal and action selection tasks in several different domains with different task speciﬁcation modalities.
We ﬁnd that ZeST is quite effective in zero-shot goal-selection and results in a 14-fold increase in performance over a random guessing baseline.
In ofﬂine RL, we ﬁnd that using
ZeST scores as a proxy for the reward function enables the learning of policies that perform better than a behavior cloning baseline.
Our results show the potential of foundation models and their applications in closing the domain gap for robot learning between real world and simulation.
Instantiations of ZeST in this work are limited in several aspects.
The embedding model can be sensitive to occlusions, which often requires removing the robot arm from the camera view.
Existing computer vision techniques for object segmentation and in-painting can be leveraged to address this problem for real applications.
At the same time, our proposed approach is limited to tasks that have salient visual feature changes from unsuccessful states to successful states and such visual feature is not positional/location-based.
However, explicit localization modules can be developed to address such issue and we leave this for future investigation.
Table of Contents
A Details of Foundation Models under Test
B Further Analysis of ZeST and Additional Design Considerations
C Qualitative Evaluations of ZeST on Real Image Trajectories
D Details of Goal Selection Experiments in Franka Kitchen Domain
E Details of Goal Selection Experiments in SSV2 Dataset
F Goal Similarity Scores for Multi-step Tasks
G T-REX Training
Appendix A. Details of Foundation Models under Test
Vision-based robot learning enables training end-to-end robot policies that directly work with visual inputs.
However, real-world robot manipulation data are expensive and often cannot generate data at the scale required to train vision models.
Therefore, existing vision-based robot learning methods often rely on pre-trained vision models and ﬁne-tune with limited data.
Deep residual network (ResNet), proposed by He et al.
(2016), greatly increased the depth of neural networks through using residual connections.
ResNet has been used as the backbone for many state-of-the-art vision models.
Pre-trained ResNet on image classiﬁcation task on ImageNet (Deng et al., 2009) has been a popular and very effective embedding model for extracting visual features for down-stream robotic manipulation tasks (Gupta et al., 2018; Mandlekar et al., 2020; Shah and Kumar, 2021).
MoCo (He et al., 2020) is a recent self-supervised pre-training technique, i.e. pre-train without using the labels, and has shown better performance for downstream vision tasks than supervised pre-training with image classiﬁcation.
Moco employs a contrastive loss, for some query q and a set of encoded samples { k 0 , k 1 , ...
k K } (where k + is the desired key for query q ):
[Equation]
CLIP (Radford et al., 2021) is a multi-modal pre-training technique that leverages both images and text descriptions to learn embeddings of both modality in the same space.
The training objective of
CLIP is to maximize the cosine similarity between matching pairs of encoded image
i k and text t k while minimizing the similarities between unmatched pairs:
[Equation]
CLIP is trained with over 400 million pairs of internet image and text and has shown state-of-the-art zero-shot performance on image classiﬁcation tasks.
Self-supervision models such as Moco and
CLIP have found wide applicability in computer vision as backbone representations.
However, they have not been widely used in RL, and our work explores this possibility.
Appendix B. Further Analysis of ZeST and Additional Design Considerations Viewpoints
To mitigate potential viewpoint differences between the observations and goal speciﬁcation images, it is desired to use multi-viewpoint observations and take the average similarity score across different viewpoints.
As shown in Figure 10, we observe that the cosine similarity for same trajectories of opening a microwave differ from different viewpoints.
However, this may not be possible in certain real world applications and can be mitigated by ﬁltering or reweighing goal speciﬁcations using similarities with embeddings of the robot’s observations.
Ensemble of task speciﬁcations
To reduce potential noise in the computed signals, it is desired to use an ensemble of goal speciﬁcation images.
When multiple goal and initial images are provided, for each goal image, we will ﬁnd an initial image that is the most similar to the goal image’s embedding using cosine similarity measure and then compute the delta feature between the two.
With this ensemble of images, the average of all cosine similarity scores is used as the ﬁnal prediction.
We ﬁnd the use of an ensemble of goal speciﬁcations help to reduce noise and improve the smoothness of the predicted signal.
We further investigate how the performance of one instantiation of ZeSTS change based on the number of goal samples used in the dataset.
We evaluated with goal selection task in the Franka Kitchen domain using internet images as the goal speciﬁcation modality.
Each task has a total of 50 goal samples (for both the initial and goal state).
We sample a desired number of goal samples from the dataset and repeat for 5 times.
The average performance with standard errors is plotted in Figure 11.
In general, more samples would improve the performance of the model, but the initial number of samples necessary to achieve a certain performance varies from task to task.
Pixel differences and Cos+Delta scores
A natural question to ask when considering whether cosine similarities between delta features are meaningful is that “Does pixel differences between image frames correlate with high cosine similarities?”.
Figure 12 shows that large pixel differences do not correlate with high similarity scores.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
clip
[Table]
[Figure]
[Table]
resnet
[Table]
clip
[Table]
drawings resnet
[Table]
Appendix C. Qualitative Evaluations of ZeST on Real Image Trajectories
In this section we present qualitative evaluation of Cos+Delta with all three embedding models for real world object manipulation trajectories.
Sample task speciﬁcation images used, robot trajectory frames, and the corresponding normalized cosine similarity score are plotted.
Figure 15,16,13,14,17, and 18 shows the evaluation for tasks of opening fridge top door, opening fridge bottom door, opening a microwave, opening a cabinet, opening an oven, and opening a drawer respectively.
As shown in these ﬁgures, ZeST signals are sensitive to occlusions caused by the robot manipulator as well as different croppings of the trajectory images.
Appendix D. Details of Goal Selection Experiments in Franka Kitchen Domain
SGSD : As shown in Table 2, CLIP outperform other embedding models with image-based task speciﬁcation modalities using Cos+Delta . 2.
DGSD : As shown in Table 3, CLIP does not outperform the other two embedding models.
[Figure]
[Table] [Table]
[Table]
Moco 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
CLIP 0.00% 24.00% 0.00% 33.33% 33.33% 0.00% 15.11% 3*online imgs
ResNet 6.00% 0.00% 11.56% 0.00% 0.00% 0.00% 2.93%
Moco 0.00% 32.89% 0.00% 0.00% 14.00% 0.00% 7.81%
CLIP 12.00% 33.33% 0.00% 0.00% 0.00% 0.00% 7.56% 3*drawings
ResNet 0.00% 0.00% 0.00% 0.00% 0.00% 0.00% 0.00%
Moco 0.00% 33.33% 0.00% 0.00% 0.00% 0.00% 5.56%
CLIP 18.22% 0.00% 30.67% 0.00% 32.89% 0.00% 13.63% text
CLIP 0.00% 0.00% 2.67% 33.33% 0.00% 0.00% 6.00%
[Table] [Table]
SGDD : As shown in Table 4, CLIP outperforms the other two models under 2 out of the 3 image-based task speciﬁcation modalities in the goal selection task in the Franka kitchen domain (using Cos+Delta ).
DGDD :
As shown in Table 5, CLIP outperforms the other two models in this hardest setting as well.
However, the overall performance is not very good.
ASDD (Action Selection): As shown in Table 6, CLIP outperforms the other two embedding models in this scenario.
[Table] [Table]
[Table] [Table]
Appendix E. Details of Goal Selection Experiments in SSV2 Dataset
For this experiment, we test Cos+Delta with three different embedding models.
We conducted each experiment with 10 different repetitions, each time sampling a different trajectory for specifying the task.
The results for the ﬁve different task scenarios SGSD,DGSD,SGDD,DGDD, and ASDD are shown in Table 7,8,9,10 and 11 respectively.
As shown, the performance of CLIP is much lower than that of ResNet and Moco with SSV2 dataset.
Appendix F. Goal Similarity Scores for Multi-step Tasks
Many realworld tasks are multi-step and consist of different numbers of sub-goals.
In this experiment, we evaluate how our proposed method work for multi-step tasks and investigate whether specifying sub-goals improves its performance.
More speciﬁcally, we study the task of table assembly with real world video data from the IKEA furniture assembly dataset (Ben-Shabat et al., 2020) and synthetic visual trajectories in the IKEA assembly environment (Lee et al., 2021).
Sample trajectories from the dataset and the simulator are shown in Figure 19.
This setting corresponds to when an agent is learning from human demonstrations in the form of video recordings and it has access to past experience data in simulation.
Figure 20 shows the consine similarities to each subgoal for different simulated trajectories.
The average similarity across time increases from the ﬁrst subgoal to the ﬁnal goal since the ﬁnal goal contains the subgoals.
However, the similarity score pattern is not clear enough for telling apart at which point a subgoal is achieved.
A more complex method is needed to ﬁnd subgoals in these trajectories.
[Table]
Appendix G. T-REX Training
We also employ a ranking-based reward learning method, T-REX (Brown et al., 2019), to learn a reward function.
We train T-REX with trajectories ranked by aggregated cosine similarities.
We then test on classifying sub-trajectories (relative ranking) with ground-truth dense reward rankings.
Table 12 shows the maximum test accuracy under each model for different tasks.
Figure 21 shows the relationship between predicted returns and ground truth returns on both seen and unseen trajectories.
The return predicted by T-REX models correlate well with ground truth return for four out of the ﬁve tasks.
We observe that the predicted rewards correlate well with ground-truth rewards.
We then use the learned rewards to train policy gradient agents.
Figure 22 shows the learned policies’ performance with different methods under each task.
The return is normalized to the expert’s performance as 1.0.
T-REX training is able to generate policies that outperform behavioral cloning in 2 out of the 5 tasks, while T-REX does not need the action labels from the demonstrations.