Leveraging Language for Accelerated Learning of Tool Manipulation
Abstract: Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools.
We investigate whether linguistic information about a tool ( e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task.
We obtain diverse descriptions of various tools in natural language and use pre-trained language models to generate their feature representations.
We then perform language-conditioned meta-learning to learn policies that can efﬁciently adapt to new tools given their corresponding text descriptions.
Our results demonstrate that combining linguistic information and meta-learning signiﬁcantly accelerates tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.
[Figure]
Phase 1: Meta-training with tool descriptions
[Figure]
Introduction
The ability to quickly learn how to use a broad range of new tools is considered one of the hallmarks of human intelligence [1].
Humans are able to attain a degree of proﬁciency with a new tool ( e.g., a new hand tool or a virtual tool such as a joystick or remote) within just a few minutes of interaction [2].
This rapid learning relies on the ability to understand or discover the affordances [3] of a new tool, i.e., the ability to perform a certain action with the tool in a given environment.
For example, a hammer affords the opportunity to grasp it, use it to hammer a nail, or use it to push another object.
In contrast, a spatula affords the opportunity to ﬂip a pancake or sweep food ingredients into a bowl.
Endowing robots with the ability to quickly discover and exploit affordances of a new tool in order to learn how to perform a given task has been a long-standing grand challenge in robotics [4].
While humans routinely rely on geometric priors and visual observations for tool manipulation, they can also ‘read’ text manuals or linguistic descriptions to understand affordances of new tools and quickly adapt to using them.
In this work, we investigate whether language can help robots accelerate the process of learning to use a new tool.
In particular, consider the following descriptions of two types of tools:
The shape of tongs is typically that of a V, with two long, thin handles that come to a point at the top, and a gripping area in the middle.
A spatula is a kitchen utensil for ﬂipping over food while cooking.
The head of the spatula is usually rectangular or oval in shape.
The handle of the spatula is usually long and thin.
Our key intuition is that such natural language descriptions of tools contain information about the affordances of the tools, how to exploit these affordances for a given task, and how perceptual features of tools ( e.g., their visual appearance and geometry) relate to their affordances.
Moreover, language can help capture the shared structure of tools and their affordances.
Thus, if one has previously learned to use a set of tools (with corresponding language descriptions), a description of a new tool can help to exploit this prior knowledge in order to accelerate learning.
Statement of Contributions.
Our primary contribution is to propose ATLA — A ccelerated Learning of T ool Manipulation with LA nguage — a meta-learning framework leveraging large language models (LLMs) to accelerate learning of tool manipulation skills.
The overall approach is illustrated in Fig.
We propose to use LLMs in two distinct ways: to generate the language descriptions for tools and to obtain the corresponding feature representations.
At meta-training time, the metalearner updates a base-learner that quickly ﬁne-tunes a manipulation policy; this ﬁne-tuning process is conditioned on the LLM representations corresponding to the language descriptions of each tool.
Speciﬁcally, we propose a simple gradient-based meta-learning setup based on Reptile [5] that performs off-policy updates.
At test time, the base-learner adapts to a new tool using its language descriptions and interactions with it.
To our knowledge, our approach is the ﬁrst to utilize LLMs to accelerate learning of new tools.
We demonstrate the beneﬁts of using language in a diverse set of tool-use tasks including pushing, lifting, sweeping, and hammering.
Related Work
Tool Manipulation.
Tool manipulation [6, 7, 8, 9, 10] is one of the long-standing problems in robotics research.
A major challenge is understanding the affordances of the tool in different tasks.
Previous work has modeled and learned affordances from parameterized keypoints on the tools [7, 10], from human demonstrations [8], and from spatial-temporal parsed graphs of the tools [11].
Our work instead leverages natural language ( e.g., describing affordances of the tools in words) for generalization of affordances in tool manipulation and is compatible with previous approaches.
Language-informed Control.
Natural language has been applied to enable efﬁcient robotic learning through (1) generating primitive language instructions for producing control actions ( i.e., instruction following task)
[12, 13, 14, 15, 16, 17, 18, 19], (2) learning language-informed reward functions for training control policies [20, 21, 22, 23, 24, 25], and (3) using language to correct or adapt the behavior of the robot [26, 27].
However, these works primarily translate natural language into action policies for a speciﬁc task with the text providing information on the desired actions that optimize returns ( e.g., “ push the door ”).
This means that the text is tightly coupled with the task seen during training, making it difﬁcult to generalize to a new distribution of tasks with different dynamics.
In contrast, the text in our work only provides a high-level description of the property of each tool, encouraging the agent to extract useful information to generalize to a new task.
Some prior work [28, 29, 30, 31] also uses language descriptions of environment dynamics to enable generalization of policies but does not leverage meta-learning.
Meta-learning.
Our work uses the framework of meta-learning [32, 33, 34, 35, 36], in which the agent is trained with a distribution of tasks, and later adapts quickly to a previously unseen task.
“Reptile” is proposed in [5] as a simple ﬁrst-order, gradient-based meta-learning algorithm that learns an initialization of the neural network’s parameters for fast adaptation at test time.
Recent papers [37, 38, 39, 40] have also explored providing additional context information ( e.g., the property of the task) to encode task-speciﬁc knowledge for a meta-learning agent.
However, all these works directly provide the context information either through scalar signals or a learned task embedding, which require domain expertise or a pre-training stage.
In this work, we assume that the agent is provided with a text description of the tool, which is more accessible and easier to collect.
Problem Formulation
We consider the following goal: given a new tool and corresponding language description(s), we aim to quickly learn a policy for using the tool to achieve a given task.
We pose this problem in a meta-learning setting in which a policy is trained with a distribution of tools for a given task, and later adapts quickly to a previously unseen tool sampled from this distribution in the same task.
Meta-training.
During meta-training, we assume access to a set T = { τ i } Ki =1 of tools, where K is the number of available tools.
For each tool τ i , we are also provided a set of corresponding language descriptions L i = { l ij } N i j =1 , l ij ∈ L , where N i is the number of available descriptions for tool τ i and
L is the set of possible textual descriptions.
In addition, each l ij can describe a different property of tools such as shape and common use.
Given a particular robotic manipulator and a particular task ( e.g., pushing, lifting, sweeping, or hammering), each tool τ induces a partially-observable Markov decision process (POMDP): (cid:104)S τ , A , O , P τ , R τ (cid:105) .
Here, S τ is the state of the entire environment ( i.e., combined state of the robot, tool, and potentially other objects to be manipulated using the tool).
The robot’s action space A ( e.g., corresponding to robot joint torques) and observation space ( e.g., the space of RGB-D observations from a camera) are ﬁxed across tools.
The transition probabilities are given by P τ : S τ ×
A × S τ → [0 , 1] , and the reward function is R : S τ ×
A × S τ → [0 , 1] .
During meta-training, our goal is to learn a policy π θ : O × L → A parameterized by θ ( e.g., weights of a neural network) that can be quickly ﬁne-tuned at test time.
Meta-testing.
At test time, we are provided a new tool τ ν and corresponding language descriptions
L ν = { l νj } N ν j =1 , l νj ∈ L .
We aim to let the meta-learned policy quickly adapt to this new tool in a ﬁxed number of interactions with the tool in order to maximize the expected cumulative reward.
This is a challenging task since the new tool can be quite different in terms of visual appearance and affordances as compared to previously seen tools in meta-training.
Approach
The key idea behind our approach is to collect and embed language information of the environment into meta-learning , allowing the policy to adapt faster and better to unseen environments.
4.1 Collecting Language Information Using Pre-Trained Large Language Models
A common use of language in robotics is to use it to provide an instruction to the robot ( e.g., “ pick up the green block on the table ”).
Such instructions are typically speciﬁed by humans manually through crowd-sourcing, which can be labor intensive.
In our setting, we consider language as additional information about the environment ( e.g., “ the hammer has a long handle and large head at the top ”).
The language here is not used to describe the goal ( e.g., what to do), but to provide information about properties of the environment ( e.g., tool shape).
This makes the text here taskagnostic , forcing the agent to learn generalizable policies.
To obtain a diverse set of language descriptions, we are inspired by the recent advances in LLMs that are trained with vast amounts of online data and imbued with rich, semantic knowledge of different objects.
We propose using
LLMs to provide language descriptions of the tools in the form of question answering .
Speciﬁcally, we provide the GPT-3 [41] model with the following template prompt through the OpenAI API: “ Describe the [feature] of [name] in a detailed and scientiﬁc response: ” where “feature” is selected from one of [“shape”, “geometry”] or one of [“common use”, “purpose”] and “name” describes the tool ( e.g., “ a hammer ”, “ a pair of tongs ”).
We ﬁnd that adding “ detailed and scientiﬁc ” to the prompt signiﬁcantly improves the quality of the texts generated.
For each tool, we generate 10 different paragraphs of descriptions for each of the four features, and then combine paragraphs in each of the four permutations of the features (“shape” and “common use”, etc).
Each tool τ i is thus paired with a diverse set of 800 language descriptions
L i (see Appendix A1 for more examples).
Each description l ij ∼ L i is approximately 2-4 sentences long.
4.2 Obtaining Feature Representations from Large Language Models
With the collected language descriptions, we now incorporate them into policy training.
One common choice is to train a language module ( e.g., long short-term memory (LSTM) [42]) from scratch “Describe the [ ] of [ ] in a detailed and scientific response” [Figure] name
Prompt GPT-3 via OpenAI API Pre-trained BERT model Gradient via critic objective Gradient via actor objective name: “a hammer”: “shape”, “geometry”, “common use”, “purpose” feature Language head
Image encoder
Actor head
Critic head
Cat ...
Fixed
Learned “A hammer is a hand tool that consists of a handle attached to a head, usually made of metal.
The head is usually in the shape of a rectangle with a flat surface on one side and a cylindrical surface on the other, which is used for striking nails.”
to embed features of the language input, which can take substantial time and effort to tune.
Instead, we use a pre-trained LLM to distill the language descriptions into feature representations.
Since
LLMs are trained with vast amounts of data, they can better interpret and generalize to the diverse set of long descriptions.
Since the GPT-3 model is not publicly available, we opt for the Google BERTBase [43] model on HuggingFace, which has 110.1M parameters and outputs a 768-dimensional vector representation for each description input.
T-SNE analysis shown in Fig.
3 demonstrates that, without any ﬁne-tuning, the model already captures semantic meanings of the descriptions among tools ( e.g., hammer and mallet are close to each other).
Fig.
2 shows the overall neural network architecture.
We ﬁrst prompt GPT-3 to obtain a set L i of text descriptions for tool τ i via the procedure in Sec.
During meta-training, we randomly sample l ij from L i for each episode to ensure that the policy sees a diverse set of descriptions.
We then freeze the BERT model during policy training.
The output from BERT is fed into a single fullyconnected layer with ReLU (language head, θ l ).
The image observations (possibly from two camera angles–one from overhead and one from the wrist) are passed through convolutional layers (image encoder, θ o ), whose output is then concatenated with that from the language head.
The actor head ( θ a ) and critic head ( θ c ) then output the action a t and the corresponding value for the Q function Q ( o t , a t ) .
See Appendix A3 for more details of the neural network setup.
4.3 Meta-training and Testing Language-Conditioned Manipulation Policies
We hypothesize that additional language information of the tools promotes generalization.
However, zero-shot transfer to unseen tools can be difﬁcult given the distinct geometries and affordances.
Thus we perform meta-training for explicitly training the policy to adapt to distinct tools within the same task.
Our algorithm (shown in Algorithm 1) is based on Reptile [5], a simple ﬁrst-order gradientbased meta-learning algorithm, but with an additional sampling strategy to prevent overﬁtting.
At each iteration of the meta-training, one tool τ is sampled from the training set T .
At the base level (line 5), we run the current policy with the language description l ij sampled from L i .
We then add the collected experiences in a base replay buffer β base , and perform B iterations of off-policy updates using Soft Actor Critic (SAC) [44] in order to obtain the ﬁnal policy parameters θ (cid:48)
= [ θ (cid:48) l , θ (cid:48) o , θ (cid:48) a , θ (cid:48) c ] .
Then at the meta level (line 12), the network is updated with a gradient step towards θ (cid:48) :
[Equation] where α is the meta-learning rate and θ is the collection of the old policy parameters θ l , θ o , θ a , θ c .
We also highlight the following remarks including differences to the Reptile algorithm:
• To reduce variance, the actor and critic share the parameters of language head θ l and the image encoder θ o , and both modules are updated only with the critic objective.
• To prevent overﬁtting to language descriptions of one tool during training, experiences collected from all tools are saved in a meta replay buffer β meta (line 14), and β base for each tool is initialized with random samples from β meta (line 3).
During policy update at base training, 30% of the experiences are sampled from β meta .
During test time, we do not use any experiences from β meta . We demonstrate the effectiveness in Sec. 6.
• In applications of Reptile in supervised learning, the meta update is often averaged over N sampled environments: θ new ← θ + α (cid:80) Ni ( θ (cid:48)
i − θ ) .
However, we ﬁnd that N = 1 trains faster and also matches the test objective of adapting to a single tool.
N > 1 does not offer better performance at test time.
See Appendix A6 for sensitivity analysis.
• Performing only a single meta update after B = 5 iterations of base update can be inefﬁcient.
Instead of using a large learning rate α which causes unstable training, we perform M = 2 meta updates for each adaptation to one tool but collect experiences to β base only at the
ﬁrst update
Algorithm 2 shows the procedure of adaption at test time.
First, the model is provided with a target test tool τ ν and a set of language descriptions.
With the adaptation budget B ν , we run the policy with the language description l j sampled from L ν .
The collected experiences are stored in the buffer β base and used to update the policy parameters.
Require: T = { τ i } Ki =1 : training set of tools; { L i } Ki =1 : sets of language descriptions;
θ l , θ o , θ a , θ c : policy modules; β meta : meta replay buffer.
13: end
Add β meta 15: end while Algorithm
Require: τ ν : test tool; L ν : set of language descriptions; θ l , θ o , θ a , θ c : policy modules; β base ← ∅ : base replay buffer for b = 1 to B ν do
Experiment Setup
Through different tool manipulation tasks in simulation, we aim to investigate the following questions: (1) Does language information help achieve better adaptation to new tools?
(2) Does metalearning improve adaptation to new tools?
(3) How does the choice of pre-trained LLMs affect policy training?
(4) Does language information help the policy utilize tools’ affordances effectively?
Tasks.
Four different tool manipulation tasks are implemented (see bottom of Fig.
1): (1) pushing: pushing the tool to a ﬁxed location on the table; (2) lifting: reaching and lifting the tool up from the table to some target height; (3) sweeping: using the tool to sweep a cylinder to a ﬁxed location on the table; (4) hammering: using the tool to hammer a peg further into a hole in a block.
Solving these tasks beneﬁts from an understanding of the geometric affordances of the tools such as the grasp location.
See App.
A2 for more details of the task setup including the reward functions.
Robot.
We build custom simulation environments with a 7-DOF Franka Panda arm in the PyBullet simulator [45].
We use RGB cameras with 128 × 128 image outputs, placed at different off-arm locations and at the arm wrist depending on the needs of the tasks.
For all tasks, we use 4-DOF cartesian velocity (3D translation and yaw) as the action output from the policy.
The arm joints are then commanded with a jacobian-based velocity controller at 5Hz.
The policy does not command the gripper; instead, we use the heuristic that once the gripper is below some height, the gripper closes to grasp the object.
If the grasp fails, the gripper re-opens if it rises above the threshold.
Tools.
We collect a total of 36 objects (See App.
A1 for the full list) from open sources.
Most of the objects are common tools such as a hammer and an axe.
Some of them are less used as tools but have distinct geometry and affordances, such as a banana whose inner curvature may help push other objects.
We split the objects into a training set of 27 and a test set of 9 — we try to separate objects with similar geometry or affordances ( e.g., hammer and mallet) into different sets.
Baselines.
We compare ATLA (ours) with the following (Fig. 4): (a) AT-TinyLA (ours): ATLA with a smaller BERT encoder (BERT-Tiny [43] with 4.4 million parameters and 128-dimensional output).
(b) AT : ATLA without
language information. (c) AT-XL : ATLA
without language information but
Metric.
For all experiments we save the model checkpoint with the highest running-average reward on the training dataset.
After training, for each test tool we load the checkpoint and run a ﬁxed number of iterations of adaptation.
In Fig.
we report the highest reward at adaptation, averaged over 3 seeds for each test tool.
See Appendix A6 for reward in numbers for each tool and task
Results
[Figure]
Q1: Does language information help achieve better adaptation to new tools?
Fig.
4 shows that
ATLA and AT-TinyLA perform better than AT and AT-XL among most tools in the 4 tasks.
The differences are more signiﬁcant in sweeping and hammering, which are more difﬁcult and language information can better inform the affordances of the tools.
Among the 9 tools, ATLA and ATTinyLA always perform better with crowbar, plier, hammer, scissors, faucet, and trowel.
ATLA and AT-TinyLA do not perform better mostly when low reward is achieved for all baselines for that tool, such as wineglass in pushing and lifting and paint roller in sweeping.
Fig.
5 also shows that language helps faster learning in ATLA compared to AT in most cases, with the agent achieving higher rewards with fewer episodes of adaptation.
We also ﬁnd that different tools learn better for different tasks.
For example, hammer is better in sweeping than plier probably due to its long bar, but worse in pushing also due to the small inertia along the long bar causing instability during pushing.
As ATLA performs better among most tools, language information can provide useful information about the tool affordances in different tasks.
Q2: Does meta-learning improve adaptation to new tools?
Fig.
4 shows that with or without language information (ATLA / AT-TinyLA vs.
SAC-LA, or AT / AT-XL vs.
SAC), meta-learning improves ﬁnal performance after adaptation.
Without meta-learning, SAC-LA shows smaller improvement over SAC ( e.g., plier, hammer, scissors, and trowel).
This demonstrates that language information particularly helps when combined with meta-learning.
With
Without
[Figure]
Fig.
5 also compares the adaptation curves between
ATLA and SAC-LA: those of SAC-LA tend to stagnate or ﬂuctuate while those of ATLA tend to rise steadily.
This indicates meta-learning trains the policy to better adapt to new tools after training.
In Appendix A6 we also perform sensitivity analysis on the number of inner adaptation during meta-training, B , which highlights the effectiveness of performing a few steps of inner adaptation.
Ablation: meta replay buffer.
We investigate the effect of re-using experiences from other tools (saved in β meta during adaptation).
For this, we run ATLA without β meta on the pushing task.
Across the 9 test tools, the post-adaptation reward is mostly lower ( − 20% , − 10% , − 13% , − 18% , +5% , − 25% , − 18% , − 18% , − 5% ) compared to ATLA with β meta .
Note that the effect is more prominent when the reward difference between ATLA and AT is larger ( e.g., 20% with crowbar and 25% with scissors), indicating that language information is more effective if β meta is applied.
We also ﬁnd using β meta accelerates the meta-learning process (Fig. 6) — demonstrating that sharing experiences among tools makes training more efﬁcient.
Q3:
How does the choice of pre-trained LLMs affect policy training?
Fig.
4 shows that ATLA usually attains higher post-adaptation reward than AT-TinyLA, which uses a smaller pre-trained
BERT model, indicating that the policies beneﬁt from the richer representation of the language descriptions that the bigger BERT model offers.
Q4: Does language information help the policy utilize tools’ affordances effectively?
The results above have shown that language descriptions of the geometric features and common use of the tools help policies adapt to new tools for a given task.
Fig.
7(a) visualizes the effect in the example of using a crowbar for sweeping.
Language descriptions of a crowbar often contain phrases including “long and thin bar”, “curved”, “hook”, “used to leverage”, and “used to pry open things”.
With the descriptions, ATLA (orange curve in Fig. 7(a)) enables the policy to adapt quickly to this tool unseen during meta-training — the policy learns to use the curved hook to better steer the cylinder towards the target.
As a comparison, we replace the descriptions with only the sentence “A crowbar is a long and thin bar,” and the policy (green curve in Fig. 7(a)) does not adapt as well.
One common feature among tools is the handle.
Language descriptions of a trowel includes phrases like “ﬂat, triangular blade”, “handle to be grasped”, and “used for scooping”.
While ATLA learns to grasp at the handle (Fig. 7(b) top), when we remove “handle” from all the descriptions, the robot fails to grip ﬁrmly on the handle and loses the grip eventually (Fig. 7(b) bottom).
Limitations.
In some cases, we observe that the policies can still fail to correctly utilize the tool affordances with language information.
In the sweeping task, the paint roller is the only tool that
ATLA fails to perform the best with.
Fig.
8 (top) shows the grasp learned by the policy.
It failed to use the bigger opening between the roller and the handle on the other side to sweep the cylinder.
Fig.
8 (bottom) shows that in the hammering task, the policy fails to use the faucet, speciﬁcally, its relatively ﬂat head to push towards the nail.
However, these affordances can be sensitive to the initial pose of the tool and can be difﬁcult to explore.
This also highlights one of the limitations of our work.
We use a relatively simple task policy setup for tool manipulation tasks, which is directly mapping image inputs to Cartesian velocity commands.
This creates challenges in exploration and learning the skills even with language information.
One remedy is to combine with approaches like keypoint-based methods [7] that inject additional domain knowledge into the policy.
In addition, we use a relatively small dataset of tools, which may limit the potential of using language information.
It would be particularly interesting to model revolute joints of the tools and perform more complex tasks such as picking up objects with a pair of tongs.
Furthermore, our current evaluation does not consider real robot experiments, which is not the focus of this work.
Conclusion
In this work, we investigate using large language models (LLMs) to accelerate adaptation of policies to new tools in tool manipulation tasks.
We use LLMs to both (1) generate diverse language descriptions of the tool geometry and common use, and (2) obtain vector representations of the descriptions.
We then propose language-conditioned meta-learning that trains policies to quickly adapt to new tools.
The results demonstrate that combining language information and meta-learning signiﬁcantly improves the performance when adapting to unseen tools.
Acknowledgments
The authors were partially supported by the Toyota Research Institute (TRI), the NSF CAREER Award [ # 2044149], the Ofﬁce of Naval Research [N00014-21-1-2803, N00014-18-1-2873] and the School of Engineering and Applied Science at Princeton University through the generosity of
William Addy ’82.
This article solely reﬂects the opinions and conclusions of its authors and not
NSF, ONR, Princeton SEAS, TRI or any other Toyota entity.
Appendix A1 Tool Information
Table A1 shows additional information about the tools used in the paper, including a sample view of the object mesh, sample language descriptions
[Table]
and the train-test split.
[Table]
One common use of a paint roller is to apply paint evenly to a surface such as walls or ceilings.
A paint roller consists of
[Table]
[Table]
is a hand-held tool with two arms that meet at a point.
A tweezer is a small hand-held tool with two pointed jaws that are used to pick up small objects or to remove unwanted hair or debris from the body.
[Table]
A whisk typically has a long, thin handle with a series of loops at the end.
The loops are usually made of metal and are arranged in a spiral pattern.
A whisk is a com- mon kitchen utensil that is used to mix ingredients together or to incorporate air into a mixture.
[Table]
A needlenose plier has a long, tapered nose with a small jaw, and is used for gripping and bending wire.
Needlenose pliers are a type of plier that has a long, slender nose and is used for gripping small objects and for working in tight spaces. NO
[Table]
A plier is a hand tool used for gripping objects.
It con- sists of a pair of metal jaws with teeth that open and close when the handles are moved.
Plier typically has a long, narrow neck and a tapered head that becomes progressively thinner as it extends from the neck to the tip.
Yes
[Table]
A gooseneck plier is a type of plier that has a long, narrow neck and a slightly curved head.
The neck allows the plier to reach into tight spaces, and the curved head provides extra leverage.
A gooseneck plier is commonly used to grip and bend small objects.
No
[Table]
The shape of a plier is typically long and skinny with a grip at the end.
Plier is a hand tool used for various purposes such as gripping, bending and cutting.
No
[Table]
No
[Table]
with several
[Table]
a
[Table]
a
[Table]
shaped container with a
[Table]
extending
[Table]
or
[Table]
end
[Table]
A knife typically has a sharp, narrow blade with a pointed tip.
A knife is a common kitchen utensil used for cutting and slicing food.
[Table]
A spatula is a kitchen utensil that is used to turn or lift food that is being cooked.
It has a ﬂat, usually slightly convex, blade that is attached to a handle.
A spatula is commonly used to mix, spread, and ﬂip food items.
[Table]
A pair of scissors is a cutting tool that consists of two metal blades that are connected at a pivot point.
A pair of scissors typically has two blades that are joined at a pivot point.
[Table]
A wrench is a tool that is used to apply torque to an object in order to loosen or tighten it.
A wrench is typically long and slender with a small, metal handle.
[Table]
The geometry of a screwdriver can be described as a cylindrical shape with a pointed end.
A screwdriver is a tool that is used to insert and remove screws.
[Table]
A clamp is a mechanical device that is used to temporar- ily hold two or more objects together.
The geometry of a clamp is typically that of a rectangular or U-shaped object with two handles.
[Table]
The shape of a wok is a deep, round bowl with sloping sides.
A wok is a concave-shaped cooking utensil that is most commonly used in Chinese cuisine.
No
[Table]
A pickaxe is used to break up rocks and other materials.
A pickaxe is a tool that has a handle attached to a head.
[Table]
A faucet is typically a small, thin, spout-like ﬁxture that protrudes from a wall or sink.
A faucet is a valve used to release water from a plumbing ﬁxture, such as a sink or bathtub.
[Table]
A dustpan is a tool used for sweeping up dust and small debris from ﬂoors and other surfaces.
It consists of a small, shallow pan with a handle attached to one side.
A dustpan is a concave scoop with a ﬂat bottom and ﬂared sides.
No
[Table]
A
[Table]
is a tool used to transfer liquids from one con- tainer to another.
A ladle typically has a long, curved handle and a large, deep, spoon-like bowl.
[Table] A pair of
[Table]
has a thin, curved metal shaft with two ﬂat metal paddles at the end.
A pair of tongs is a device used to grip and hold objects.
[Table]
A gavel is a small hammer that is [Table] strike a sound block, typically made of wood.
A gavel is a mallet used to strike a block of wood, typically used by a presiding ofﬁcer or auctioneer to maintain order or to signal the start and end of an auction.
[Table]
The purpose of a squeegee is to remove water or other liquid from a surface.
A squeegee is a rod-shaped tool with a ﬂat, blunt edge, and a small handle.
[Table]
A powerdrill is typically cylindrical in shape, with a handle attached to one side and a chuck on the other side for holding drill bits.
A power drill is a tool that is used to create holes in various materials, or to fasten screws or bolts.
[Table]
A wineglass is a glass with a small bowl and a long stem.
They are used to serve wine and are often used in restaurants.
A wineglass is typically shaped with a long, thin stem and a bowl that is larger at the bottom than the top.
[Table]
A shoehorn is a curved, rod-shaped object used to assist in putting on shoes.
A shoehorn is a curved or stepped tool designed to help slide a shoe onto the foot.
[Table]
A horseshoe is a U-shaped metal bar that is nailed to the hooves of a horse.
A horseshoe is typically U-shaped, with two large curves and two smaller curves at either end.
No
[Table]
Fig.
3 shows the t-SNE analysis of the BERT embeddings of all the tools.
First we use PCA to project the 768-dimensional embeddings to 50-dimensional, and then perform t-SNE to project them to 2-dimensional for visualization.
A2 Task Information
Table.
A2 shows the episode length, reward function, and action space of the tasks.
We ﬁnd the policy can explore well in pushing and lifting tasks with relatively simple reward functions; in sweeping and hammering task, we tune the reward function carefully to guide the arm towards the cylinder/nail.
Fig.
A1 shows the camera observations for the four tasks.
We use a single view for the pushing task as it is sufﬁcient for the task, and dual views for other tasks.
A wrist view is used in the lifting task.
Fig.
A2 visualizes the workspace of the tasks including the initial position of the tools and the target.
For the hammering task, we set the lateral and torsional friction coefﬁcient of the nail to be high (and 0 . 1 ) in the simulator.
We also make the gripper ﬁngers longer to prevent the gripper hitting the block when attempting to hammer the nail.
Please see the included video for more visualization of the tasks.
[Figure]
Reward function
Action space
Pushing max(0 , 1 − distance-tool-target [ − 0 .
15] m/s in x [ − 0 .
1] m/s in y [ − π/ 4 ,π/ 4] rad /s in yaw
Lifting 0 .
1 ∗ max(0 , 1 − distance-EE-tool )
+ 0 .
5 ∗ max(0 , 1 − distance-tool-target ) [ − 0 .
1] m/s in x , y , and z [ − π/ 4 ,π/ 4] rad /s in yaw
Sweeping 0 .
1 ∗ max(0 , 1 − distance-EE-tool )
+ 0 .
1 ∗ max(0 , 1 − distance-tool-cylinder )
+ 0 .
5 ∗ max(0 , 1 − distance-cylinder-target ) [ − 0 .
2] m/s in x , y , and z [ − π/ 4 ,π/ 4] rad /s in yaw
Hammering 0 .
1 ∗ max(0 , 1 − distance-EE-tool )
+ 0 .
1 ∗ max(0 , 1 − distance-tool-nail )
+ 0 .
5 ∗ max(0 , 1 − distance-nail-hole end )
[ − 0 .
2] m/s in x , y , and z [ − π/ 4 ,π/ 4] rad /s in yaw
[Figure]
[Figure]
A3 Model Architecture
For all policies, the image encoder π o contains three convolutional layers for either view of the image input; the three layers have kernel size [7 × 7 , 5 × 5 , 3 × 3] , stride size [4 , 3 , 2] , no padding, and channel size [4 , 8 , 16] .
The language head π l contains a single fully-connected layers with 128-dimensional output.
Both the actor head π a and critic head π c have two hidden layers of size 128 (except for AT-XL with hidden size 256).
All convolutional layers and fully-connected layers are followed with a ReLU activation.
The ﬁrst layer in π a and π c are additionally normalized by
Layernorm [46].
A4 Multi-environment Training Algorithm
Algorithm 3 below shows the procedures of the SAC and SAC-LA baselines.
Multi-environment training does not involve any inner adaptation ( B = 0 ).
In the context of multi-environment training,
N means the number of environments where, experiences that each gradient update uses, belong.
N = ∞ means experiences can come from any environment.
Algorithm 3 SAC, SAC-LA
Require: T = { τ i } Ki =1 : training set of tools; { L i } Ki =1 : sets of language descriptions (for SAC-LA only); θ l (for SAC-LA only) , θ o , θ a , θ c : policy modules; β : replay buffer; N 1: while training do 2:
Sample τ i from T ; reset environment with τ i 3:
Collect episode (with l ij ∼ L i for SAC-LA); add to β 4: if N = ∞ then 5:
Sample experiences from β that are from any environment 6: else 7:
Sample experiences from β that are from N different environments 8: end if 9:
Update θ o , θ l (only for SAC-LA) , θ c , θ a with sampled experiences 10: end while
A5 Training Hyper-parameters
The hyper-parameters used for meta-learning (shared among AT-LA, AT-TinyLA, AT, AT-XL) and multi-environment learning (shared between SAC-LA, SAC) are outlined in Table A3.
We ensure meta-learning and multi-environment learning sample the same amount of transitions from environments. [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
A6 Results and Additional Studies
Table A4 below shows the results in Fig.
4 in numbers.
[Table]
Effect of B in meta-learning.
The value of B determines the number of inner adaptation for each tool during meta-training.
To evaluate the effect of B , we perform sensitivity analysis varying B in the sweeping task, and the results are shown in Table A5 (values for B = 5 are from Table A4)
[Table]
With B = 1 , the post-adaptation performance is generally worse than B >
[Table]
and it is close to the performance of multi-environment training (SAC-LA in Table A4).
Higher value of B helps
[Table]
but there is no signiﬁcant performance gain with B > 2 .
This highlights the importance of performing a few steps of inner adaptation at training time for better adaptation to new tools.
[Table]
Effect of N in meta-learning.
The value of N determines how many environments the meta update gradient is averaged over in Eq.
To evaluate the effect of N , we perform sensitivity analysis varying N in the hammering task, and the results are shown in Table A6 (values for N =are from Table A4).
Although typical Reptile-style meta learning in supervised learning uses N >[5], the results here do not show the improvement with N > 1 .
N = 1 also matches our setup of adapting to a single tool at test time.
We also show the meta training reward curves in Fig.
A3.
The ﬁnal training rewards are similar for N = 1 , 2 , 5 , but N = 1 trains slightly faster.
Surprisingly we also ﬁnd N > 1 exhibits larger variances in reward at later iterations, which is contrary to the idea that N > 1 stabilizes training.
Thus we use N = 1 in our main experiments.
[Table]
SAC-LA with N = 1 .
Typically multi-environment training uses gradient update averaged over a batch of experiences sampled from any environment ( N = ∞ ); we perform the same setup in our main experiments.
It is also worth investigating N = 1 , meaning each gradient update uses experiences from only one environment, since we ﬁnd N = 1 works for the meta learning setup as shown above.
In Table A4 we show the results for all four tasks (SAC-LA-N=1).
In Pushing,
Lifting, and Sweeping tasks, the results are similar to SAC-LA with N = ∞ .
However, in the hammering task, the results are improved from SAC-LA across all tools, although they are still worse than meta-learning baselines.
It is possible that N = 1 mitigates training instability of multienvironment training.
We use N = ∞ in our main experiments since it is a common setup for multi-environment training.