Learning Gradient Fields for Object Rearrangement
Anonymous Author(s)
AffiliationAddress email
Abstract
Object Rearrangement is to move objects from an initial state to a goal state. Here,we focus on a more practical setting in object rearrangement, i e ., rearrangingobjects from shuffled layouts to a normative target distribution without explicitgoal specification.
However, it remains challenging for AI agents, as it is hardto describe the target distribution (goal state) for reward engineering or collectexpert trajectories as demonstrations.
Hence, it is infeasible to directly employreinforcement learning or imitation learning algorithms to address the task.
Thispaper aims to search for a policy only with a set of examples from a target distribution instead of a handcrafted reward function.
We employ the score-matchingobjective to train a target gradient field (TarGF) , indicating a direction on eachobject to increase the likelihood of the target distribution.
For object rearrangement,the TarGF can be used in two ways: 1) For model-based planning, we can castthe target gradient into a reference control and output actions with a distributedpath planner; 2) For model-free reinforcement learning, the TarGF is not onlyused for estimating the delta likelihood as a reward but also provides suggestedactions in residual policy learning.
Experimental results in ball rearrangement androom rearrangement demonstrate that our method significantly outperforms thestate-of-the-art methods in the quality of the terminal state, the efficiency of thecontrol process, and scalability.
Introduction
[Figure]
❌
✅ ✅
[Figure]
Given a set of target examples, the agents need to move objects to meet a normative distribution.
Object rearrangement [1, 2] is to manipulate a setof objects from an initial layout to a normativedistribution, as shown in Fig.
This task is takenfor granted by humans [3] and widely exists in ourdaily life, such as tidying up a table [4], placingfurniture [5], and sorting parcels [6].
However, compared with conventional robottasks [7, 8, 9, 10], there are two critical challengesin rearrangement tasks: 1) Hard to clearly definethe target state (goal), as the target distribution isdiverse and the patterns are difficult to describe inthe program, e.g ., how to evaluate the cleannessquantitatively?
As a result, it is difficult to manually design a reward/objective function, whichis essential to modern control methods, e.g ., deepreinforcement learning (DRL) [11].
In the physical world, the agents also need to find an executable path to reach the target state safely and efficiently,i.e ., finding a short yet collision-free path to reach the target distribution.
To this end, the agents arerequired to jointly learn to evaluate the quality of the state (reward) and take actions to move objects.
In other words, the agents should answer the two questions: What kind of layouts does the user prefer ,and How to manipulate objects to meet the target distribution ?
Thus, it is necessary to find a way to build an agent without the dependence on the explicit goalstate and shaped reward.
Imitation learning (IL) can implicitly learn the reward function from experttrajectories [12, 13, 14].
However, collecting a large number of expert trajectories is expensive.
Recently, example-based reinforcement learning (RL) [15, 16] has tried to reduce the reliance onexpert trajectories, i.e ., learning a policy only with examples of successful outcomes.
However, itis still difficult to learn a classifier to provide accurate reward signals for learning in the case ofhigh-dimensional state and action space, as there are many objects to control in object rearrangements.
In this work, we separate the task into two stages: 1) learning a score function to qualify the state,and 2) learning/building a policy to rearrange objects with the score function.
Inspired by the recentadvances in score-based generative models [17, 18, 19, 20, 21], we employ the score-matchingobjective to learn a target gradient field (TarGF) by perturbing the well-arranged examples.
Thetarget gradient field can be viewed as object-wise guidance that provides a pseudo velocity on eachobject, pointing to regions with a higher density of the target distribution, e.g ., cleaner configurations.
To manipulate objects in the physical world, we further demonstrate two ways to leverage the targetgradient field in control.
We integrate the guidance with a model-based path planner to rearrangeobjects without collision.
We further derive the target gradient to a delta log-likelihood as a rewardfunction, key ingredients to reinforcement learning.
With the TarGF-based reward and a TarGF-guidedresidual policy network [22], we can train an effective RL-based policy from interactions.
In the experiments, we evaluate the effectiveness of our methods on two typical rearrangementtasks: Ball Rearrangement and Room Rearrangement .
First, we introduce three metrics to analysethe quality of the terminal state and the control process.
In Ball Rearrangement , we demonstratethat both of our control methods can efficiently reach a target distribution with high diversity andrealism.
We also observe that our methods can find a path with fewer collisions and fewer steps toconverge compared with learning-based baselines [13, 16] and planning-based baselines [23].
In Room Rearrangement , we further evaluate the feasibility of our methods in complex scenes wherethere are multiple heterogeneous objects.
Our contributions are summarised as follows:
• We introduce a score-based method to learn a target gradient field (TarGF) by perturbingthe state of well-arranged target examples.
• We point out two usages of the target gradient field in object rearrangement: 1) providingguidance in moving direction; 2) deriving a reward for learning.• We conduct experiments to demonstrate that both traditional planning and RL methods
can benefit from the target gradient field and significantly outperform the state-of-the-art methods in sample diversity, realism and safety.
Related Works2.1 Object and Scene (Re-)Arrangements
The object arrangement has been previously studied by researchers from the robotics community [24,2, 25] and the graphics community [26, 27, 28, 29, 30].
However, most of them focus on manuallydesigning rules/ energy functions to find a goal [24, 2, 25] or synthesising a scene configuration [27,28, 29, 30] that satisfies the preference of the user.
The most recent work [4] tries to learn a GNN tooutput an arrangement tailored to human preferences.
However, they neglect the physical constraintsduring the process.
Hence, the accessibility and transition cost from the initial and target states are notguaranteed.
Considering the physical interaction, the recent works on scene rearrangement [31, 32]mitigate the goal specification problem by providing a goal scene to the agent, making the rewardshaping feasible.
Concurrent works [33, 34] also notice the necessity of automatic goal inference fortying rooms and exploit the commonsense knowledge from Large Language Model (LLM) or memexgraph to infer rearrangements goals when the goal is unspecified. Another concurrent work [35] aimsat discovering generalisable spatial goal representations via graph-based active reward learning for2D object rearrangement. In this paper, we focus on a more fundamental problem in rearrangement:how to manipulate objects to maximise the similarity (i.e., the target likelihood) between the currentstate and the example sets . Here, only a set of positive target examples are required to learn thegradient field, rather than the prior knowledge about specific scenarios.
2.2 Learning without Reward Engineering
Without reward, previous works explore algorithms to learn a control policy from expert demonstrations via imitation learning [36, 37, 38, 12] or inverse reinforcement learning [39, 14, 40, 41].
Imitation learning (IL) [36, 37, 38] aims to directly optimise the policy by cloning the behaviourfrom the expert trajectories.
The inverse reinforcement learning (IRL) [39, 14, 40, 41] tries to learna reward function from data for reinforcement learning.
Considering the difficulty in collectingexpert trajectories, there are some example-based methods proposed to learn policy only with a setof successful examples [15, 39, 13, 16].
The most recent work is RCE [16] which directly learnsa recursive classifier from successful outcomes and interactions for policy optimisation.
We alsotry to find an example-based control method for rearrangement with a set of examples.
Our methodcan learn the target gradient fields from the examples, without any additional efforts in rewardengineering.
The target gradient fields can provide meaningful reward signals and action guidancefor reinforcement learning and traditional planners.
To the best of our knowledge, such usage ofgradients fields is not explored in previous works.
Problem Statement
We aim to learn a policy that moves objects to reach states close to a target distribution without relyingon reward engineering.
Similar to the example-based control [16], the grounded target distributionp tar ( s ) is unknown to the agent, and the agent is only given a set of target examples S ∗ =
{ s ∗ }where s ∗ ∼ p tar ( s ) .
In practice, these examples can be provided by the user without accessingthe dynamics of the robot.
The agent starts from an initial state s 0 ∼ p 0 ( s 0 ) , where N objects arerandomly placed in the environment.
At time step t , the agent takes action π ( a t | s t ) imposed on theobjects with dynamics p ( s t +1 | s t , a t ) .
Here, the goal of the agent is to search for a policy π ∗ ( a t | s t )that maximises the discounted log-target-likelihood of the future states:
[Equation]
where γ ∈ (0 , 1] denotes the discount factor.
Notably, we assume that p tar ( s ) > 0 everywhere,since we can perturb the original data with a tiny noise ( e.g ., N (0 , 0 . 00001) ) to ensure the perturbeddensity is positive everywhere.
[17] also used this trick to tackle manifold hypothesis issue.
This objective reveals three challenges: 1) Inaccessibility problem: The grounded target distributionp tar in Eq.
1 is inaccessible.
Thus, we need to learn a function to approximate the log-targetlikelihood for policy search.
2) Sparsity problem: The log-target-likelihood is sparse in the statespace, as mentioned in [17].
Hence, even if we have access to the target distribution p tar , it is stilldifficult to explore the high-density region due to the Manifold Hypothesis mentioned in [17]. 3)
Adaptation problem: The dynamics p ( s t +1 | s t , a t ) is also inaccessible.
To maximise the cumulativesum in Eq.
1, the agent is required to adapt to the dynamics to efficiently increase the target likelihood.
To address these problems, we partition the task into 1) learning to estimate the log-target-likelihood(reward) of the state and 2) learning/building a policy to rearrange objects to adapt to the dynamics.
Method
4.1 Motivation
We argue that estimating the gradients of the log-target-likelihood ∇ s log p tar ( s ) can tackle the firsttwo difficulties mentioned in Sec.
For the inaccessibility problem, we can approximate the rewardincrement between two adjacent states by the first-order Taylor expansion using the gradient fieldlog p tar ( s t +1 ) − log p tar ( s t )
≈ ⟨∇ s log p tar ( s t ) , s t +1 − s t ⟩ .
As shown in Sec.
4.4, this helps toderive a surrogate objective of Eq.
For the sparsity problem, we can leverage the gradient field to help with exploration since the gradient∇ s log p tar ( s ) indicates the direction that points to the higher density region of the target distributionp tar .
To estimate the gradients of the log-target-likelihood ∇ s log p tar ( s ) , noted as the target gradientfield (TarGF) Φ tar , we adopt the deep score-matching technique [18] that has achieved impressiveresults in many research areas recently[19, 21, 20]. 𝑎 𝑡𝑔
Env
Policy Network 𝜋
[Figure]
𝑂𝑅𝐶𝐴Planner 𝜙 𝑠 𝑡+
[Figure]
∗ Perturbed
[Figure]
∗ Perturbation
[Figure]
(0,𝜎 2 𝑰)
[Figure]
Matching ℒ2(𝛿,𝑔) (b) Control Objects with TarGF (a) Learning Target Gradient Field (TarGF)
𝑠
𝑡 𝑎 𝑡 = 𝜋 𝜃 𝑠 𝑡 ,𝒈 𝒕 +𝒢(𝒈 𝒕 ) 𝑎 𝑡
RL
[Figure]
Control [Figure] Planning
[Figure]
෩𝑠 ∗ 𝛿 = [Figure] ∗ [Figure]
[Figure] [Figure] + To address the adaptation problem, we further demonstrate two ways to incorporate the trained targetgradient field Φ tar with control algorithms for object rearrangement: 1) For the model-based setting,our framework casts the target gradient into a reference control and outputs actions with a distributedpath planner.
For the model-free setting (reinforcement learning), we leverage the target gradientto estimate reward and provide suggested actions in residual policy learning [22].
In the followings, we will first introduce how to train the target gradient field Φ tar via score-matchingin Sec.
and then introduce our rearrangement framework under the model-based and model-freesettings in Sec.
4.3 and Sec.
4.4 respectively.
4.2 Learning the Target Gradient Field from
Examples
The target score network Φ tar : S → S is the core module of our framework, which aims atestimating the score function ( i.e .the gradient of log-density) of the target distribution ∇ s log p tar ( s ) .
To train the target score network, we adopt the Denoising Score-Matching (DSM) objective proposedby [42], which can guarantee a reasonable estimation of the ∇ s log p tar ( s ) .
In training, we prespecify a noise distribution q σ ( (cid:101) s | s ) on the collected target state s .
Then, we match the output of thetarget gradient field with the score of the perturbed target distribution q σ ( (cid:101) s ) = (cid:82) q σ ( (cid:101) s | s ) p tar
[Equation]
s
[Equation]
d s :
[Equation]
The DSM objective guarantees the optimal score network satisfies Φ ∗ tar ( s ) = ∇ s q σ ( s ) almost surely.
When σ is small enough, we have ∇ s q σ ( s )
≈ ∇
s log p tar ( s ) , so that Φ ∗ tar ( s )
≈ ∇
s log p tar ( s ) .
In practice, we adopt a variant of DSM proposed by [18], which conducts DSM under different noisescales simultaneously.
We implement the target score network as a graph-based network Φ tar ( s , t )that is conditioned on a noise level t ∈ (0 , 1] .
The input state s =
[ s 1 , s 2 , ..., s N ] is constructed asa fully-connected graph that takes each object’s state( e.g .position, orientation) s
i ∈ R f s and staticproperties ( e.g .category, bounding-box) p i ∈
R f p as the node feature [ s i , p i ] ∈
R f s + f p .
The inputgraph is pre-processed by linear initial feature extraction layers and then passed through two EdgeConvolution layers.
After message-passing, the output on each node serves as the component of thetarget gradient on the i-th object’s state Φ itar ( s ) ∈ R f s .
We defer structure details to supplementary.
[Figure]
4.3 Model-based Planning with the Target Gradient Field
[Figure]
The target gradient g =
Φ tar ( s ) can be translated into a gradient-based action a g = G ( g ) , where 167 G denotes the gradient-to-action translation.
A gradient over the state space can be viewed asvelocities imposed on each object.
For instance, if the state of each object is a 2-dimensional positions i =
[ x, y ] , then the target gradient component on the i-th object can be viewed as linear velocitieson two axes Φ tar ( s )
i =
[ v x , v y ] .
When the action space is also velocities, we can construct a g bysimply projecting the target gradient a g = G ( g ) =
v max
|| g
|| 2 · g into the action space, where the v maxdenotes the speed limit.
Following the gradient-based action, objects will move towards directionsthat may increase the likelihood of the next state, which meets the need of the rearrangement task.
However, the gradient-based action is imperfect ( e.g .leading to collisions between the objects), as thetarget gradient has no access to the information of environment dynamics.
This may be inefficientand unsafe due to the severe collisions between objects.
To adapt the gradient-based action to the environment dynamics, we incorporate the target gradientfield with a model-based off-the-shelf collision-avoidance algorithm ORCA .
The ORCA planner ϕ θtakes the gradient-based action a g as reference velocities and then outputs the collision-free velocitiesa =
ϕ θ ( a g ) with the least modification to the reference velocities.
In this way, the adapted action canlead objects to move towards a higher likelihood region efficiently and safely.
For more details of the
ORCA planner, we defer to supplementaries.
4.4 Learning Policy with the Target Gradient Field
[Figure] We further propose a framework based on the model-free reinforcement learning (RL) algorithm toaccomplish the rearrangement tasks in the model-free scenario, where the agent needs to adapt thedynamics via interacting with the environment.
To tackle the inaccessibility problem mentioned in Sec. 3, we first derive an equivalent J ∗ ( π ) ofthe original objective in Eq.
1 and further obtain a surrogate objective ˆ J ( π ) to approximate thisequivalent:
[Equation]
Further, we derive a surrogate objective ˆ J ( π ) to approximate the equivalent objective J ∗ ( π ) .
Wenotice that in most physical simulated tasks, the distance between two adjacent states || s t − s t −
|| 2is quite small, which inspires us to approximate the delta log-likelihood using the first order Taylorexpansion:
[Equation]
In this way, we can optimise the surrogate objective ˆ J ( π ) by assigning (cid:80)
[Equation]
as the immediate reward.
In practice, we simply assign the last term of the cumulative summation asthe immediate reward r t = ⟨ Φ tar ( s t ) , s t − s t − 1 ⟩ , and this version is shown to be the most effective.
To tackle the sparsity problem mentioned in Sec. 3, our idea is to build our policy π θ upon thegradient-based action a gt mentioned in Sec.
4.3, as a gt is the best direction to increase the likelihood: a
[Equation]
= [Equation] t [Equation]
+
[Equation]
targetgradient
[Equation]
This approach is similar to the residual policy learning proposed by [22].
The a gt serves as aninitial policy, and the agent’s policy network π θ takes the target gradient g t and the current states t as input and outputs π ( s t , g t ) to ‘polish’ the a gt .
The agent finally outputs ‘polished’ actiona t =
π ( s t , g t ) + a gt .
In this way, the agent can not only benefit from the efficient exploration butalso have less training burden as it ‘stands on the giant’s shoulder’
( i.e .the gradient-based action a gt ).
We implement the multi-agent Soft-Actor-Critic(SAC) as the reinforcement learning algorithmbackbone.
Each object is regarded as an agent, and each agent can communicate with all the otheragents.
At each time step t , the reward for the i-th agent is r t =
r tart + λ ∗ r
it , where r tart =(cid:80) 1 ≤ k ≤ t ⟨ Φ tar ( s t ) , s k − s k − 1 ⟩ is a centralised target likelihood reward, λ > 0 is a hyperparameter andr it =
(cid:80)
j ̸
= i
c i,j is a decentralised collision-aware reward where c i,j is a collision penalty.
Specifically,when a collision is detected between object i and j ,
c i,j = − 1 , otherwise c i,j = 0 .
Experiment Setups5.1 Environment
We design two object rearrangement tasks for evaluation: Ball Rearrangement and Room Rearrangement , where the former uses controlled environments for numerical analysis and the latter uses realdata to demonstrate the potential of our method on a more challenging task.
Ball Rearrangement includes three environments with increasing difficulty of rearrangement:
Circling , Clustering , and the hybrid of the first two, Circling + Clustering .
There are 21 balls withthe same geometry bounded in a square in each environment.
In Clustering and Circling+Clustering ,the balls are divided equally into three colours.
The goal of each environment is as follows:
Circlingrequires all balls to form into a circle.
The centre of the circle can be anywhere in the environment;
Clustering requires all balls to form into three clusters by colour.
The joint centre of each cluster hastwo choices; Circling + Clustering requires all balls to form into a circle where balls of the sametype are adjacent.
To slightly increase the complexity of the physical dynamics, we enlarge the lateralfriction coefficient of each ball and make each ball with different physical properties, e.g ., frictioncoefficient and mass.
The target examples of each task are sampled from an explicit process.
Thus,we can define a ‘pseudo likelihood’ on a given state according to the sampling program to measurethe similarity between the state and the target distribution.
Room Rearrangement is built on a more realistic dataset, 3D-FRONT [43].
After cleaning the data,we get a dataset with 839 rooms, 756 for training( e.g .target examples), and 83 for testing.
In eachepisode, we sample a room from the test dataset and shuffle the room with a Brownian Process at thebeginning to get an initial state.
The agent can assign velocities for each object in the room at eachtime step.
The agent has to learn implicit knowledge learned from the target examples to tidy up theroom.
More details are in the supplementary.
[Figure]
5.2 Evaluation
For both baseline comparison and ablation studies, we collect a fixed number of trajectories T
= { τ i }starting from the same initial states for different methods.
Then we calculate the following metricsusing the collected trajectories, where the PL is only reported in ball sorting tasks:
Pseudo-Likelihood (PL) measures the efficiency of the rearrangement process and the sample qualityof the rearrangement results.
We specify a pseudo-likelihood function F proxy for each environmentthat measures the similarity between a given state and the target examples.
For each time step t , the
PL is defined as F proxy ( s t ) .
We report the mean-variance curve of PL with respect to the time step t ,where the mean and variance are calculated over the trajectories T .
We do not report PL on roomrearrangement as it is hard to program the human preferences.
Coverage Score (CS): measures the diversity and fidelity of the rearrangement results ( i.e .the terminalstates S T = { s T } of the trajectories) by calculating the Minimal-Matching-Distance(MMD)
[44]between S T and a fixed set of examples S gt from p tar :
[Equation]
Averaged Collision Number (ACN) reflects the efficiency of the rearrangement process since objectcollision will lead to object blocking and deceleration.
ACN is calculated as (cid:80) τ ∈ T (cid:80) s t ∈ τ c t , where 248 c t denotes the total collision number at time step t .
More details are in the supplementary.
5.3 Baselines
We compare our framework with planning-based baselines under the model-based setting and withlearning-based baselines under the model-free setting.
Besides, we design heuristic baselines servingas the upper bounds for ball rearrangement tasks.
Leaning-based Baselines: GAIL: A classical inverse RL method that trains a discriminator asthe reward function.
RCE:
The SOTA example-based reinforcement learning method.
SQIL: Acompetitive example-based imitation learning method.
Goal-RL: An ‘open-loop’ method that firstgenerates a goal at the beginning of the episode via a VAE trained from the target examples and thentrained to reach the goal using goal-conditioned RL.
Planning-based Baselines: Goal-ORCA
: This method first generates a goal via the same VAE as in
Goal-RL .
Then the agent assigns an action pointing to the goal as reference control and adjusts thepreference velocity using ORCA, similar to Sec. 4.3.
Heuristic-based Baseline: Oracle:
We slightly perturb the samples from the target distribution andtake these samples as the rearrangement results.
[Figure]
PL
Circling
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
O u r s ( O
R C A )
G o a l ( O
R C A )
O u r s w / o
R L
O u r s ( S A
C )
G
[Table]
a l ( S A
C )
[Table]
u r
[Table]
w / o
[Table]
i d u
[Table]
l
[Table]
I
[Table]
A
[Table]
[Figure]
CS
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
R L
G A I L ( S A
[Table] [Table]
s (
[Table] [Table]
l (
[Table]
A
[Table] [Table]
l (
[Table]
R
[Table]
A )
[Table]
r s ( S A
C )
[Table]
w / o
R L
G A I L ( S A
C ) S Q I L
O u r s w / o
R e s i
d u a l R C E
O u r s ( S A
C )
G A I L ( S A
C )
G o a l ( S A
C )
R C E
S Q I L
Room
[Table]
O u r s w / o
R e s i
d u a l
O u r s ( O
R C A )
G o a l ( S A
C )
G o a l ( O
R C A )
O u r s ( S A
C )
G A I L ( S A
C )
O u r s w / o
R L R C E S Q I L 0.
ACN
O u r s ( O
R C A )
O u r s w / o
R e s i
d u a l
O u r s ( S A
C )
G o a l ( S A
C )
G o a l ( O
R C A )
O u r s w / o
R L
G A I L ( S A
C ) R C E S Q I L
O u r s ( O
R C A )
O u r s w / o
R e s i
d u a l
O u r s ( S A
C )
G o a l ( O
R C A )
G o a l ( S A
C )
O u r s w / o
R L
G A I L ( S A
C ) S Q I L R C E
O u r s ( S A
C )
G o a l ( S A
C )
G A I L ( S A
C )
S Q I L
R C E
Result Analysis
6.1 Baseline Comparison
Ball Rearrangement: We present the quantitative results on ball rearrangement tasks in the firstthree columns of Fig.
As seen from the top row of Fig.
5, the PL curves of our model-freeframework(Ours-SAC) and model-based framework (Ours-ORCA) are significantly higher andsteeper than the baselines in all three tasks, which shows the effectiveness of our method in arrangingobjects towards the target distribution efficiently.
In the middle row of Fig.
5, Ours-ORCA achievesthe best performance across all three tasks, while Goal-SAC outperforms Ours-SAC on Clusteringand Circling+Clustering since the goal is proposed by a generative model that can naturally generatediverse and realistic goals.
From the qualitative results shown in Fig.
3, our methods also significantlyoutperform the baselines in realism.
Our methods also achieve the lowest ACN in most cases, exceptthat Goal-SAC is slightly better than Ours-SAC in Circling .
The Goal-SAC is the strongest baseline yet loses to our framework in efficiency since the generatedgoal may be far away from the initial state.
As shown in Fig.
3, the Goal-SAC and Goal-ORCAcan rearrange objects in the right trend, but the objects may block each other’s way.
Besides, in thesupplementary, we show the VAE used in Goal-SAC is not guaranteed to generate a legal state( e.g .theballs overlap in goal).
Therefore, the balls may have unexpected behaviour during the process.
The classifier-based reward learning method, GAIL, RCE, and SQIL, basically fails in most tasks,as shown in Fig.
3 and Fig.
This is due to the over-exploitation problem that commonly exists inclassifier-based reward learning methods.
We defer the experiments to supplementary.
Room Rearrangement: The dynamics of the room rearrangement task are too complex to applyexplicit planning-based methods, so we only compare our model-free framework with learning-basedbaselines.
As shown in Fig.
4 and the last column of Fig.
5, our method significantly outperformsother baselines in sample realism of rearrangement results and the safety of the rearrangement process,which proves our method can also work on a more complex task.
6.2 Ablation Studies and Analysis
We conduct ablation studies on ball rearrangement tasks to investigate: 1) The effectiveness of theexploration guidance provided from the target gradient field and 2)
The necessity of combining thetarget gradient field with control algorithms to adapt to the environment dynamics.
To this end, weconstruct an ablated version named Ours w/o Residual that drops the residual policy learning andanother one named Ours w/o RL that only takes gradient-based action a g as policy.
Ours w/o Residual: The CS-bar in Fig.
5 shows that Ours significantly outperforms the Ours w/o
Residual in CS, which means the agent is easy to over-exploit few modes with high density.
We alsoshow qualitative results in supplementary that without the residual policy learning, the objects areusually arranged into a single pattern.
Ours w/o RL: The PL-curve in Fig.
5 shows that without the control algorithm serving as thebackbone, the rearrangement efficiency of Ours w/o RL is significantly lower than Ours.
Meanwhile,the ACN of Ours w/o RL is significantly larger than Ours, which indicates the efficiency drop may bedue to the collisions caused by the gradient-based action.
[Figure]
RCESQIL
Generalisation: We further evaluate [Figure] the generalisation of our methods todifferent numbers of objects.
To bespecific, we train all the policies inthe ball environment with 3 × 7 = 21balls.
During the testing phase, wedirectly transfer each policy to environments with increased ball number,i.e., 3 × 8 , 3 × 9 , and 3 × 10 , withoutany fine-tuning.
As shown in Fig.
6.2,we report the averaged PL incrementfrom initial to target states of each policy in each configuration.
The results indicate that our framework generalises well to different numbers of balls with the least performance drop compared withbaselines in all three ball rearrangement tasks.
Conclusion
In this study, we first analyse the challenge of object rearrangement and then incorporate a gradientfield method with either model-based planning or model-free learning approaches to achieve objectrearrangement tasks.
Experiments demonstrate our effectiveness by comparison with learning-basedand planning-based baselines.
Limitation and Future Work.
This work only considers the 2-D state-based rearrangement withsimplified dynamics that the objects can directly control their own velocities.
Future works mayextend our framework to more realistic scenarios: 1) Visual Observation: We may explore howto conduct efficient and accurate score-based reward estimation from image-based examples andleverage the recent progress on visual state representation learning [45, 46, 47] to develop the visualbased gradient-based action.
Complex Dynamics: When the agent can only move one object at atime, we may explore hierarchical methods where high-level policy chooses which object to moveand low-level policy leverage the gradient-based action for motion planning.
When we can only applyforce to objects instead of velocity, we may explore a hierarchical policy where a high-level policyoutputs expected velocities and a low-level employ a velocity controller, such as PID controller, tominimises the velocity errors.
Ethics Statement.
Our method has the potential to build home-assistant robots.
We evaluate ourmethod in simulated environments, which may introduce data bias.
However, similar studies alsohave such general concerns.
We do not see any possible
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] see Sec.
(b) Did you describe the limitations of your work?
[Yes] see Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] seesupplementary(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes] see Sec.
7 and supplementary
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] please see thesupplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] more details in the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] more details in the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes] more details in the supplemental
material.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable
[Table]