Learning Gradient Fields for Object Arrangement
Anonymous Author(s)
AffiliationAddress email
Abstract
Object Arrangement is to move objects from shuffled layouts to a normative targetdistribution, e.g. , tidy rooms.
However, it remains challenging for AI agents, asit is hard to describe the target distribution (goal state) for reward engineering orcollect expert trajectories as demonstrations.
Hence, it is infeasible to directlyemploy reinforcement learning or imitation learning algorithms to address thetask.
This paper aims to search for a policy only with a set of examples from atarget distribution instead of a handcrafted reward function.
We employ the scorematching objective to train a target gradient field (TarGF) , indicating a directionon each object to increase the likelihood of the target distribution.
For objectarrangement, the TarGF can be used in two ways: 1) For model-based planning,we can cast the target gradient into a reference control and output actions with adistributed path planner; 2) For model-free reinforcement learning, the TarGF is notonly used for estimating the delta likelihood as a reward but also provides suggestedactions in residual policy learning.
Experimental results in ball arrangement androom arrangement demonstrate that our method significantly outperforms the stateof-the-art methods in the quality of the terminal state, the efficiency of the controlprocess, and scalability.
Introduction
[Figure]
❌
✅ ✅
[Figure]
Given a set of target examples, the agents need to move objects to meet a normative distribution.
Object arrangement [1, 2] is to manipulate a setof objects from an initial layout to a normativedistribution, as shown in Fig.
This task is takenfor granted by humans [3] and widely exists in ourdaily life, such as tidying up a table [4], placingfurniture [5], and sorting parcels [6].
However, compared with conventional robottasks [7, 8, 9, 10], there are two critical challengesin arrangement tasks: 1) Hard to clearly definethe target state (goal), as the target distribution isdiverse and the patterns are difficult to describe inthe program, e.g ., how to evaluate the cleannessquantitatively?
As a result, it is difficult to manually design a reward/objective function, whichis essential to modern control methods, e.g ., deepreinforcement learning (DRL) [11].
In the physical world, the agents also need to find an executable path to reach the target state safely and efficiently,i.e ., finding a short yet collision-free path to reach the target distribution.
To this end, the agents arerequired to jointly learn to evaluate the quality of the state (reward) and take actions to move objects.
In other words, the agents should answer the two questions: What kind of layouts does the user prefer ,and How to manipulate objects to meet the target distribution ?
Thus, it is necessary to find a way to build an agent without the dependence on the explicit goalstate and shaped reward.
Imitation learning (IL) can implicitly learn the reward function from experttrajectories [12, 13, 14].
However, collecting a large number of expert trajectories is expensive.
Recently, example-based reinforcement learning (RL) [15, 16] has tried to reduce the reliance onexpert trajectories, i.e ., learning a policy only with examples of successful outcomes.
However, itis still difficult to learn a classifier to provide accurate reward signals for learning in the case ofhigh-dimensional state and action space, as there are many objects to control in object arrangements.
In this work, we separate the task into two stages: 1) learning a score function to qualify the state,and 2) learning/building a policy to arrange objects with the score function.
Inspired by the recentadvances in score-based generative models [17, 18, 19, 20, 21], we employ the score-matchingobjective to learn a target gradient field (TarGF) by perturbing the well-arranged examples.
Thetarget gradient field can be viewed as object-wise guidance that provides a pseudo velocity on eachobject, pointing to regions with a higher density of the target distribution, e.g ., cleaner configurations.
To manipulate objects in the physical world, we further demonstrate two ways to leverage the targetgradient field in control.
We integrate the guidance with a model-based path planner to arrangeobjects without collision.
We further derive the target gradient to a delta log-likelihood as a rewardfunction, key ingredients to reinforcement learning.
With the TarGF-based reward and a TarGF-guidedresidual policy network [22], we can train an effective RL-based policy from interactions.
In the experiments, we evaluate the effectiveness of our methods on two typical arrangement tasks:
Ball Arrangement and Room Arrangement .
First, we introduce three metrics to analyse the qualityof the terminal state and the control process.
In Ball Arrangement , we demonstrate that both of ourcontrol methods can efficiently reach a target distribution with high diversity and realism.
We alsoobserve that our methods can find a path with fewer collisions and fewer steps to converge comparedwith learning-based baselines [13, 16] and planning-based baselines [23].
In Room Arrangement ,we further evaluate the feasibility of our methods in complex scenes where there are multipleheterogeneous objects.
Our contributions are summarised as follows:
• We introduce a score-based method to learn a target gradient field (TarGF) by perturbingthe state of well-arranged target examples.
• We point out two usages of the target gradient field in object arrangement: 1) providingguidance in moving direction; 2) deriving a reward for learning.
• We conduct experiments to demonstrate that both traditional planning and RL methods
can benefit from the target gradient field and significantly outperform the state-of-the-artmethods in sample diversity, realism and safety.
Related Works2.1 Object and Scene Arrangements
The object arrangement has been previously studied by researchers from the robotics community [24,2, 25] and the graphics community [26, 27, 28, 29, 30].
However, most of them focus on manuallydesigning rules/ energy functions to find a goal [24, 2, 25] or synthesising a scene configuration [27,28, 29, 30] to satisfy the preference of the user.
The most recent work [4] tries to learn the preferencesfrom data with graph neural networks to organise the objects on a desk.
However, they neglect thephysical constraints during the arrangement process.
Hence, the accessibility and transition cost fromthe initial and target states are not guaranteed.
Considering the physical interaction, the recent workson scene rearrangement [31, 32] mitigate the goal generation problem by giving a specific goal stateto the agent, making the reward shaping feasible.
Different from these methods, our problem settingis more practical but challenging, as we have to build an interactive agent to arrange objects withoutknowing the specific goal state.
2.2 Learning without Reward Engineering
Without reward, previous works explore algorithms to learn a control policy from expert demonstrations via imitation learning [33, 34, 35, 12] or inverse reinforcement learning [36, 14, 37, 38].
Imitation learning (IL)
[33, 34, 35] aims to directly optimise the policy by cloning the behaviourfrom the expert trajectories.
The inverse reinforcement learning (IRL) [36, 14, 37, 38] tries to learna reward function from data for reinforcement learning.
Considering the difficulty in collectingexpert trajectories, there are some example-based methods proposed to learn policy only with aset of successful examples [15, 36, 13, 16].
The most recent work is RCE [16], a modification ofthe actor-critic-based method [39, 40], which directly learns a recursive classifier from successfuloutcomes and interactions for policy optimisation.
We also try to find an example-based controlmethod to arrange objects with a set of target examples.
Our proposed approach can learn the targetgradient fields from the examples, without any additional efforts in reward engineering. The targetgradient fields can provide meaningful reward signals and action guidance for reinforcement learningand traditional planners. To the best of our knowledge, such usage of gradients fields is not exploredin previous works.
Problem Statement
We aim to learn a policy that moves objects to reach states close to a target distribution without relyingon reward engineering.
Similar to the example-based control [16], the grounded target distributionp tar ( s ) is unknown to the agent, and the agent is only given a set of target examples S ∗ =
{ s ∗ } ,where s ∗ ∼ p tar ( s ) .
In practice, these examples can be provided by the user without accessingthe dynamics of the robot.
The agent starts from an initial state s 0 ∼ p 0 ( s 0 ) , where N objects arerandomly placed in the environment.
At time step t , the agent takes an action π ( a t | s t ) imposed on theobjects with dynamics p ( s t +1 | s t , a t ) .
Here, the goal of the agent is to search for a policy π ∗ ( a t | s t )that maximises the discounted log-target-likelihood of the future states, where γ ∈ (0 , 1] denotes thediscount factor:
[Equation]
This objective reveals three challenges in object arrangement: 1) Inaccessibility problem:
Thegrounded target distribution p tar in Eq.
1 is inaccessible.
Thus, we need to learn a function toapproximate the log-target-likelihood for policy search.
2) Sparsity problem: The log-targetlikelihood is sparse in the state space, as mentioned in [17].
Hence, even if we have access to the targetdistribution p tar , it is still difficult to explore the high-density region due to the Manifold Hypothesismentioned in [17].
3) Adaptation problem: The dynamics p ( s t +1 | s t , a t ) is also inaccessible.
Tomaximise the cumulative log-likelihood in Eq.
1, the agent is required to adapt to the dynamics toincrease the state likelihood as fast as possible.
To address these problems, we separate the task into two stages: 1) learning to estimate the log-targetlikelihood (reward) of the state and 2) learning/building a policy to arrange objects to adapt to thedynamics.
Method
4.1 Motivation
We argue that, estimating the gradients of the log-target-likelihood ∇ s log p tar ( s ) can tackle the firsttwo difficulties mentioned in Sec.
For the inaccessibility problem, we can approximate the rewardincrement between two adjacent states by the first-order Taylor expansion using the gradient fieldlog p tar ( s t +1 ) − log p tar ( s t )
≈ ⟨∇ s log p tar ( s t ) , s t +1 − s t ⟩ .
As shown in Sec.
4.4, this helps toderive a surrogate objective of the arrangement objective.
For the sparsity problem, we can leverage the gradient field to help with exploration, since thegradient ∇ s log p tar ( s ) indicates the direction that points to the higher density region of the targetdistribution p tar .
To estimate the gradients of the log-target-likelihood ∇ s log p tar ( s ) , noted asthe target gradient field (TarGF) Φ tar , we adopt the deep score-matching technique [18] that hasachieved impressive results in many research areas recently[19, 21, 20].
𝑎 𝑡𝑔
Env
Policy Network 𝜋 𝜃 𝑂𝑅𝐶𝐴Planner 𝜙 𝑠 𝑡+
[Figure]
𝑠 ∗
Perturbed State ෩𝑠 ∗
Perturbation 𝒩(0,𝜎 2 𝑰)
Score Matching ℒ2(𝛿,𝑔) (b) Control Objects with TarGF (a) Learning Target Gradient Field (TarGF)
𝑠
𝑡 𝑎 𝑡 = 𝜋 𝜃 𝑠 𝑡 ,𝒈 𝒕 +𝒢(𝒈 𝒕 ) 𝑎 𝑡
RL-based Control
Model-based Planning 𝑔 = Φ ෩𝑠 ∗ 𝛿 = ෡𝑠 ∗ −𝑠 ∗ 𝜎
[Figure] [Figure]
Grad2Action
𝒢 𝑔
[Figure]
𝑟 𝑡 (𝑠 𝑡 ,𝒈 𝒕 ) 𝑎 𝑡𝑔 𝑔 𝑡 + To address the adaptation problem, we further demonstrate two ways to incorporate the trained targetgradient field Φ tar with control algorithms for object arrangement: 1) For the model-based setting,our framework casts the target gradient into a reference control, and outputs actions with a distributedpath planner.
For the model-free setting (reinforcement learning), we leverage the target gradientto estimate reward and provide suggested actions in residual policy learning [22].
In the followings, we will first introduce how to train the target gradient field Φ tar via score-matchingin Sec.
and then introduce our arrangement framework under the model-based and model-freesettings in Sec.
4.3 and Sec.
4.4 respectively.
4.2 Learning the Target Gradient Field from Examples
The target score network Φ tar : S → S is the core module of our framework, which aims atestimating the score function ( i.e .the gradient of log-density) of the target distribution ∇ s log p tar ( s ) .
To train the target score network, we adopt the Denoising Score-Matching (DSM) objective proposedby [41], which can guarantee a reasonable estimation of the ∇ s log p tar ( s ) .
In training, we prespecify a noise distribution q σ ( (cid:101) s | s ) on the collected target state s .
Then, we match the output of thetarget gradient field with the score of the perturbed target distribution q σ ( (cid:101) s ) = (cid:82) q σ ( (cid:101) s | s ) p tar
[Equation]
s
[Equation]
s
[Equation] [Equation]
The DSM objective guarantees the optimal score network satisfies Φ ∗ tar ( s ) = ∇ s q σ ( s ) almost surely.
When σ is small enough, we have ∇ s q σ ( s )
≈ ∇
s log p tar ( s ) , so that Φ ∗ tar ( s )
≈ ∇
s log p tar ( s ) .
In practice, we adopt a variant of DSM proposed by [18], which conducts DSM under different noisescales simultaneously.
We implement the target score network as a graph-based network Φ tar ( s , t )that is conditioned on a noise level t ∈ (0 , 1] .
The input state s =
[ s 1 , s 2 , ..., s N ] is constructed asa fully-connected graph that takes each object’s state( e.g .position, orientation) s
i ∈ R f s and staticproperties ( e.g .category, bounding-box) p i ∈
R f p as the node feature [ s i , p i ] ∈
R f s + f p .
The inputgraph is pre-processed by linear initial feature extraction layers and then passed through two EdgeConvolution layers.
After message-passing, the output on each node serves as the component of thetarget gradient on the i-th object’s state Φ itar ( s ) ∈ R f s .
We defer structure details to supplementary.
[Figure]
4.3 Model-based Planning with the Target Gradient Field
The target gradient g =
Φ tar ( s ) can be translated into a gradient-based action a g = G ( g ) forarrangement, where G denotes the gradient-to-action translation.
A gradient over the state spacecan be viewed as velocities imposed on each object.
For instance, if the state of each object is a2-dimensional position s i =
[ x, y ] , then the target gradient component on the i-th object can beviewed as linear velocities on two axes Φ tar ( s )
i =
[ v x , v y ] .
When the action space is also velocities,we can construct a g by simply projecting the target gradient a g = G ( g ) = v max
|| g
|| 2 · g into the actionspace, where the v max denotes the speed limit.
Following the gradient-based action, objects willmove towards directions that may increase the likelihood of the next state, which meets the need ofthe arrangement task.
However, the gradient-based action is imperfect ( e.g .leading to collisions between the objects), as thetarget gradient has no access to the information of environment dynamics.
This may be inefficientand unsafe due to the severe collisions between objects.
To adapt the gradient-based action to the environment dynamics, we incorporate the target gradientfield with a model-based off-the-shelf collision-avoidance algorithm ORCA .
The ORCA planner ϕ θtakes the gradient-based action a g as reference velocities and then outputs the collision-free velocitiesa = ϕ θ ( a g ) with the least modification to the reference velocities.
In this way, the adapted action canlead objects to move towards a higher likelihood region efficiently and safely.
For more details of the ORCA planner, we defer to supplementaries.
4.4 Learning Policy with the Target Gradient Field
We further propose a framework based on the model-free reinforcement learning (RL) algorithm toaccomplish the arrangement tasks in the model-free scenario, where the agent needs to adapt thedynamics via interacting with the environment.
To tackle the inaccessibility problem mentioned in Sec. 3, we first derive an equivalent J ∗ ( π ) ofthe original objective in Eq.
1 and further obtain a surrogate objective ˆ J ( π ) to approximate thisequivalent:
[Equation]
Further, we derive a surrogate objective ˆ J ( π ) to approximate the equivalent objective J ∗ ( π ) . Wenotice that in most physical simulated tasks, the distance between two adjacent states || s t − s t −
|| 2is quite small, which inspires us to approximate the delta log-likelihood using the first order Taylorexpansion:
[Equation]
In this way, we can optimise the surrogate objective ˆ Jπ by assigning (cid:80) [Equation] , s
[Equation]
⟩ asthe immediate reward.
In practice, we simply assign the last term of the cumulative summation as theimmediate reward r t = ⟨ Φ tar ( s t ) , s t − s t − 1 ⟩ , and this version is shown to be the most effective.
To tackle the sparsity problem mentioned in Sec. 3, our idea is to build our policy π θ upon thegradient-based action a gt mentioned in Sec.
4.3, as a gt is the best direction to increase the likelihood:
[Equation]
targetgradient
[Equation]
[Equation] This approach is similar to the residual policy learning proposed by [22].
The a gt serves as aninitial policy, and the agent’s policy network π θ takes the target gradient g t and the current states t as input and outputs π ( s t , g t ) to ‘polish’ the a gt .
The agent finally outputs ‘polished’ actiona t =
π ( s t , g t ) + a gt .
In this way, the agent can not only benefit from the efficient exploration butalso have less training burden as it ‘stands on the giant’s shoulder’
( i.e .the gradient-based action a gt ).
We implement the multi-agent Soft-Actor-Critic(SAC) as the reinforcement learning algorithmbackbone.
Each object is regarded as an agent, and each agent can communicate with all the otheragents.
At each time step t , the reward for the i-th agent is r t = r tart + λ ∗ r it , where r tart =(cid:80) 1 ≤ k ≤ t ⟨ Φ tar ( s t ) , s k − s k − 1 ⟩ is a centralised target likelihood reward, λ > 0 is a hyperparameter andr it = (cid:80)
j ̸
= i c i,j is a decentralised collision-aware reward where c i,j is a collision penalty. Specifically,when a collision is detected between object i and j , c i,j = − 1 , otherwise c i,j = 0 .
Experiment Setups
[Figure]
5.1 Environment
We design two object arrangement tasks for evaluation: Ball Arrangement and Room Arrangement , where the former uses controlled environments for numerical analysis and the latter uses real data todemonstrate the potential of our method on a more challenging task.
Ball Arrangement includes three environments with increasing difficulty of arrangement:
Circling ,
Clustering , and the hybrid of the first two, Circling + Clustering .
There are 21 balls with the samegeometry bounded in a square in each environment.
In Clustering and Circling+Clustering , the ballsare divided equally into three colours.
The goal of each environment is as follows:
Circling requiresthe arrangement of all balls into a circle.
The centre of the circle can be anywhere in the environment;
Clustering requires the arrangement of all balls into three clusters by colour.
The joint centre of eachcluster has two choices; Circling + Clustering requires the arrangement of all balls into a circle whereballs of the same type are adjacent.
To slightly increase the complexity of the physical dynamics,we enlarge the lateral friction coefficient of each ball and make each ball with different physicalproperties , e.g ., friction coefficient and mass.
The target examples of each task are sampled from anexplicit process.
Thus, we can define a ‘pseudo likelihood’ on a given state according to the samplingprogram to measure the similarity between the state and the target distribution.
Room Arrangement is built on a more realistic dataset, 3D-FRONT [42].
After cleaning the data,we get a dataset with 839 rooms, 756 for training( e.g .target examples), and 83 for testing.
In eachepisode, we sample a room from the test dataset and shuffle the room with a Brownian Process at thebeginning to get an initial state.
The agent can assign velocities for each object in the room at eachtime step.
The agent has to learn implicit knowledge learned from the target examples to tidy up theroom.
More details are in the supplementary.
[Figure]
5.2 Evaluation
For both baseline comparison and ablation studies, we collect a fixed number of trajectories T
= { τ i }starting from the same initial states for different methods.
Then we calculate the following metricsusing the collected trajectories, where the PL is only reported in ball sorting tasks:
Pseudo-Likelihood (PL) measures the efficiency of the arrangement process and the sample qualityof the arrangement results.
We specify a pseudo-likelihood function F proxy for each environmentthat measures the similarity between a given state and the target examples.
For each time step t , the
PL is defined as F proxy ( s t ) .
We report the mean-variance curve of PL with respect to the time step t ,where the mean and variance are calculated over the trajectories T .
We do not report PL on roomarrangement as it is hard to program the human preferences.
Coverage Score (CS): measures the diversity and fidelity of the arrangement results ( i.e .the terminalstates S T = { s T } of the trajectories) by calculating the Minimal-Matching-Distance(MMD)
[43]between S T and a fixed set of examples S gt from p tar : (cid:80)
[Equation]
Averaged Collision Number (ACN) reflects the efficiency of the arrangement process since objectcollision will lead to object blocking and deceleration.
ACN is calculated as (cid:80) τ ∈ T (cid:80) s t ∈ τ c t , where 243 c t denotes the total collision number at time step t .
More details are in the supplementary.
5.3 Baselines
We compare our framework with planning-based baselines under the model-based setting and withlearning-based baselines under the model-free setting.
Besides, we design heuristic baselines servingas the upper bounds for ball arrangement tasks.
Leaning-based Baselines: GAIL: A classical inverse RL method that trains a discriminator asthe reward function.
RCE:
The SOTA example-based reinforcement learning method.
SQIL: Acompetitive example-based imitation learning method.
Goal-RL: An ‘open-loop’ method that firstgenerates a goal at the beginning of the episode via a VAE trained from the target examples and thentrained to reach the goal using goal-conditioned RL.
Planning-based Baselines: Goal-ORCA
: This method first generates a goal via the same VAE as in
Goal-RL .
Then the agent assigns an action pointing to the goal as reference control and adjusts thepreference velocity using ORCA, similar to Sec. 4.3.
Heuristic-based Baseline: Oracle:
We slightly perturb the samples from the target distribution andtake these samples as the arrangement results.
[Figure]
PL
Circling
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
O u r s ( O
R C A )
G o a l ( O
R C A )
O u r s w / o
R L
O u r s ( S A
C )
G
[Table]
a l ( S A
C )
[Table]
u r
[Table]
w / o
[Table]
i d u [Table]
l
[Table]
I
[Table]
A
[Table]
[Figure]
CS
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
R L
G A I L ( S A [Table]
[Table]
s (
[Table]
[Table]
l (
[Table]
A
[Table]
[Table]
l (
[Table]
R
[Table]
A )
[Table]
r s ( S A
C )
[Table]
w / o
R L
G A I L ( S A C ) S Q I L
O u r s w / o
R e s i d u a l R C E O u r s ( S A C )
G A I L ( S A
C )
G o a l ( S A C ) R C E S Q I L
Room
[Table]
O u r s w / o
R e s i d u a l O u r s ( O R C A )
G o a l ( S A
C )
G o a l ( O
R C A )
O u r s ( S A
C )
G A I L ( S A
C )
O u r s w / o
R L R C E S Q I L 0.
ACN
O u r s ( O
R C A )
O u r s w / o
R e s i d u a l
O u r s ( S A
C )
G o a l ( S A
C )
G o a l ( O
R C A )
O u r s w / o
R L
G A I L ( S A
C ) R C E S Q I L
O u r s ( O
R C A )
O u r s w / o
R e s i d u a l
O u r s ( S A
C )
G o a l ( O
R C A )
G o a l ( S A
C )
O u r s w / o
R L
G A I L ( S A
C ) S Q I L R C E
O u r s ( S A
C )
G o a l ( S A
C )
G A I L ( S A
C )
S Q I L
R C E
Result Analysis
[Figure]
6.1 Baseline Comparison
[Figure] Ball Arrangement: We present the quantitative results on ball arrangement tasks in the first threecolumns of the Fig.
As seen from the top row of Fig.
5, the PL curves of our model-freeframework(Ours-SAC) and model-based framework (Ours-ORCA) are significantly higher andsteeper than the baselines in all three tasks, which shows the effectiveness of our method in arrangingobjects towards the target distribution efficiently.
In the middle row of Fig.
5, Ours-ORCA achievesthe best performance across all three tasks, while Goal-SAC outperforms Ours-SAC on Clusteringand Circling+Clustering since the goal is proposed by a generative model that can naturally generatediverse and realistic goals.
From the qualitative results shown in Fig.
3, our methods also significantlyoutperform the baselines in realism.
Our methods also achieve the lowest ACN in most cases, exceptthat Goal-SAC is slightly better than Ours-SAC in Circling .
The Goal-SAC is the strongest baseline yet loses to our framework in efficiency, since the generatedgoal may be far away from the initial state.
As shown in Fig.
3, the Goal-SAC and Goal-ORCAcan arrange objects in the right trend, but the objects may block each other’s way.
Besides, in thesupplementary, we show the VAE used in Goal-SAC is not guaranteed to generate a legal state( e.g .theballs overlap in goal).
Therefore, the balls may have unexpected behaviour during the process.
The classifier-based reward learning method, GAIL, RCE, and SQIL, basically fails in most tasks,as shown in Fig.
3 and Fig.
This is due to the over-exploitation problem that commonly exists inclassifier-based reward learning methods.
We defer the experiments to supplementary.
Room Arrangement: The dynamics of the room-arrangement task are too complex to applyexplicit planning-based methods, so we only compare our model-free framework with learning-basedbaselines.
As shown in Fig.
4 and the last column of Fig.
5, our method significantly outperformsother baselines in sample realism of arrangement results and the safety of the arrangement process,which proves our method can also work on a more complex task.
6.2 Ablation Studies and Analysis
We conduct ablation studies on ball arrangement tasks to investigate: 1) The effectiveness of theexploration guidance provided from the target gradient field and 2)
The necessity of combining thetarget gradient field with control algorithms to adapt to the environment dynamics.
To this end, weconstruct an ablated version named Ours w/o Residual that drops the residual policy learning andanother one named Ours w/o RL that only takes gradient-based action a g as policy.
Ours w/o Residual: The CS-bar in Fig.
5 shows that Ours significantly outperforms the Ours w/o
Residual in CS, which means the agent is easy to over-exploit few modes with high density.
We alsoshow qualitative results in supplementary that without the residual policy learning, the objects areusually arranged into a single pattern.
Ours w/o RL: The PL-curve in Fig.
5 shows that without the control algorithm serving as thebackbone, the arrangement efficiency of Ours w/o RL is significantly lower than Ours.
Meanwhile,the ACN of Ours w/o RL is significantly larger than Ours, which indicates the efficiency drop may bedue to the collisions caused by the gradient-based action. [Figure]
Generalisation: We further evaluatethe generalisation of our methods todifferent numbers of objects.
To bespecific, we train all the policies inthe ball environment with 3 × 7 = 21balls.
During the testing phase, wedirectly transfer each policy to environments with increased ball number,i.e., 3 × 8 , 3 × 9 , and 3 × 10 , withoutany fine-tuning.
As shown in Fig.
6.2,we report the averaged PL incrementfrom initial to target states of each policy in each configuration.
The results indicate that our framework generalises well to different numbers of balls with the least performance drop compared withbaselines in all three ball arrangement tasks.
Conclusion
In this study, we first analyse the challenge of object arrangement and then incorporate a gradientfield method with either model-based planning or model-free learning approaches to achieve objectarrangement tasks.
Experiments demonstrate our effectiveness by comparison with learning-basedand planning-based baselines.
Limitation and Future Work.
This work only considers the 2-D state-based arrangement withsimplified dynamics that the objects can arrange themselves and directly control the velocities.
Futureworks may extend our framework to more realistic scenarios: 1) Visual Observation: We mayexplore how to conduct efficient and accurate score-based reward estimation from image-basedexamples and leverage the recent progress on visual state representation learning [44, 45, 46] todevelop the visual-based gradient-based action. Complex Dynamics: When the agent can onlymove one object at a time, we may explore hierarchical methods where high-level policy chooseswhich object to move and low-level policy leverage the gradient-based action for motion planning. When we can only apply force to objects instead of velocity, we may explore a hierarchical policywhere a high-level policy outputs expected velocities and a low-level employ a velocity controller,such as PID controller, to minimises the velocity errors.
Ethics Statement.
Our method has the potential to build home-assistant robots.
We evaluate ourmethod in simulated environments, which may introduce data bias.
However, similar studies alsohave such general concerns.
We do not see any possible major harm of our study.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] see Sec.
(b) Did you describe the limitations of your work?
[Yes] see Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] see supplementary
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] see Sec.
7 and supplementary
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
[Table] (a) Did you include the code, data, and instructions needed to reproduce the main experi- [Table] mental results (either in the supplemental material or as a URL)?
[Yes] please see the
[Table]
supplemental material.
[Table] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] more details in the supplemental material.
[Table] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] more details in the supplemental material.
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., type [Table] of GPUs, internal cluster, or cloud provider)?
[Yes] more details in the supplemental [Table] material. [Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[N/A]
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[N/A]
[Table] (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table]
information or offensive content?
[Yes]5.
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]
spent on participant compensation?
[N/A]
[Table]