Learning Gradient Fields for Object Arrangement
Anonymous Author(s)
AffiliationAddress email
Abstract
Object Arrangement is to move objects from shuffled layouts to a normative targetdistribution, e.g. , tidy rooms.
However, it remains challenging for AI agents, asit is hard to describe the target distribution (goal state) for reward engineering orcollect expert trajectories as demonstrations.
Hence, it is infeasible to directlyemploy reinforcement learning or imitation learning algorithms to address thetask.
This paper aims to search for a policy only with a set of examples from atarget distribution instead of a handcrafted reward function.
We employ the scorematching objective to train a target gradient field (TarGF) , indicating a directionon each object to increase the likelihood of the target distribution.
For objectarrangement, the TarGF can be used in two ways: 1) For model-based planning,we can cast the target gradient into a reference control and output actions with adistributed path planner; 2) For model-free reinforcement learning, the TarGF is notonly used for estimating the delta likelihood as a reward but also provides suggestedactions in residual policy learning.
Experimental results in ball arrangement androom arrangement demonstrate that our method significantly outperforms the stateof-the-art methods in the quality of the terminal state, the efficiency of the controlprocess, and scalability.
Introduction
[Figure]
❌
✅ ✅
[Figure]
Given a set of target examples, the agents need to move objects to meet a normative distribution.
Object arrangement [1, 2] is to manipulate a setof objects from an initial layout to a normativedistribution, shown as Fig.
This task is takenfor granted by humans [3] and widely exists in ourdaily life, such as tidying up a table [4], placingfurniture [5], and sorting parcels [6].
However, compared with conventional robottasks [7, 8, 9, 10], there are two critical challengesin arrangement tasks: 1) Hard to clearly definethe target state (goal), as the target distribution isdiverse and the patterns are difficult to describe inthe program, e.g ., how to evaluate the cleannessquantitatively?
As a result, it is difficult to manually design a reward/objective function, whichis essential to modern control methods, e.g ., deepreinforcement learning (DRL) [11].
In the physical world, the agents also need to find an executable path to reach the target state safely and efficiently,i.e ., finding a short yet collision-free path to reach the target distribution.
To this end, the agents arerequired to jointly learn to evaluate the quality of the state (reward) and take actions to move objects.
In other words, the agents should answer the two questions: What kind of layouts does the user prefer ,and How to manipulate objects to meet the target distribution .
Thus, it is necessary to find a way to build an agent without the dependence on the explicit goalstate and shaped reward.
Imitation learning (IL) can implicitly learn the reward function from experttrajectories [12, 13, 14].
However, collecting a large number of expert trajectories is expensive.
Recently, example-based reinforcement learning (RL) [15, 16] tries to reduce the reliance on experttrajectories, i.e ., learning a policy only with examples of successful outcomes.
However, it is stilldifficult to learn a classifier to provide accurate reward signals for learning in the case of highdimensional state and action space, as there are many objects to control in object arrangements.
In this work, we separate the task into two stages: 1) learning a score function to qualify the state,and 2) learning/building a policy to arrange objects with the score function.
Inspired by the recentadvances in score-based generative models [17, 18, 19, 20, 21], we employ the score-matchingobjective to learn a target gradient field (TarGF) by perturbing the well-arranged examples.
Thetarget gradient field can be viewed as object-wise guidance that provides a pseudo velocity oneach object, pointing to the regions with higher density of the target distribution, e.g ., cleanerconfigurations.
To manipulate objects in the physical world, we further demonstrate two ways toleverage the target gradient field in control.
We integrate the guidance with a model-based pathplanner to arrange objects without collision.
We further derive the target gradient to a deltalog-likelihood as a reward function, key ingredients to reinforcement learning.
With the TarGF-basedreward and a TarGF-guided residual policy network [22], we can train an effective RL-based policyfrom interactions.
In the experiments, we evaluate the effectiveness of our methods on two typical arrangement tasks: Ball Arrangement and Room Arrangement .
First, we introduce three metrics to analyze the qualityof the terminal state and the control process.
In Ball Arrangement , we demonstrate that both of ourcontrol methods can efficiently reach a target distribution with high diversity and realism.
We alsoobserve that our methods can find a path with fewer collisions and fewer steps to converge comparedwith learning-based baselines [13, 16] and planning-based baselines [23].
In Room Arrangement ,we further evaluate the feasibility of our methods in complex scenes where there are multipleheterogeneous objects.
Our contributions are summarized as follows:
• We introduce a score-based method to learn a target gradient field (TarGF) by perturbingthe state of well-arranged target examples.•
We point out two usages of the target gradient field in object arrangement: 1) providingguidance in moving direction; 2) deriving a reward for learning. • We conduct experiments to demonstrate that both traditional planning and RL methods
can benefit from the target gradient field and significantly outperform the state-of-the-artmethods in sample diversity, realism and safety.
Related Works2.1 Object and Scene Arrangements
The object arrangement is previously studied by researchers from robotics [24, 2, 25] and graphicscommunity [26, 27, 28, 29, 30].
However, most of them focus on manually design rules/ energyfunctions to find a goal [24, 2, 25] or synthesis a scene configuration [27, 28, 29, 30] to satisfy thepreference of the user.
The most recent work [4] tries to learn the preferences from data with graphneural networks to organize the objects in desk.
However, they neglect the physical constraintsduring the arrangement process.
Hence, the accessibility and transition cost from the initial andtarget states are not guaranteed.
Considering the physical interaction, the recent works on scenerearrangement [31, 32] mitigate the goal generation problem by giving a specific goal state to theagent, making the reward shaping feasible.
Different from these methods, our problem setting ismore practical but challenging, as we have to learn a interactive agent to arrange objects withoutknowing specific goal state.
2.2 Learning without Reward Engineering
Without reward, previous works explore algorithms to learn a control policy from expert demonstrations via imitation learning [33, 34, 35, 12] or inverse reinforcement learning [36, 14, 37, 38].
Imitation learning (IL) [33, 34, 35] aims at directly optimize the policy by cloning the behavior fromthe expert trajectories.
The inverse reinforcement learning (IRL) [36, 14, 37, 38] tries to learn areward function from data for reinforcement learning.
Considering the difficulty in collecting experttrajectories, there are some example-based methods proposed to learn policy only with a set of successexamples [15, 36, 13, 16].
The most recent work is RCE [16], a modification of actor-critic-basedmethod [39, 40], which directly learns a recursive classifier from successful outcomes and interactionsfor policy optimization.
We do also tries to find a example-based control method to arrange objectswith a set of target examples.
Differently, our proposed approach focus on learning the target gradientfields from the examples. Besides, both RL method and traditional planner can be supported by ourtarget gradient fields in the object arrangement task.
Problem Statement
We aim to learn a policy that moves objects to reach states close to a target distribution, withoutrelying on reward engineering.
Similar to the example-based control [16], the grounded targetdistribution p tar ( s ) is unknown to the agent, and the agent is only given a set of target examples S ∗ =
{ s ∗ } , where s ∗ ∼ p tar ( s ) .
In practice, these examples can be provided by the user, withoutaccessing the dynamics of the robot.
The agent starts from an initial state s 0 ∼ p 0 ( s 0 ) , where Nobjects are randomly placed in the environment.
At time step t , the agent takes an action π ( a t | s t )imposed on the objects with dynamics p ( s t +1 | s t , a t ) .
Here, the goal of the agent is to search a policyπ ∗ ( a t | s t ) that maximizes the discounted log-target-likelihood of the future states, where γ ∈ (0 , 1]denotes the discount factor:
[Equation]
This objective reveals three challenges in object arrangement: 1) Inaccessibility problem: Thegrounded target distribution p tar in Eq.
1 is inaccessible.
Thus, we need to learn a function to approximate the log-target-likelihood for policy search.
2) Sparsity problem: The log-target-likelihoodis sparse in the state space as mentioned in [17].
Hence, even if we are accessible to the targetdistribution p tar , it is still difficult to explore the high density region due to the Manifold Hypothesismentioned in [17].
3) Adaptation problem: The dynamics p ( s t +1 | s t , a t ) is also inaccessible.
Tomaximise the cumulative log-likelihood in Eq.
1, the agent is required to adapt to the dynamics toincrease the state likelihood as fast as possible.
In address these problems, we separate the task into two stages: 1) learning to estimate the logtarget-likelihood (reward) of the state, 2) learning/building a policy to arrange objects to adapt thedynamics.
Method
4.1 Motivation
We argue that, estimating the gradients of the log-target-likelihood ∇ s log p tar ( s ) can tackle the firsttwo difficulties mentioned in Sec.
For the inaccessibility problem, we can approximate the rewardincrement between two adjacent states by the first-order Taylor expansion using the gradient fieldlog p tar ( s t +1 ) − log p tar ( s t )
≈ ⟨∇ s log p tar ( s t ) , s t +1 − s t ⟩ .
As shown in Sec.
4.4, this helps toderive a surrogate objective of arrangement objective.
For the sparsity problem, we can leverage the gradient field to help with exploration, since thegradient ∇ s log p tar ( s ) indicates the direction that points to the higher density region of the targetdistribution p tar .
To estimate the gradients of the log-target-likelihood ∇ s log p tar ( s ) , noted as targetgradient field (TarGF) Φ tar , we adopt deep score-matching technique [18] that achieves impressiveresults on many research areas recently[19, 21, 20].
To address the adaptation problem, we further demonstrate two ways to incorporates the trainedtarget gradient field Φ tar with control algorithms for object arrangement: 1) For model-based setting, 𝑎 𝑡𝑔
Env
Policy Network 𝜋 𝜃 𝑂𝑅𝐶𝐴Planner 𝜙 𝑠 𝑡+
[Figure]
𝑠 ∗
Perturbed State ෩𝑠 ∗
Perturbation 𝒩(0,𝜎 2 𝑰)
Score Matching ℒ2(𝛿,𝑔) (b) Control Objects with TarGF (a) Learning Target Gradient Field (TarGF)
𝑠
𝑡 𝑎 𝑡 = 𝜋 𝜃 𝑠 𝑡 ,𝒈 𝒕 +𝒢(𝒈 𝒕 ) 𝑎 𝑡
RL-based Control
Model-based Planning 𝑔 = Φ ෩𝑠 ∗ 𝛿 = ෡𝑠 ∗ −𝑠 ∗ 𝜎
[Figure]
[Figure]
Grad2Action
𝒢 𝑔
[Figure]
𝑟 𝑡 (𝑠 𝑡 ,𝒈 𝒕 ) 𝑎 𝑡𝑔 𝑔 𝑡 + our framework casts the target gradient into a reference control, and outputs action with a distributedpath planner.
For model-free setting (reinforcement learning), we leverage the target gradient toestimate reward and provide suggested actions in residual policy learning [22].
In the followings, we will first introduce how to train the target gradient field Φ tar via score-matchingin Sec.
4.2, then introduce our arrangement framework under the model-based and model-free settingsin Sec.
4.3 and Sec.
4.4 respectively.
4.2 Learning the Target Gradient Field from Examples
The target score network Φ tar : S → S is the core module of our framework, which aims atestimating the score function ( i.e .the gradient of log-density) of the target distribution ∇ s log p tar ( s ) .
To train the target score network, we adopt Denoising Score-Matching (DSM) objective proposedby [41], which can guarantee a reasonable estimation of the ∇ s log p tar ( s ) .
In training, we prespecifies a noise distribution q σ ( (cid:101) s | s ) on the collected target state s .
Then, we matches the output oftarget gradient field and the score of the perturbed target distribution q σ ( (cid:101) s [Equation] tar ( s )
[Equation]
s
[Equation]
[Equation]
The DSM objective guarantees the optimal score network satisfies Φ ∗ tar ( s ) = ∇ s q σ ( s ) almost surely.
When σ is small enough, we have ∇ s q σ ( s )
≈ ∇
s log p tar ( s ) , so that Φ ∗ tar ( s )
≈ ∇
s log p tar ( s ) .
In practice, we adopt a variant of DSM proposed by [18] which conducts DSM under different noisescales simultaneously.
We implement the target score network as graph-based network Φ tar ( s , t )that conditioned on a noise level t ∈ (0 , 1] .
The input state s =
[ s 1 , s 2 , ..., s N ] is constructed as afully-connected graph that takes each object’s state( e.g .position, orientation) s i ∈
R f s and staticproperties ( e.g .category, bounding-box) p i ∈
R f p as node feature [ s i , p i ] ∈
R f s + f p .
The inputgraph are pre-processed by linear initial feature extraction layers, and then passed through two EdgeConvolution layers.
After message-passing, the output on each node serves as the component of thetarget gradient on the i-th object’s state Φ i tar ( s ) ∈ R f s .
We defer structure details to supplementary.
4.3 Model-based Planning with the Target Gradient Field
The target gradient g =
Φ tar ( s ) can be translated into a gradient-based action a g = G ( g ) forarrangement, where G denotes the gradient-to-action translation.
A gradient over the state space [Figure] can be viewed as velocities imposed on each object.
For instance, if the state of each object is2-dimensional position s i =
[ x, y ] , then the target gradient component on i-th object can be viewedas linear velocities on two axis Φ tar ( s )
i =
[ v x , v y ] .
When the action space is also velocities, wecan construct a g by simply projecting the target gradient a g = G ( g ) = v max
|| g
|| 2 · g into the actionspace, where the v max denotes the speed limit.
Following the gradient-based action, objects willmove towards directions that may increase the likelihood of the next state, which meets the need ofarrangement task.
However, the gradient-based action is imperfect ( e.g .leading to collisions between the objects), as thetarget gradient have no access to information of environment dynamics.
This may be inefficient andunsafe due to the severe collisions between objects.
To adapt the gradient based action with the environment dynamics, we incorporate the target gradientfield with a model-based off-the-shelf collision-avoidance algorithm ORCA .
The ORCA plannerϕ θ takes the gradient based action a g as reference velocities, and then outputs the collision-freevelocities a = ϕ θ ( a g ) with the least modification to the reference velocities.
In this way, the adaptedaction can lead objects move towards higher likelihood region efficiently and safely.
For more detailsof ORCA planner we defer to supplementaries.
4.4 Learning Policy with the Target Gradient Field
We further propose a framework based on the model-free reinforcement learning (RL) algorithmto accomplish the arrangement tasks in model-free scenario, where the agent needs to adapt thedynamics via interacting with the environment.
To tackle the inaccessibility problem mentioned in Sec. 3, we first derive an equivalent J ∗ ( π ) of theorigin objective in Eq.
1 and further obtain a surrogate objective ˆ J ( π ) to approximates this equivalent:
[Equation]
Further, we derive a surrogate objective ˆ J ( π ) to approximate the equivalent objective J ∗ ( π ) Wenotice that in most physical simulated tasks, the distance between two adjacent states || s t − s t −
|| 2is quite small, which inspires us to approximate the delta log-likelihood using the first order Taylorexpansion:
[Equation]
In this way, we can optimise the surrogate objective ˆ Jπ by assigning (cid:80)
[Equation]
as immediate reward.
In practice, we simply assign the last term of the cumulative summation asimmediate reward r t = ⟨ Φ tar ( s t ) , s t − s t − 1 ⟩ , and this version is shown to be most effective.
To tackle the sparsity problem mentioned in Sec. 3, our idea is to build our policy π θ upon thegradient-based action a gt mentioned in Sec.
4.3, as a gt is the best direction to increase the likelihood:
[Equation]
targetgradient (
[Equation]
This approach is similar to the residual policy learning proposed by [22].
The a gt serves as an initialpolicy, and the agent’s policy networks π θ takes the target gradient g t and current state s t as input andoutputs π ( s t , g t ) to ‘polish’ the a gt .
The agent finally outputs ‘polished’ action a t =
π ( s t , g t ) + a gt .
In this way, the agent can not only benefits from the efficient exploration but also reduce the trainingburden as it ‘stands on the giant’s shoulder’
( i.e .the gradient based action a gt ).
We implement multi-agent Soft-Actor-Critic(SAC) as the reinforcement learning algorithm backbone.
Each object is regraded as an agent, and each agent can communicate with all the otherobjects.
All the agents receives a centralised reward and a decentralised reward from the environmentfeedback( i.e .collision number at t ).
For more implementation details we defer to supplementaries.
Experiment Setups
5.1 Environment
We design two object arrangement tasks for evaluation: Ball Arrangement and Room Arrangement ,where the former uses controlled environments for numerical analysis and the latter uses real data todemonstrate the potential of our method on a more challenging task.
Ball Arrangement includes three environments with increasing difficulty of arrangement:
Circling , Clustering , and the hybrid of the first two, Circling + Clustering .
There are 21 balls with the samegeometry bounded in a square in each environment.
In Clustering and Circling+Clustering , the ballsare divided equally into three colors.
The goal of each environment is as follows:
Circling requiresthe arrangement of all balls into a circle.
The center of the circle can be anywhere of the environment;
Clustering requires the arrangement of all balls into three clusters by color.
The joint center of eachcluster has two choices; Circling + Clustering requires the arrangement of all balls into a circle whereballs of the same type are adjacent.
To slightly increase the complexity of the physical dynamics,we enlarge the lateral friction coefficient of each ball and make each ball with different physicalproperties , e.g ., friction coefficient and mass.
The target examples of each task are sampled from anexplicit process.
Thus, we can define a ‘pseudo likelihood’ on a given state according to the samplingprogram to measure the similarity between the state and the target distribution.
Room Arrangement is built on a more realistic dataset, 3D-FRONT [42].
After cleaning the data,we get a dataset with 839 rooms, 756 for training( e.g .target examples), and 83 for testing.
In eachepisode, we sample a room from the test dataset and shuffle the room with a Brownian Process at thebeginning to get an initial state.
The agent can assign velocities for each object in the room at eachtime step.
The agent has to learn implicit knowledge learned from the target examples to tidy up theroom.
More details are in the supplementary.
[Figure]
5.2 Evaluation
For both baseline comparison and ablation studies, we collect a fixed number of trajectories T
= { τ i }starting from the same initial states for different methods.
Then we calculate the following metricsusing the collected trajectories, where the PL is only reported in ball sorting tasks:
Pseudo Likelihood (PL) measures the efficiency of the arrangement process and the sample qualityof the arrangement results.
We specify a pseudo likelihood function F proxy for each environmentthat measures the similarity between a given state and the target examples.
For each time step t , the
PL is defined as F proxy ( s t ) .
We report the mean-variance curve of PL with respect to the time step t ,where the mean and variance are calculated over the trajectories T .
Coverage Score (CS): measures the diversity and fidelity of the arrangement results ( i.e .the terminalstates S T = { s T } of the trajectories) by calculating the Minimal-Matching-Distance(MMD)
[43]between S T and a fixed set of examples S gt from p tar :
[Equation]
Averaged Collision Number (ACN) reflects the efficiency of the arrangement process since objectcollision will lead to object blocking and deceleration.
ACN is calculated as (cid:80) τ ∈ T (cid:80) s t ∈ τ c t where 239 c t denotes the total collision number at time step t .
More details are in the supplementary.
5.3 Baselines
We compare our framework with planning-based baselines under the model-based setting and withlearning-based baselines under the model-free setting.
Besides, we design heuristic baselines servingas the upper bounds for ball arrangement tasks.
Leaning-based Baselines: GAIL: A classical inverse RL method that trains a discriminator asthe reward function.
RCE:
The SOTA example-based reinforcement learning method.
SQIL: Acompetitive example-based imitation learning method.
Goal-RL: An ‘open-loop’ method that firstgenerates a goal at the beginning of the episode via a VAE trained from the target examples and thentrained to reach the goal using goal-conditioned RL.
Planning-based Baselines: Goal-ORCA
: This method first generates a goal via the same VAE as in
Goal-RL .
Then the agent assigns an action pointing to the goal as reference control and adjusts thepreference velocity using ORCA, similar to Sec. 4.3.
Heuristic-based Baseline: Oracle:
We slightly perturb the samples from the target distribution andtake these samples as the arrangement results.
[Figure]
[Table]
Circling
[Figure]
[Table]
[Figure]
[Table]
O u r s ( O
RC A )
G o a l ( O
RC A )
O u r s w / o
R L
O u r s ( S A
C )
G o a l ( S A
C )
O u r s w / o
[Table]
e s i
[Table]
l
GA I L
[Table]
A
[Table]
[Figure]
[Table]
[Figure]
CS
[Table]
u r s (
[Table]
RC A ) G o a l ( O
RC A )
G
[Table]
a l ( S A C ) [Table]
u r s w / o [Table] e
[Table]
i
[Table] l [Table] r s ( S A
C )
[Table]
/ o
[Table]
A
[Table]
RC
[Table]
[Table]
a l ( S A
[Table] [Table] l ( O RC A )
[Table]
[Table]
/ o
[Table] S A
C ) S [Table] I [Table]
u r s w / o
R e s i
[Table] l [Table]
O u r s ( S A
C )
[Table] A
C )
[Table]
o a l ( S A
C )
[Table]
S [Table] I L
[Table]
O u r s w / o
R e s i du a l
O u r s ( O
RC A )
G o a l ( S A
C )
G o a l ( O
RC A )
O u r s ( S A
C )
GA I L ( S A
C )
O u r s w / o
R L RC E S Q I L 0.
ACN
O u r s ( O
RC A )
O u r s w / o
R e s i du a l
O u r s ( S A
C )
G o a l ( S A
C )
G o a l ( O
RC A )
O u r s w / o
R L
GA I L ( S A
C ) RC E S Q I L
O u r s ( O
RC A )
O u r s w / o
R e s i du a l
O u r s ( S A
C )
G o a l ( O
RC A )
G o a l ( S A
C )
O u r s w / o
R L
GA I L ( S A
C ) S Q I L RC E
O u r s ( S A
C )
GA I L ( S A
C )
G o a l ( S A
C )
S Q I L
RC E
Result Analysis
6.1 Baseline Comparison
Ball Arrangement: We present the quantitative results on ball arrangement tasks in the first threecolumns of the Fig.
As seen from the top row of Fig.
5, the PL curves of our model-freeframework(Ours-SAC) and model-based framework (Ours-ORCA) are significantly higher andsteeper than the baselines in all three tasks, which shows the effectiveness of our method in arrangingobjects towards the target distribution efficiently.
In the middle row of Fig.
5, Ours-ORCA achievesthe best performance across all three tasks, while Goal-SAC outperforms Ours-SAC on Clusteringand Circling+Clustering since the goal is proposed by a generative model that can naturally generatediverse and realistic goals.
From the qualitative results shown in Fig.
3, our methods also significantlyoutperform the baselines in realism.
Our methods also achieve the lowest ACN in most cases, exceptthat Goal-SAC is slightly better than Ours-SAC in Circling .
The Goal-SAC is the strongest baseline yet loses to our framework in efficiency, since the generatedgoal may be far away from the initial state.
As shown in Fig.
3, the Goal-SAC and Goal-ORCAcan arrange objects in the right trend, but the objects may block each other’s way.
Besides, insupplementary we show the VAE used in Goal-SAC is not guaranteed to generate a legal state( e.g .theballs overlap in goal).
Therefore, the balls may have unexpected behavior during the process.
PL
Circling   30 Number of Objects
Clustering
Circling + Clustering
Ours (ORCA) Goal (ORCA)
Ours (SAC) Goal (SAC) GAIL (SAC) RCE SQIL The classifier-based reward learning method, GAIL, RCE, and SQIL, basically fails in most tasks,as shown in Fig.
3 and Fig.
This is due to the over-exploitation problem that commonly exists inclassifier-based reward learning methods.
We defer the experiments to supplementary.
Room Arrangement: The dynamics of the room-arrangement task are too complex to applyexplicit planning-based methods, so we only compare our model-free framework with learning-basedbaselines.
As shown in Fig.
4 and the last column of Fig.
5, our method significantly outperformsother baselines in sample realism of arrangement results and the safety of the arrangement process,which proves our method can also work on a more complex task.
6.2 Ablation Studies and Analysis
We conduct ablation studies on ball arrangement tasks to investigate: 1) The effectiveness of theexploration guidance provided from the target gradient field and 2)
The necessity of combining thetarget gradient field with control algorithms to adapt to the environment dynamics.
To this end, weconstruct an ablated version named Ours w/o Residual that drops the residual policy learning andanother one named Ours w/o RL that only takes gradient-based action a g as policy.
Ours w/o Residual: The CS-bar in Fig.
5 shows that Ours significantly outperforms the Ours w/o
Residual in CS, which means the agent is easy to over-exploit few modes with high density.
We alsoshow qualitative results in supplementary that without the residual policy learning, the objects areusually arranged into a single pattern.
Ours w/o RL: The PL-curve in Fig.
5 shows that without the control algorithm serving as thebackbone, the arrangement efficiency of Ours w/o RL is significantly lower than Ours.
Meanwhile,the ACN of Ours w/o RL is significantly larger than Ours, which indicates the efficiency drop may bedue to the collisions caused by the gradient-based action.
We further demonstrate the surprising performance of our method in generalization to differentnumbers of objects.
As Fig.
6 illustrates, our framework can generalize to different numbers of ballswith the least performance drop compared with baselines in all three ball arrangement tasks.
Conclusion
In this study, we first analyse the challenge of object arrangement, and then incorporate a gradientfield method with either model-based planning or model-free learning approaches to achieve objectarrangement tasks.
Experiments demonstrate our effectiveness by comparing with learning-basedand planning-based baselines.
Limitation and Future Work.
This work only considers the arrangement in planar environments.
Future works may extend our framework to more realistic scenarios, e.g ., arranging objects in a 3Denvironment, or arranging objects with a real robot.
Ethics Statement.
Our method has the potential to build home-assistant robots, reducing the risk ofobjects being collided.
We evaluate our method in simulated environments, which may introducedata bias.
However, similar studies also have such general concerns.
We do not see any possiblemajor harm of our study.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] see Sec.
(b) Did you describe the limitations of your work?
[Yes] see Sec.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] seesupplementary (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] see Sec.
7 and supplementary
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] please see the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] more details in the supplemental material.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] more details in the supplemental material.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] more details in the supplementalmaterial.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]