Learn2Assemble with Structured Representations and Search for Robotic Architectural Construction
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Autonomous robotic assembly requires a well-orchestrated sequenceof high-level actions and smooth manipulation executions.
Learning to assemblecomplex 3D structures remains a challenging problem that requires drawing connections between target designs and building blocks, and creating valid assemblysequences considering structural stability and feasibility.
To address the combinatorial complexity of the assembly tasks, we propose a multi-head attention graphrepresentation that can be trained with reinforcement learning (RL) to encode thespatial relations and provide meaningful assembly actions.
Combining structuredrepresentations with model-free RL and Monte-Carlo planning allows agents tooperate with various target shapes and building block types.
We design a hierarchical control framework that learns to sequence the building blocks to constructarbitrary 3D designs and ensures their feasibility, as we plan the geometric execution with the robot-in-the-loop.
We demonstrate the ﬂexibility of the proposedstructured representation and our algorithmic solution in a series of simulated 3Dassembly tasks with robotic evaluation, which showcases our method’s ability tolearn to construct stable structures with a large number of building blocks. Videosare available at: https://sites.google.com/view/learn2assemble
Introduction
Construction and manufacturing are becoming increasingly automated in the last decades.
However,there is an essential need for sustainable autonomous architectural assembly [1], where a gamechanger would come with intelligent robot assembly abilities that optimally decide over plans, actions, execution and efﬁciency [2].
In this work, our main focus is the combinatorial optimizationproblem of autonomously assembling complex structures with robotic manipulators without an apriori deﬁned task plan and goal poses for the sequential picking-placing actions.
For constructingabstract target designs, we must consider the combinatorics of the growing action space w.r t. thenumber of available modules and the size of the structure.
Therefore, an effective representation ofthe assembly problem is essential.
Moreover, the geometric execution of the picking and placingactions by the robot imposes constraints to the assembly sequence, as those actions are subject tothe kinematic feasibility in the robot’s workspace.
Eventually, the problem of assembling structureslies in the area of long-horizon manipulation tasks, where most methods in the literature consider aknown task plan, and focus on ﬁne manipulability and structural stability, or learn action sequencesfrom demonstrations [3, 4, 5, 6, 7, 8].
In this paper, we propose a novel algorithmic solution for robotic assembly that combines highlevel decision-making on the construction sequence with the geometric execution by the robot thatshould ensure feasibility and stability.
We propose a graph-based representation that captures therelations between target shapes and available building blocks.
Notably, we design a multi-head attention-based graph neural network (GNN) architecture with a purposefully induced inductive bias forencoding the structural representation of the assembly task.
The GNN is trained through reinforcement learning (RL) to explore feasible actions, resulting in an expressive and ﬂexible representation.
When combined as prior to Monte Carlo Tree Search (MCTS), it extrapolates to out-of-distribution(OOD) assembly tasks, i.e., tasks with a higher number and different types of blocks and vari- able target shapes.
Ultimately, we provide a solution for the motion generation of the sequentialpicking-placing actions when having the robot-in-the-loop to ensure reachable and feasible actions,which do not disrupt the assembly process.
The proposed learn2assemble algorithm provides aﬂexible, autonomous robotic assembly agent for constructing 3D shapes using more than ten blocks.
Our main contributions are threefold.
(1) We propose a multi-headattention-based graph representation for the 3D assembly problemthat is ﬂexible enough for representing arbitrary, stable structures andtheir relations to different building blocks.
(2) We design an integratedlong-horizon manipulation algorithm that learns through exploration ,which combined with model-based search leads to generalizable skills.
Our method considers the robot-in-the-loop , integrating high-level action planning with low-level motion generation for learning policies thatensure the kinematic feasibility and stability of the constructed structure.
Finally, (3) we developed a novel benchmarking environment for 3Drobot assembly , which is modular for testing with and without the robot,for arbitrary target designs, and with adjustable types and number of objects.
Our empirical results on a series of representative experimentsshowcase the generalization power of the proposed algorithmic solution,drawing interesting insights on the combination of integrated learningand planning for long-horizon manipulation that can apply to a range ofrobotic applications.
Related works.
Autonomous robotic assembly is essential for manufacturing and construction, and therefore, many works tried to tacklethe problem of automatizing task execution and ﬁne manipulation.
Autonomous assembly is a long-horizon manipulation task with multiplestages of decisions and subtask executions to be made alongside controlling the dynamic execution of the assembly process.
Thus, researchersproposed methods for Task and Motion Planning (TAMP)
[9], and RL[10] to address the challenges of combinatorial optimization over highlevel action sequencing and low-level motion generation.
In [11, 12] a TAMP method is proposed for robotic architectural construction and extrusion, requiring though exact domain speciﬁcations to ﬁnd an assembly sequence; TAMP with logic-geometricprogramming is proposed by [13] where the focus lies in optimizing the structure’s stability.
In [14],the authors propose a hierarchical planner using hybrid dynamics models for the “toy-airplane”assembly task.
A method based on neuro-symbolic planning is proposed by [15] for learning topredict sequences of actions for stacking.
The authors of [16] propose an end-to-end approach forsequential pick-and-place tasks using shape correspondences and learning to assemble by collectingdemonstrations from a human operator disassembling.
Conversely, we learn2assemble arbitrary designs from scratch, learning both the sequence of actions and discovering goal positions per buildingblock.
Autonomous assembly appeals to the machine learning community due to its combinatorial complexity, which, depending on the structure’s size and availability of building blocks and their possiblecombinations [17], can by far surpass the state-action combinatorics of problems like chess and Go[18, 19].
The relational representation [20, 21] and generalization power of GNNs was thoroughlyexplored in solving combinatorial optimization tasks [22], with successful applications in 2D assembly [23] when combined with RL.
In robot learning, earlier works use deep RL for short-horizonchallenges like peg-in-a-hole [10, 24, 25].
In [26] the use of structured representations in model-free
RL is proposed to induce inductive biases in different stages of a curriculum for executing assemblytasks of building towers of variable heights.
While the general motivation of this work is close toours, [26] does not learn the sequence of the building blocks for constructing arbitrary structures, butit is essentially a goal-based method for learning pick-and-placing manipulation with known goalobject positions.
In a similar direction, the authors of [27] use learning from demonstrations to traintwo GNNs, one that selects objects in the scene and another one that selects a suitable goal statefrom a set of possible goal positions.
Proposed method
The main research question we pose in this work is “how can a robot perform combinatorial assembly tasks of abstract architectural designs?” .
Fig.
1 depicts a typical setup containing a 7DoF manipulator robot, a varying number of building blocks, and one possible architectural target design.
We formalize the task of autonomous construction of abstract targetshapes given some available building blocks as a Markov decision process (MDP) [28].
As illustrated in Figs.
1 & 4 we can build one- up to four-sided structures.
The overall desiredstructure is given by each side’s target topology, which is in turn deﬁned by the coordinates ofthe target points spanning the area to be ﬁlled (see illustrative example in Fig. 2).
The desired shape is thus parameterized by the set of target points S T =
{ x T i | i ∈ N i } (yellow).
The state also contains a set of blocks that are already placed in the scene
S P = { x P j | j ∈ N j } (red) and a set of unplaced blocks S U
= { x U k | k ∈ N k }(gray) that need to be added.
The vector x ∈ R 5 includes the 3D positionand two booleans indicating the element’s properties (placed/unplaced, targetpoint/building block).
The state s t at any time step t is thus given by thethree sets s = ( S T , S P , S U ) and has N = N i + N j + N k elements.
Theobjective is to use the available blocks to construct a stable structure thatﬁlls the desired target design.
We use a discrete action space and deﬁne ﬁverelative placement actions, i.e., any unplaced block can be put on top, onthe left, on the right, behind, in front w.r.t.
any already placed block.
Inthe robot experiments, however, the action space is augmented to also allowthe speciﬁcation of the manipulator’s orientation during grasping and placing.
For most experiments, we choose over two possible grasping and placing orientations, resulting infour grasp-place combinations.
Together with the placement actions, this yields N a = 4 ×
= 20actions when placing one block w.r.t.
another.
Since the possible actions a t depend on the numberof placed and unplaced blocks, we end up with a time-varying action space of size N j × N k × N a .
We assign a positive reward r ( s t , a t ) on all actions that increase the ﬁlling of the target structurewhile preserving its stability.
Given the problem’s combinatorial complexity of discovering theoptimal high-level action plan that will allow the assembly of complex 3D structures which are stableand kinematically feasible, we decompose it into three main problems: (i) ﬁnding an expressiverepresentation for our state space, (ii) decoding states into meaningful actions, and (iii) ensuringstability and kinematic feasibility.
Our research provides a thorough study of how we can tacklethese problems, and we provide a novel 3D assembly algorithm with the robot-in-the-loop throughcombining a learned multi-head attention (MHA) representation with MCTS.
2.1 Multi-head attention graph representation
Graph-based representations [29] are an effective tool when dealing with combinatorial problems 132 [30, 31, 32].
Compared to classical satisﬁability solvers, their main advantage lies in their real-timecapabilities, while their architectural properties allow generalization to problems of different sizes 134 in contrast to most standard neural network architectures, which operate on ﬁxed-size inputs andoutputs.
As those are essential properties for learning to assemble structures of combinatorial com- 136 plexity, we will introduce below our proposed GNN model that is inspired by the combination ofgraphs and attention [33, 34].
GNNs receive as input a graph G =
( N , E ) , and return a high-level 138 encoding over nodes and edges to be further exploited for deciding which action to take.
In our case,the set of nodes N = {S T , S P , S U } = { n i } i =1 ..N is given by the current state, and the connectivity 140 information E is deﬁned as a matrix of size N × N .
If there is an edge connection between nodesn i and n j entry E ( i, j ) equates to 1, otherwise it is 0.
142 Attention mechanisms [33] were introduced in GNNs [34] to enable nodes to attend over theirneighbours’ features and learn different weights for different nodes without requiring costly ma- 144 trix operations.
We want to exploit this ﬂexibility as solving the assembly task necessitates drawingconnections on multiple levels, i.e., between nodes of the same type to encode the already existing 146 structure or the target shape, as well as between all nodes to come up with a meaningful representation for action-decision.
In the following, we will introduce the proposed MHA architecture, that 148 naturally reﬂects the necessary multi-level decision process of the assembly task, and has proven tobe effective when combined with policy search for solving combinatorial problems, like the travelling salesman [31].
In the ﬁrst step of MHA, the initial node embeddings n (0) i
= x i are projected 151 into a higher dimensional space by
[Equation]
[Figure]
using a fully-connected (FC) layer followed by a rectiﬁed linear unit (ReLU) activation function.
153 Note that the function g will be used repeatedly as we progress with the graph update, yet, assuming different weights on each further appearance.
Next follow L rounds of message passing, 155 i.e.,applying (2) L times to obtain the ﬁnal embedding for each node i according to
[Equation]
with a skip connection layer h
( f ( x ))
=
x + f ( x ) , the current round of message passing l , allnode embeddings from the previous round N ( l − 1) , and a MHA mechanism introduced below.
Asillustrated in Fig.
3, for a single out of the M attention heads with index o , we ﬁrst compute three 159 values – the key k , query q , and value v – using three different weight matrices W k,o , W q,o , W v,o ,respectively ( k i,o = W k,o n i
( l − 1) , q i,o = W q,o n i ( l − 1) , v i,o =
W v,o n i
( l − 1) ).
Multiplying the keyand query of all nodes results in a compatibility score
c i,j for the
i th -to- j th node connection
[Equation]
otherwise , (
[Equation]
with d a normalizing constant.
From this score, we can then compute the attention weights usinga softmax a i,j,o =
[Equation]
(cid:80) j (cid:48)
e ci,j (cid:48) ,o to aggregate the values for each node, resulting in the output messagem i,o = (cid:80) j (cid:48) a i,j,o v j,o .
A weighted sum of all messages yields the ﬁnal result of the MHA module
MHA ( N ( l − 1) , i ) =
M (cid:88) o =
W m,o m i,o , (4) with weights W m,o controlling the inﬂuence of each one of the single attention heads.
Fig.
3 depictsa simpliﬁed example of the encoding performed by the MHA architecture.
2.2 Learning assembly policies
MHA-GNNs can only unfold their potential when combined with algorithms that reﬁne their weights 169 to form expressive representations exploited herein for action selection (Fig. 3).
The GNN shouldthus be shaped based on the reward signal, resulting in a RL setup with the goal of obtaining per- 171 formant policies.
This powerful combination results in agents that (i) can be applied to differentproblem instances due to the representation’s ﬂexibility, (ii) are reactive, despite the problem’s com- 173 binatorial complexity, and (iii) can be trained directly in simulation environments which include thenonlinearities of the robot and the contacts.
Due to the problem’s combinatorial complexity and its 175 discrete, time-varying action space, we use model-free Q-learning, [35] which has been successfullyapplied to complex tasks such as playing Atari games from images.
Moreover, we investigate its in- 177 tegration with model-based planning, as the addition of search can counteract the overoptimism ofthe Q-function approximation and result in more robust behaviour [18, 36].
179 Action Decoding.
The encoded graph representation from Sec.
2.1 needs further processing to decide on the next action to take.
As actions are deﬁned relatively between unplaced and placed blocks 181 (i.e., nodes) we can directly assign a value to all available actions
[Equation]
with the total number of N nodes in the graph.
Note that the Q-value does not only depend on thetwo nodes’ embedding, but also on a global feature based on averaging over all embeddings.
As allthe operations are deﬁned over the set of nodes, this encoding-decoding architecture can seamlesslygeneralize to different problem sizes, i.e.,different number of blocks or target shapes.
DQN:
For our setting, we deﬁne the loss function as the smooth L1 loss between the current actionvalue estimate of the GNN, noted as Q ( n i , n j , a s ) and the value obtained from the rollouts using a target network, noted as Q T , i.e., ˆ Q ( n i , n j , a s ) =
r ( s t , a t ) + γ max n i ∈S U ,n j ∈S P Q T ( n i , n j , a s ) withdiscount factor γ and the selected action a s ∈ N a .
MCTS: MCTS [37] has proven effective for solving tasks with discrete action spaces [38], hence,suitable for our problem.
Contrary to model-free RL, MCTS relies on a model of the environmentto perform tree search for action selection and does not require any form of function approximation.
However, in scenarios with signiﬁcant branching factors or expensive model evaluations, the timecomplexity quickly increases, and the use of MCTS becomes intractable.
Consequently, [18, 19]exploit Q-learning for estimating the value of the nodes in MCTS and show impressive results insolving tasks of great complexity, like the game of Go.
Also, in the context of accelerating andgeneralizing skill-learning, the combination of learning and planning has received increasing attention [19, 36, 39, 40].
In this work, inspired by these advantages, we explore different algorithmicvariations of the interplay between Q-learning and planning, as described below.
Pseudocode for allvariants is given in Appendix B.1)
DQN+MCTS: Inspired by [18], we propose a variant of AlphaGo, that uses a pretrained Qnetwork as a prior for evaluating the leaf nodes of MCTS, leading to increased efﬁciency.
As weseek to effectively combine learning and planning for autonomous assembly with the robot-in-theloop, we use small search budgets for which the prominent solution for MCTS expansion - theexploration strategy of UCT [18, 41] - is unsuitable because it becomes over-optimistic [42].
Therefore, we use an (cid:15) -greedy expansion strategy during search, that allows better exploration than UCT.2) Q-MCTS: We follow a generalization of Q-learning using samples based on planning.
As in[19, 36], we combine DQN with MCTS during training to perform an informed exploration andcollect good samples through search.
However, as motivated previously, due to small search budgets, our approach uses an (cid:15) -greedy expansion strategy over unexplored states, instead of UCT.
Let Q S ( s t , a t ) =
Q S ( s t , a t ) + r ( s t , a t ) + γ max a (cid:48)
t Q ( s (cid:48) t , a (cid:48) t ) be the updated value of an expandednode, where s (cid:48) t is the new state arriving during search.
The resulting Q-MCTS methods augmentsthe DQN learning objective with a cross-entropy loss deﬁned on the values explored during search,i.e., loss = − softmax ( Q S ( s t , a t ))
T log( softmax ( Q ( s t , a t ))) which is intended to regularize andimprove the Q-function estimate based on the experience collected while searching.3) (cid:15) -MCTS:
This implementation follows [36] more closely.
The main differences w.r.t.
the QMCTS policy are that ﬁrst, there is an (cid:15) -greedy decision on whether to do search or uniformly sample a random action, whereas Q-MCTS always conducts search.
Secondly, during search, (cid:15) -MCTSfollows the UCT expansion strategy.
The Q-learning objective remains the same as with Q-MCTS,however, the cross-entropy loss is computed only on the samples where search has been conducted.
In essence, the method’s difference is in the way of collecting the model-based samples.
2.3 Integrated learning and planning for robotic assembly
All previous components can now be combined to obtain an algorithm capable of training agentsto build desired target shapes, i.e., a robot capable of abstracting the sequence of actions for building arbitrary stable target structures from individual elements while executing feasible actions in itsworkspace.
Note that the only control component we assume given is a point-to-point control strategy based on the robot’s inverse kinematics.
To enable this combined decision-making strategy thattouches on the ground of TAMP literature, we propose the combination of the previously describedgraph representations and learning algorithms to provide a novel learn2assemble method.
Brieﬂy, asingle step of a learning episode starts with selecting actions (with or without tree-search dependingon the learning algorithm), i.e., the next object to be placed, the grasping pose, the goal position, andorientation.
A path from the picking to the placing position is computed, and the robot executes theplacement, for which it receives a reward.
Consequently, we store the current graph’s state, action,reward, and new state in the replay memory to be later used for learning.
Experimental Results
For evaluating the different components of the proposed method and their respective contribution,we designed speciﬁc experimental scenarios.
We start with an investigation over graph architectures,continuing with the different learning methods in environments for 3D assembly without includinga robot yet.
Selecting the best settings from the previous tests, we experiment with the robot-in-theloop for our ﬁnal empirical evaluations of the proposed algorithm.
[Figure]
[Table]
Simulation environments. The assembly environments are depicted in Fig.
We investigate, one-,two- and four-sided setups with each side of the target shape deﬁned by the position of 3 (see Fig.2), or 5 target points.
The number of target points and their locations are sampled randomly fromgrids of different sizes, ranging from 3-by-3 (i e., the sampled target points can at maximum spanan area of height and width of 3 times the cube’s edge length) to 6-by-6.
Due to the relative actionspace, every scene is always initialized with one initially placed element, marked in red, servingas the building base.
For evaluating the assembly progression, we use depth cameras, placed oneach side of the speciﬁed target structure.
By projecting the target points into the images after eachaction, we obtain the change in the target shape’s ﬁlling.
The reward functions endorse actions thatlead to improvement in the ﬁlling (cf.
Appx. C.2).
The construction process is ﬁnished, once thetotal coverage exceeds a threshold, whenever there are no more unplaced blocks available, or uponexecuting an invalid action, i.e., an action resulting in an unstable conﬁguration, in destructing thecurrent structure, or if it is kinematically infeasible.
In all environments without the robot, the placing is done by directly specifying one of the ﬁve placement actions, resulting in a reduced actionspace of N a = 5 .
If not stated differently we use partially connected graphs (see Fig. 3), inducinga stronger inductive bias on the structured representation compared to fully connected (FC) graphs(cf.
Appx.
A.3 & D.3).
In the following tables, for evaluating the agent’s performance, we report thecumulative discounted return R , the ratio of runs that ended with failure, i.e., upon an invalid actionf , the ratio of runs that ended without success and no more blocks remaining b , as well as the meannumber of actions conducted per run ¯ a .
The star(*) marks the agents’ evaluation in the same settingas in training, while the rest are OOD experiments, i.e., exclusively evaluating the agents in settingswith previously unseen target shapes or number of blocks. For more details, see Appx. C.
Graph Architectures.
We evaluate the proposed MHA representation against the commonly used
Structure2Vector (S2V) architecture [23, 30] (cf.
Appx. A.1) in a simple environment (see Fig.
4a)only considering one type of object but omitting the robot.
The learning is conducted with DQN.
Results.
As shown in the ﬁrst column of Table 1, already in the original training environment, the MHA approach outperforms S2V signiﬁcantly.
The high rates of failure and exceeding the numberof available blocks indicate that S2V cannot draw the connection between the target shape and thecurrent structure.
This might be due to S2V’s different message passing, which cannot weigh theimportance of different nodes as with the attention mechanism.
When increasing the number ofavailable blocks and the size of the structure to be built (columns 3 & 4), we see an evident advantage of MHA in handling OOD tasks.
In Appx.
D.2.1, we provide additional results when usingonly a single attention head, which conﬁrm that using attention is advantageous, and MHA yieldsthe best performance.
We, thus, continue our experimentation using the MHA architecture.
Learning algorithms.
To investigate the performance of the learning algorithms (Sec. 2.2), wewill use the two-sided environment shown in Fig.
4b without the robot, thus using the reduced actionspace.
Results.
Table 2 summarizes the results, starting without any search budget (i.e., no tree search)to evaluate the learned Q-functions.
The DQN and (cid:15) -MCTS agents perform similarly, with DQNslightly outperforming (cid:15)
-MCTS through lower failure rates across tasks.
Moreover, DQN can solve [Table] the task with fewer actions.
Contrarily, the Q-MCTS agents perform signiﬁcantly worse.
We believethat this difference in performance is due to the constant cross-entropy regularization in Q-MCTSfrom the beginning of the training, especially when search samples might be bad, while for the (cid:15) -
MCTS agent the regularization is only slowly added as training proceeds and less random actions aretaken (search is better), which seems to be beneﬁcial.
Note that without the addition of search, increasing the number of blocks, as well as the target size, results in a quick increase of the failure ratefor all methods.
Adding a search budget of 10 already counteracts this trend, especially consideringthe DQN agent, where the rates of failure can be reduced for all the tasks by using DQN+MCTS attest time.
For the Q-MCTS agent, the same trend is noticed, while the gains of the (cid:15) -MCTS agentare marginal.
This suggests that the (cid:15) -MCTS agent is overoptimistic and seems to choose similar actions as to when not using search at all.
Overall, combining MCTS with a pretrained DQN results inthe best performance in our experiments.
The last row of the table underlines the problem’s combinatorial complexity, illustrating that performing pure UCT without any prior and a search budget of1000 performs signiﬁcantly worse for the simplest experimental setting (more details in Appx. D.4).
Learn2assemble with the robot-in-the-loop.
Next, we evaluate the DQN agent combined withtree-search in our target environments, including the robot manipulator.
The task’s difﬁculty increases, as also the grasping and placing poses have to be speciﬁed while ensuring action feasibilityby the robot and the structure’s stability.
We start with constructing single-sided designs (Fig. 4a) toillustrate the necessity of training with the robot-in-the-loop before evaluating the proposed methodwith multi-sided designs (Fig. 4b & 4c).
Results.
We ﬁrst compare two policies, one trained with, and the other without the robot using plain DQN without any search in a single-sided environment (Fig. 4a).
The agent trainedwith the robot-in-the-loop outperforms the other one resulting in a signiﬁcantly reduced failure rate of 15 % and consistently higher rewards (cf. Appx. D.5).
This shows the necessity of including geometric planning during training for obtaining high-level decisions that arecompatible with the low-level execution.
It is thus not sufﬁcient to only ﬁgure out whereto place the parts; the kinematic constraints and robot motion have to be considered.
In thesubsequent experiments, we build 3D structures with the robot as shown in Figs. 4b & 4c.
[Table]
Our results in Tables 3 and 4 show that MCTS (search budget of 10) consistentlyimproves performance in terms of higherreturns and lower failure rate.
In particular, the experiments demonstrate that ourproposed pipeline can execute multiple sequential pick-and-placing actions, with amaximum of 17 correctly placed blocks for the 5-by-5 grid in the two-sided environment and up to22 correct placements in the four-sided environment, using DQN+MCTS with the robot-in-the-loop.
Compared to the previous experiments, without the robot-in-the-loop, the failure rate is higher, indicating the task’s increased difﬁculty, and the need of adding a soft-placing controller. Generalization w.r.t. randomized scenes.
To evaluate our algorithm’s robustness w.r.t.
changesin the scene, we transfer the previously trained policies and evaluate them in scenarios where theunplaced blocks are placed randomly around the structure to be built, as shown in Fig. 4d. Results. Rows 3 & 4 of Table 4 reveal that the policies indeed generalize to these novel scenarios,as the percentage of unsuccessful experiments only increases at maximum by 11 % for the mostcomplex scenario, compared to their performance in the original environment (rows 1 & 2).
Thisconﬁrms that our proposed method does not overﬁt the exact layout or geometry, but rather builds
[Table]
[Table]
meaningful features that allow to successfully transfer the behaviour to scenes that substantially
[Table]
different from those encountered during
Generalization w.r.t.
different building blocks. We ﬁnally investigate our method’s performancewhen using more complex objects (Fig. 4e). This makes the task signiﬁcantly more difﬁcult, as theagent has to not only learn each part’s admissible grasps but also differentiate the building blocks toselect and place the correct object type. All novel objects are a combination of primitive boxes whichallows keeping the relative action space, i.e., placing an unplaced primitive box belonging to a largerobject w.r.t.
a placed primitive box results in moving the entire object.
For the experiments withoutthe robot, we adjust the action space to enable changing the object’s orientation (cf.
Appx.
D.9).
Results.
The results in Table 5 illustrate the representation’s ﬂexibility and ability to successfullydeal with the different available building blocks.
Despite the increased complexity, we achieve similar performance in the scenes without the robot as in previous experiments (Table 2).
When trainingwith the robot, we can still handle the task’s complexity with a ∼ 80 % success rate in the simplestsetting, but observe a drop in performance with a larger number of objects and bigger target shapes.
While one cause for the performance drop is the setting’s increased complexity, there are also severalplacement actions that would require 3D gripper orientation control allowing for a smooth insertionof the complex blocks.
As our current top-down placing controller only offers planar orientationcontrol, some placement actions cannot be executed appropriately resulting in increased failures.
Remarks. We have conducted extensive experiments to show the superiority of our proposed MHAGNN approach for solving combinatorial assembly tasks with the robot-in-the-loop.
The resultsdemonstrate how strong inductive biases combined with attention can shape meaningful relationalrepresentations.
When combined with deep Q-learning, this representation allows us to take decisions over long horizons despite an increasing action space.
Adding search at test time improvesperformance across all experiments, demonstrating that the proposed method can generalize w.r.t.different target shape sizes, number of building blocks, and different scenes.
While we show verypromising results in combining learning high-level action decisions with planning geometric executions, we can see a combinatorial barrier in the decision-making that might be tackled with amore informed search in the graph space.
Our experiments on using different objects underline therepresentation’s ﬂexibility, but also reveal current limitations in the deﬁnition of the action space,especially considering the robotic execution, which may be mitigated by enriching the action spacethrough 6D grasping and placing.
Moreover, several assemblies failed due to “rough” robot actions,meaning that a more sophisticated motion generator might be needed for ﬁner placement.
Conclusion
We presented a new learn2assemble algorithm for learning autonomous robotic 3D assembly fromscratch without prior knowledge of any task plan.
For addressing the problem’s combinatorial complexity while maintaining adaptability to different scenarios, we propose a graph-based multi-headattention representation that captures the spatial relationships between target construction designsand unplaced blocks, and is trained through deep Q-learning. The powerful representation forms thebasis for our hierarchical controller that jointly conducts high-level learning over action sequencesand goal speciﬁcations together with low-level path planning, ensuring the execution of long-horizontasks. Our extensive experiments conﬁrm the representation’s effectiveness and show extrapolationto environments with previously unseen target shapes, larger numbers of available elements, anddifferent object types.
When combining the learned Q-network with MCTS with computationallytractable small search budgets, we manage to improve performance and reliability across all tasks.
Notably, we resolve the sequential long-horizon character of the assembly task by including therobot-in-the-loop to decide over feasible grasps and placing actions that ensure the stability of theconstruction.
Our algorithm manages to correctly build structures using up to 22 building blockswith good success rates.
In the future, we want to extend the algorithm to allow for a richer set of6D grasping and placing poses, learn ﬁne-placing or even in-hand manipulation controllers on thelow level, and investigate the implementation of an assembly/disassembly strategy, so that the robotcan potentially re-use wrongly placed blocks or reconﬁgure existing structures.