Learn2Assemble with Structured Representations and Search for Robotic Architectural Construction
Anonymous Author(s)
AfﬁliationAddress email
Abstract: Autonomous robotic assembly requires a well-orchestrated sequenceof high-level actions and smooth manipulation executions.
The problem of learning to assemble complex 3D structures remains challenging, as it requires drawing connections between target shapes and available building blocks, as well ascreating valid assembly sequences with respect to stability and kinematic feasibility in the robot’s workspace.
We design a hierarchical control frameworkthat learns to sequence the building blocks to construct arbitrary 3D designsand ensures that they are feasible, as we plan the geometric execution with therobot-in-the-loop.
Our approach draws its generalization properties from combining graph-based representations with reinforcement learning (RL) and ultimatelyadding tree-search.
Combining structured representations with model-free RL and
Monte-Carlo planning allows agents to operate with various target shapes andbuilding block types.
We demonstrate the ﬂexibility of the proposed structuredrepresentation and our algorithmic solution in a series of simulated 3D assemblytasks with robotic evaluation, which showcases our method’s ability to learn toconstruct stable structures with a large number of building blocks.
Introduction
Construction and manufacturing are becoming increasingly automated in the last decades. However,there is an essential need for sustainable autonomous architectural assembly [1], where a gamechanger would come with intelligent robot assembly abilities that optimally decide over plans, actions, execution, and efﬁciency [2].
In this work, our main focus is the combinatorial optimizationproblem of autonomously assembling complex structures with robotic manipulators without an apriori deﬁned task plan and goal poses for the sequential picking-placing actions.
For constructingabstract target designs, we must consider the combinatorics of the growing action space w.r t. to thenumber of available modules and the size of the structure.
Therefore, the effective representationof the assembly problem is essential.
Moreover, the geometric execution of the picking and placingactions by the robot imposes constraints to the assembly sequence, as those actions are subject tothe kinematic feasibility in the robot’s workspace.
Eventually, the problem of assembling structureslies in the area of long-horizon manipulation tasks, where most methods in the literature consider aknown task plan, and focus on ﬁne manipulability and structure stability, or learn action sequencesfrom demonstrations [3, 4, 5, 6, 7, 8].
In this paper, we propose a novel algorithmic solution for robotic assembly that combines highlevel decision-making on the construction sequence with the geometric execution by the robotthat should ensure feasibility and stability.
We propose a graph-based representation that captures the relations between target shapes and available building blocks.
Notably, we designa multi-head attention-based graph neural network (GNN) architecture with a purposefully induced inductive bias for encoding the structural representation of the assembly task.
The GNNis trained through reinforcement learning (RL) to explore feasible actions, resulting in an expressive and ﬂexible representation.
When combined as prior to Monte Carlo Tree Search(MCTS), it extrapolates to out-of-distribution (OOD) assembly tasks, i.e., tasks with a highernumber and different types of blocks and variable target shapes.
Ultimately, we provide a robot-in-the-loop solution for the motion generation of the sequential picking-placing actionsto ensure that the robot decides on reachable and feasible actions, which do not disrupt theassembly process.
The proposed learn2assemble algorithm provides a ﬂexible, autonomousrobotic assembly agent for constructing 3D shapes using more than ten blocks in the scene.
Our main contributions are threefold.
(1) We propose a multi-headattention-based graph representation for the 3D assembly problemthat is ﬂexible enough for representing arbitrary, stable structures andtheir relations to different building blocks.
(2) We design an integratedlong-horizon manipulation algorithm that learns through exploration ,which combined with model-based search leads to generalizable skills.
Our method considers the robot-in-the-loop , integrating high-level action planning with low-level motion generation for learning policies thatensure the kinematic feasibility and stability of the constructed structure.
Finally, (3) we developed a novel benchmarking environment for 3Drobot assembly , which is modular for testing with and without the robot,for arbitrary target designs, and with adjustable types and number of objects.
Our empirical results on a series of representative experimentsshowcase the generalization power of the proposed algorithmic solution,drawing interesting insights on the combination of integrated learningand planning for long-horizon manipulation that can apply to a range ofrobotic applications.
Related works.
Autonomous robotic assembly is essential for manufacturing and construction, and therefore, many works tried to tacklethe problem of automatizing task execution and ﬁne manipulation.
Autonomous assembly is a long-horizon manipulation task with multiplestages of decisions and subtask executions to be made alongside controlling the dynamic execution of the assembly process.
Thus, researchersproposed methods for Task and Motion Planning (TAMP)
[9], and RL[10] to address the challenges of combinatorial optimization over highlevel action sequencing and low-level motion generation.
In [11, 12] a TAMP method is proposed for robotic architectural construction and extrusion, requiring though speciﬁc domain speciﬁcations to ﬁnd an assembly sequence; TAMP with logic-geometricprogramming is proposed by [13] where the focus lies in optimizing the structure’s stability.
In [14],the authors propose a hierarchical planner using hybrid dynamics models for the “toy-airplane”assembly task.
A method based on neuro-symbolic planning is proposed by [15] for learning topredict sequences of actions for stacking.
The authors of [16] propose an end-to-end approach forsequential pick-and-place tasks using shape correspondences and learning to assemble by collectingdemonstrations from disassembling from a human operator.
Conversely, we learn2assemble arbitrary designs from scratch, learning both the sequence of actions and discovering goal positions perbuilding block.
Autonomous assembly appeals to the machine learning community due to its combinatorial complexity, which, depending on the structure’s size and availability of building blocks and their possiblecombinations [17], can by far surpass the state-action combinatorics of problems like chess and Go[18, 19].
The relational representation [20, 21] and generalization power of GNNs was thoroughlyexplored in solving combinatorial optimization tasks [22], with successful applications in 2D assembly [23] when combined with RL.
In robot learning, earlier works use deep RL for short-horizonchallenges like peg-in-a-hole [10, 24, 25].
In [26] the use of structured representations in model-free RL is proposed to induce inductive biases in different stages of a curriculum for executing assemblytasks of building towers of variable heights.
While the general motivation of this work is close toours, [26] does not learn the sequence of the building blocks for constructing arbitrary structures, butit is essentially a goal-based method for learning pick-and-placing manipulation with known goalobject positions.
In a similar direction, the authors of [27] use learning from demonstrations to traintwo GNNs, one that selects objects in the scene and another one that selects a suitable goal statefrom a set of possible goal positions.
Proposed method
The main research question we pose in this work is “how can a robot perform combinatorial assembly tasks of abstract architectural designs?” .
Fig.
1 depicts a typical setup containing a 7DoF manipulator robot, a varying number of building blocks, and one possible architectural target design.
We formalize the task of autonomous construction of abstract targetshapes given some available building blocks as a Markov decision process (MDP) [28].
As illustrated in Figs. 1 & 4 we can build one- up to four-sided structures. The overall desiredstructure is given by each side’s target topology, which is in turn deﬁned by the coordinates ofthe target points spanning the area to be ﬁlled (see illustrative example in Fig. 2). The desired shape is thus parametrized by the set of target points S T
= { x T i | i ∈ N i } (yellow). The state also contains a set of blocks that are already placed in the scene S P = { x P j | j ∈ N j } (red) and a set of unplaced blocks S U
= { x U k | k ∈ N k }(gray) that need to be added. The vector x ∈ R 5 includes the 3D positionand two booleans indicating the element’s properties (placed/unplaced, targetpoint/building block).
The state s t at any time step t is thus given by thethree sets s = ( S T , S P , S U ) and has N = N i + N j + N k elements.
Theobjective is to use the available blocks to construct a stable structure thatﬁlls the desired target design.
We use a discrete action space and deﬁne ﬁverelative placement actions, i.e. any unplaced block can be put on top, onthe left, on the right, behind, in front w.r.t.
any already placed block.
Inthe robot experiments, however, the action space is augmented to also allowthe speciﬁcation of the manipulator’s orientation during grasping and placing.
For most experiments, we choose over [-N a = 2 × 2 × 5 = 20 actions per block. 
 However, since the-]two possible grasping and placing orientations, resulting infour grasp-place combinations.
{+Together with the placement actions, this yields N a = 4 × 
+}
= 20actions when placing one block w.r.t.
another.
Since the possible actions a t depend on the numberof placed and unplaced blocks, we end up with a time-varying action space of size N j × N k × N a . We assign a positive reward r ( s t , a t ) on all actions that increase the ﬁlling of the target structurewhile preserving its stability.
Given the problem’s combinatorial complexity of discovering theoptimal high-level action plan that will allow the assembly of complex 3D structures which are stableand kinematically feasible, we decompose it into three main problems: (i) ﬁnding an expressiverepresentation for our state space, (ii) decoding states into meaningful actions, and (iii) ensuringstability and kinematic feasibility.
Our research provides a thorough study of how we can tacklethese problems, and we provide a novel 3D assembly algorithm with the robot-in-the-loop throughcombining a learned multi-head attention (MHA) representation with MCTS.
2.1 Multi-head attention graph representation
Graph-based representations [29] are an effective tool when dealing with combinatorial problems 132 [30, 31, 32].
Compared to classical satisﬁability solvers, their main advantage lies in their real-timecapabilities, while their architectural properties allow generalization to problems of different sizes 134 in contrast to most standard neural network architectures, which operate on ﬁxed-size inputs andoutputs.
As those are essential properties for learning to assemble structures of combinatorial com- 136 plexity, we will introduce below our proposed GNN model that is inspired by the combination ofgraphs and attention [33, 34].
GNNs receive as input a graph G =
( N , E ) , and return a high-level 138 encoding over nodes and edges to be further exploited for deciding which action to take.
In our case,the set of nodes N = {S T , S P , S U } = { n i } i =1 ..N is given by the current state, and the connectivity 140 information E is deﬁned as a matrix of size N × N .
If there is an edge connection between nodesn i and n j entry E ( i, j ) equates to 1, otherwise it is 0.
142 Attention mechanisms [33] were introduced in GNNs [34] to enable nodes to attend over their neighbors’ features and learn different weights for different nodes without requiring costly matrix opera- 144 tions.
We want to exploit this ﬂexibility as solving the assembly task necessitates drawing connections on multiple levels, i.e. between nodes of the same type to encode the already existing structure 146 or the target shape, as well as between all nodes to come up with a meaningful representation foraction-decision. In the following, we will introduce the proposed MHA architecture, that naturally 148 reﬂects the necessary multi-level decision process of the assembly task, and has proven to be effective when combined with policy search for solving combinatorial problems, like the travelingsalesman [31].
In the ﬁrst step of MHA, the initial node embeddings n (0) i
= x i are projected into a 151 higher dimensional space by
[Equation]
[Figure]
using a fully-connected (FC) layer followed by a rectiﬁed linear unit (ReLU) activation function.
153 Note that the function g will be used repeatedly as we progress with the graph update, yet, assuming different weights on each further appearance.
Next follow L rounds of message passing, i.e. 155 applying (2) L times to obtain the ﬁnal embedding for each node i according to
[Equation]
with a skip connection layer h
( f ( x ))
=
x + f ( x ) , the current round of message passing l , allnode embeddings from the previous round N ( l − 1) , and a MHA mechanism introduced below.
Asillustrated in Fig.
3, for a single out of the M attention heads with index o , we ﬁrst compute three 159 values – the key k , query q , and value v – using three different weight matrices W k,o , W q,o , W v,o ,respectively ( k i,o = W k,o n i
( l − 1) , q i,o = W q,o n i ( l − 1) , v i,o =
W v,o n i ( l − 1) ).
Multiplying the keyand query of all nodes results in a compatibility score c i,j for the i th -to- j th node connection
[Equation] otherwise , ( [Equation] with d a normalizing constant.
From this score, we can then compute the attention weights usinga softmax a i,j,o = [Equation]
j (cid:48) e ci,j (cid:48) ,o to aggregate the values for each node, resulting in the output messagem i,o = (cid:80) j (cid:48) a i,j,o v j,o .
A weighted sum of all messages yields the ﬁnal result of the MHA module MHA ( N ( l − 1) , i ) =
M (cid:88) o =
W m,o m i,o , (4) with weights W m,o controlling the inﬂuence of each one of the single attention heads.
Fig.
3 depictsa simpliﬁed example of the encoding performed by the MHA architecture.
2.2 Learning assembly policies
MHA-GNNs can only unfold their potential when combined with algorithms that reﬁne their weights 169 to form expressive representations exploited herein for action selection (Fig. 3).
The GNN shouldthus be shaped based on the reward signal, resulting in a RL setup with the goal of obtaining per- 171 formant policies.
This powerful combination results in agents that (i) can be applied to differentproblem instances due to the representation’s ﬂexibility, (ii) are reactive, despite the problem’s com- 173 binatorial complexity, and (iii) can be trained directly in simulation environments which include thenonlinearities of the robot and the contacts.
Due to the problem’s combinatorial complexity and its 175 discrete, time-varying action space, we use model-free Q-learning, [35] which has been successfullyapplied to complex tasks such as playing Atari games from images.
Moreover, we investigate its in- 177 tegration with model-based planning, as the addition of search can counteract the overoptimism ofthe Q-function approximation and result in more robust behavior [18, 36].
179 Action Decoding.
The encoded graph representation from Sec.
2.1 needs further processing to decide on the next action to take.
As actions are deﬁned relatively between unplaced and placed blocks 181 (i.e. nodes) we can directly assign a value to all available actions [Equation] [Table]
[Equation]
with the total number of N nodes in the graph.
Note that the Q-value does not only depend on thetwo nodes’ embedding, but also on a global feature based on averaging over all embeddings.
As allthe operations are deﬁned over the set of nodes, this encoding-decoding architecture can seamlesslygeneralize to different problem sizes, i.e. different number of blocks or target shapes.
DQN:
For our setting, we deﬁne the loss function as the smooth L1 loss between the current actionvalue estimate of the GNN, noted as Q ( n i , n j , a s ) and the value obtained from the rollouts using a target network, noted as Q T , i.e., ˆ Q ( n i , n j , a s ) =
r ( s t , a t ) + γ max n i ∈S U ,n j ∈S P Q T ( n i , n j , a s ) withdiscount factor γ and the selected action a s ∈ N a .
MCTS: MCTS [37] has proven effective for solving tasks with discrete action spaces [38], hence,suitable for our problem.
Contrary to model-free RL, MCTS relies on a model of the environmentto perform tree search for action selection and does not require any form of function approximation.
However, in scenarios with signiﬁcant branching factors or expensive model evaluations, the timecomplexity quickly increases, and the use of MCTS becomes intractable.
Consequently, [18, 19]exploit Q-learning for estimating the value of the nodes in MCTS and show impressive results insolving tasks of great complexity, like the game of Go.
Also, in the context of accelerating andgeneralizing skill-learning, the combination of learning and planning has received increasing attention [19, 36, 39, 40].
In this work, inspired by these advantages, we explore different algorithmicvariations of the interplay between Q-learning and planning, as described below.
Pseudocode for allvariants is given in Appendix B.1)
DQN+MCTS: Inspired by [18], we propose a variant of AlphaGo, that uses a pretrained Qnetwork as a prior for evaluating the leaf nodes of MCTS, leading to increased efﬁciency.
As weseek to effectively combine learning and planning for autonomous assembly with the robot-in-theloop, we use small search budgets for which the prominent solution for MCTS expansion - theexploration strategy of UCT [18, 41] - is unsuitable because it becomes over-optimistic [42].
Therefore, we use an (cid:15) -greedy expansion strategy during search, that allows better exploration than UCT.2) Q-MCTS: We follow a generalization of Q-learning using samples based on planning.
As in[19, 36], we combine DQN with MCTS during training to perform an informed exploration andcollect good samples through search.
However, as motivated previously, due to small search budgets, our approach uses an (cid:15) -greedy expansion strategy over unexplored states, instead of UCT.
Let
Q S ( s t , a t ) =
Q S ( s t , a t )+ r ( s t , a t )+ γ max a (cid:48)
t Q ( s (cid:48) t , a (cid:48) t ) be the updated value of an expanded node,where s (cid:48) t is the new state arriving during search.
The resulting Q-MCTS methods augments the
DQN learning objective with a cross-entropy loss deﬁned on the values explored during search, i.e.loss = − softmax ( Q S ( s t , a t ))
T log( softmax ( Q ( s t , a t ))) which is intended to regularize andimprove the Q-function estimate based on the experience collected while searching.3) (cid:15) -MCTS:
This implementation follows [36] more closely.
The main differences w.r.t.
the QMCTS policy are that ﬁrst, there is an (cid:15) -greedy decision on whether to do search or uniformly sample a random action, whereas Q-MCTS always conducts search.
Secondly, during search, (cid:15) -MCTSfollows the UCT expansion strategy.
The Q-learning objective remains the same as with Q-MCTS,however, the cross-entropy loss is computed only on the samples where search has been conducted.
In essence, the method’s difference is in the way of collecting the model-based samples.
2.3 Integrated learning and planning for robotic assembly
All previous components can now be combined to obtain an algorithm capable of training agents tobuild desired target shapes, i.e., a robot able to abstract the sequence of actions for building arbitrarystable target structures from individual elements while executing feasible actions in its workspace.
Note that the only control component we assume given is a point-to-point control strategy basedon the robot’s inverse kinematics.
To enable this combined decision-making strategy that toucheson the ground of TAMP literature, we propose the combination of the previously described graphrepresentations and learning algorithms to provide a novel learn2assemble method.
Brieﬂy, a singlestep of a learning episode starts with selecting actions (with or without tree-search depending onthe learning algorithm), i.e., the next object to be placed, the grasping pose, the goal position, andorientation.
A path from the picking to the placing position is computed, and the robot executes theplacement, for which it receives a reward.
Consequently, we store the current graph’s state, action,reward, and new state in the replay memory to be later used for learning.
Experimental Results
For evaluating the different components of the learning algorithm and their respective contribution,we designed speciﬁc experimental scenarios.
We start with an investigation over graph architecturesand connectivity, continuing with the different learning methods in environments for 3D assemblywithout including a robot yet.
Selecting the best settings from the previous tests, we experimentwith the robot-in-the-loop for our ﬁnal empirical evaluations of the proposed algorithm.
Simulation environments.
The assembly environments are depicted in Fig.
As introduced in [Figure] [Table]
[Table] [Table] Sec. 2, they all consist of a set of placed and unplaced blocks, as well as the target points.
We investigate, one-, two- and four-sided setups with each side of the target shape deﬁned by the position of3 (see Fig. 2), 4, or 5 target points.
The number of points and their locations are sampled randomlyfrom grids of different sizes, ranging from 3-by-3 (i.e. maximum height and width of the grid is3 blocks) to 6-by-6.
Due to the relative action space, every scene is always initialized with oneinitially placed element, marked in red, serving as the building base.
For evaluating the assemblyprogression, we use depth cameras, placed on each side of the speciﬁed target structure.
By projecting the target points into the images after each action, we obtain the change in the target shape’sﬁlling. If not stated differently, we assign a reward of +1, if the ﬁlling increased, of -1 if the action isinvalid, i.e. results in an unstable conﬁguration or is kinematically infeasible, and 0 otherwise.
Theconstruction process is ﬁnished, once the total coverage exceeds a threshold, whenever there are nomore unplaced blocks available, or upon executing an invalid action.
In all environments withoutthe robot, the placing is done by directly specifying one of the ﬁve placement actions, resulting in areduced action space of N a = 5 .
In the result tables, the star(*) indicates the agents’ evaluation inthe training environment, while the other results are from OOD experiments, i.e., exclusively evaluating the agents in settings with previously unseen target shapes or number of blocks.
Graph Architectures.
We evaluate the proposed MHA representation against the commonly used Structure2Vector (S2V) architecture [23, 30] (see
Appx. A.1) in a simple environment (see Fig.
4a)only considering one type of object and no robot.
The learning is conducted with DQN.
Results.
Table 1 shows the obtained results.
Already in the original training environment (ﬁrst column), the MHA approach outperforms S2V signiﬁcantly.
The high rates of failure f and exceedingthe number of available blocks indicate that S2V cannot represent the connection between the target shape and the current structure.
This might be due to S2V’s different message passing, as itcannot weigh the importance of different nodes as with the attention mechanism.
When increasingthe number of available blocks and the size of the structure to be built (columns 3 & 4), we see anevident advantage of MHA in handling OOD tasks.
However, we must note that MHA is better inhandling an increasing number of blocks compared to larger target structures.
This might be dueto the fact that MHA focuses more on the connection between the current structure and the target,rather than the available number of blocks.
Due to our ﬁndings, we continue our experimentationusing the MHA architecture.
Graph Connectivity.
While we already deﬁned each node’s features (position, type information),there are different approaches for the graph connectivity. This choice is important as it inﬂuences themessage passing between nodes. We investigated the effect of using a fully connected (FC) graphand compare it to using partial connections (see Fig. 3) omitting the connections between different
[Table]
unplaced blocks, inducing a stronger inductive bias on the structured representation (more information in Appx.
A.3).
To test the generalization abilities
[Table] we test the approaches in the setting of Fig.4a with different types of blocks (i.e., rectangles of various lengths).
Those novel objects are ﬁxedconcatenations of the already existing blocks, treating the individual boxes as primitive elements.
This has the advantage of not needing any modiﬁcations to the action and observation space, whileallowing us to create more complex and diverse objects.
Results.
As shown in Table 2, partial connectivity outperforms the FC in all experiments.
Interestingly, designing a priori meaningful edge connections favors the extrapolation of the method tounseen types of objects (last column), indicating that the chosen representation is modular and effective in solving more challenging problem settings.
These results also indicate that reasoning aboutdifferent block types in the FC setting is more difﬁcult, as this has to happen based on the blocks’distances.
Contrarily, in the partial setting, the connectivity information directly reveals the blocktype.
As we want to keep the ﬂexibility of handling more complex blocks, we will exclusively focuson the partially connected architecture.
Learning algorithms.
To investigate the efﬁciency of the learning algorithms (Sec. 2.2), we willuse the two-sided environment shown in Fig.
4c but without having the robot.
However, we willnow evaluate the agents in more challenging settings, i.e., exposing them to more difﬁcult scenariosin terms of the size of the target shapes and the number of available blocks.
Results.
Table 3 summarizes the results, starting without any search budget (i.e. no tree search)to evaluate the learned Q-functions.
The DQN and (cid:15) -MCTS agents perform similarly, with DQNslightly outperforming (cid:15)
-MCTS through lower failure rates across tasks.
Moreover, DQN can solvethe task with fewer actions.
Contrarily, the Q-MCTS agents perform signiﬁcantly worse.
We believethat this difference in performance is due to the constant cross-entropy regularization in Q-MCTSfrom the beginning of the training, especially when search samples might be bad, while for the (cid:15) -
MCTS agent the regularization is only slowly added as training proceeds and less random actionsare taken (search is better), which seems to be beneﬁcial.
Note that without the addition of search,increasing the number of blocks as well as the target size results in a quick increase of the failure ratefor all the methods.
Adding a search budget of 5 already counteracts this trend, especially considering the DQN agent, where the rates of failure can be reduced for all the tasks by using DQN+MCTSat test time.
For the Q-MCTS agent, the same trend is noticed, while the gains of the (cid:15) -MCTS agentare marginal.
This suggests that the (cid:15) -MCTS agent is overoptimistic and seems to choose similaractions as to when not using search at all.
Overall, combining MCTS with a pretrained DQN resultsin the best performance in our experiments.
Q-MCTS with its constant additional cross-entropy regularization throughout the whole training process results in worse performance, which we believe isdue to overﬁtting to the respective task.
The last row of the table conﬁrms the combinatorial complexity of our problem, illustrating that performing pure UCT without any prior and a search budgetof 1000 performs signiﬁcantly worse, when tested to the simplest setting.
Learn2assemble with robot-in-the-loop.
Finally, we evaluate the DQN agent combined with treesearch in our original environment that contains the robot manipulator.
Thus, the task difﬁcultyincreases as we now also have to specify the grasping and placing poses to ensure the feasibilityand stability of the structure.
We incrementally increase the experiment’s difﬁculty.
We start presenting results on constructing a single-sided design (Fig. 4b) to compare whether training withthe robot-in-the-loop is necessary.
Then, we evaluate the proposed method in OOD structures withmulti-sided designs (Fig. 4c & 4d), aiming to showcase our pipeline’s generalizability.
Results.
In the ﬁrst experiment, we compare two policies, one trained with and the other without therobot-in-the-loop, and in plain DQN (no addition of search).
As shown in Table 4, the agent trainedwith the robot-in-the-loop outperforms the other agent.
This result underlines the necessity of including geometric planning during training for high-level decisions to complete the construction task [Table] [Table] successfully.
It is not sufﬁcient to only ﬁgure out where to place the parts; the kinematic constraints and robot motion shouldbe considered.
In the subsequent experiments, we build3D structures with the robot using the entire action space as shown in Figs. 4c& 4d.
We investigate again whether thecombination with MCTS improves performance.
Our results in Tables 5 and 6 showthat MCTS (search budget of 10) consistently improves performance in terms of higher returns and lower failure rate.
In particular, those experiments demonstrate that our proposed pipeline can execute multiple sequential pick-and-placingactions, with a maximum of 17 correctly placed blocks for the 5-by-5 grid in the two-sided environment and up to 22 correct placements in the four-sided environment, using the DQN+MCTSapproach with the robot-in-the-loop.
Compared to the previous experiments, without the robotin-the-loop, the failure rate is higher, indicating the task’s increased difﬁculty and that adding asoft-placing controller might be beneﬁcial.
[Table]
342 Remarks.
We have investigated different graph representations and their combinations with learningmethods for solving combinatorial assembly tasks with the robot-in-the-loop.
Our results demonstrate how strong inductive biases combined with attention can shape a meaningful relational representation.
When combined with deep Q-learning, this representation allows us to take decisions overlong horizons despite an increasing action space.
The addition of search with small budgets duringtraining seems to hinder the expressivity of the learned Q-function, constraining it into learning locally poorer values.
While we show very promising results in combining learning high-level actiondecisions with planning geometric executions, we can see a combinatorial barrier in the decisionmaking that might be tackled with a more informed search in the graph space.
Moreover, severalassemblies failed due to “rough” robot actions, meaning that a more sophisticated motion generatormight be needed for ﬁner placement.
Conclusion
We presented a new learn2assemble algorithm for learning autonomous robotic 3D assembly fromscratch without prior knowledge of any task plan.
Our proposed algorithm beneﬁts from an effectivegraph attention representation that captures the spatial relationships between target constructiondesigns and unplaced blocks, which trained through deep Q-learning can extrapolate to previouslyunseen target shapes, deal with a larger number of available elements, and different object types.
Moreover, when combining the learned Q-network with MCTS with small budgets, to keep thesearch computationally tractable, we manage to improve performance and reliability across all tasks.
Our method is a hierarchical controller with high-level learning over action sequences and goalspeciﬁcations combined with low-level path planning that ensures the execution of long-horizontasks.
Notably, we resolve the sequential long-horizon character of the assembly task by includingthe robot-in-the-loop to decide over feasible grasps and placing actions that ensure the stability ofthe construction.
Our algorithm manages to correctly build structures using up to 22 building blockswith good success rates.
In the future, we want to combine the high-level policy with a learnedtermination function, learn ﬁne-placing manipulation controllers on the low level, and investigatethe implementation of an assembly/disassembly strategy, so that the robot can potentially re-usewrongly placed blocks or reconﬁgure existing structures.