L EARNING S IMULTANEOUS N AVIGATION AND C ONSTRUCTION IN G RID W ORLDS
A BSTRACT
We propose to study a new learning task, mobile construction , to enable an agent to build designed structures in 1/2/3D grid worlds while navigating in the same evolving environments.
Unlike existing robot learning tasks such as visual navigation and object manipulation, this task is challenging because of the interdependence between accurate localization and strategic construction planning.
In pursuit of generic and adaptive solutions to this partially observable Markov decision process (POMDP) based on deep reinforcement learning (RL), we design a Deep Recurrent Q-Network (DRQN) with explicit recurrent position estimation in this dynamic grid world.
Our extensive experiments show that pre-training this position estimation module before Q-learning can signiﬁcantly improve the construction performance measured by the intersection-over-union score, achieving the best results in our benchmark of various baselines including model-free and model-based RL, a handcrafted SLAM-based policy, and human players.
Our code is available at: https://ai4ce.github.io/SNAC/ .
I NTRODUCTION
Intelligent agents, from animal architects (e g., mound-building termites and burrowing rodents) to human beings, can simultaneously build structures while navigating inside such a dynamically evolving environment, revealing robust and coordinated spatial skills like localization, mapping and planning.
Can we create artiﬁcial intelligence (AI) to perform similar mobile construction tasks?
To handcraft such an AI using existing robotics techniques is difﬁcult and non-trivial.
A fundamental challenge is the tight interdependence of robot localization and long-term planning for environment modiﬁcation .
If GPS and techniques alike are not available (often due to occlusions), robots have to rely on simultaneous localization and mapping (SLAM) or structure from motion (SfM) for pose estimation.
But mobile construction violates the basic static-environment assumption in classic visual SLAM methods, and even challenges SfM methods designed for dynamic scenes (Saputra et al., 2018).
Thus, we need to tackle the interdependence challenge to strategically modify the environment while efﬁciently updating a memory of the evolving structure in order to perform accurate localization and construction, as shown in Figure 1.
Deep reinforcement learning (DRL) offers another possibility, especially given its recent success in game playing and robot control.
Can deep networks learn a generic and adaptive policy that controls the AI to build calculated structures as temporary localization landmarks which eventually evolve into the designed one?
To answer this question, we design an efﬁcient simulation environment with a series of mobile construction tasks in 1/2/3D grid worlds.
This reasonably simpliﬁes the environment dynamics and sensing models while keeping the tasks nontrivial, and allows us to focus on the aforementioned interdependence challenge before advancing to other real-world complexities.
To show the tasks in grid worlds are still non-trivial and challenging, we benchmark the performance of several baselines, including human players, a handcrafted policy with rudimentary SLAM and planning, some model-free DRL algorithms which have achieved state-of-the-art performance in other learning tasks (see Table 1 for comparisons), and a model-based DRL using Deep Q-Networks (DQN) augmented with Monte Carlo tree search (MCTS) as planning mechanism.
Although our
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
A recent study (Stooke et al., 2021) found that decoupling representation learning from RL policy learning can outperform the joint learning of the two in standard RL algorithms.
Inspired by this, we propose to pre-train an explicit position estimation module using recurrent neural networks in the above DRL baselines.
Our experiment results show that this proposed method outperforms other
RL baselines.
In summary, our contributions include:
• a suite of novel and easily extensible learning tasks focusing on the interdependent localization and planning problem, which are released as open-source fast Gym environments; • a comprehensive benchmark of baseline methods, which demonstrates the learning challenge in these tasks; • an effective approach which combines DRQN with an explicit position estimation deep network, outperforming other baselines; • a detailed ablation study providing insights about the causes of the challenge, which could inspire future algorithms to solve the problem more effectively.
R ELATED W ORKS
RL baselines .
With the great success of model-free RL methods in game-playing (Mnih et al., 2013;
Silver et al., 2016) and robot control (Cheng et al., 2019; Zhang et al., 2015), we consider a family of common model-free RL methods as baselines.
Since our tasks are in grid worlds, similar to many
Atari games such as the Pac-Man, our ﬁrst choice is DQN (Mnih et al., 2013).
However, DQN has limited capability to tackle POMDP due to its poor latent state representation from long-term history, we benchmark DRQN (Hausknecht & Stone, 2015) which uses recurrent Q-network for integrating historical information.
Since our tasks may require explicit long-term planning, similar to (Silver et al., 2016; Bapst et al., 2019), we also include a model-based method which uses DQN as the prior and MCTS (Coulom, 2006) as the planning mechanism.
In addition, we add an actor-critic-based baseline, Proximal Policy Optimization (PPO) (Schulman et al., 2017).
Whereas the standard policy gradient method performs one gradient update per data sample, PPO enables multiple epochs of minibatch updates by a novel objective with clipped probability ratios.
We also include Rainbow (Hessel et al., 2017), SAC (Haarnoja et al., 2018), and DRQN combined with Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) as additional baselines in supplementary Section B.
Representation learning for RL .
Although the aforementioned RL methods work well on many robot learning tasks, it is still challenging to end-to-end learn meaningful representations and control policies jointly via DRL (Stooke et al., 2021).
Recently, a variety of studies improve the performance of RL methods through representation learning, such as internal feature learning (Lample & Chaplot, 2017), and auxiliary tasks (Jaderberg et al., 2016; Mirowski et al., 2016).
Echoing these studies, we also ﬁnd in our experiments that direct end-to-end DRL from raw observations without any representation learning could not solve our task efﬁciently.
Therefore, we propose to introduce a position-related representation learning module in our DRL framework, and pre-train this module to decouple its training from DRL.
Position-related representation learning is not new in DRL.
For example, DRL-based visual navigation studies (Zhu et al., 2017; Chaplot et al., 2020; Datta et al., 2021) use neural networks to extract from visual inputs latent representations encoding position information for RL agents to navigate in a static environment.
Differently, our task involves a dynamically evolving environment.
While work like (Hoeller et al., 2021) indeed navigates robots in dynamic environments using latent state representations in RL, distinctively, our task further involves active modiﬁcation of the environment.
Thus, we do not include this line of work as our baseline.
Non-learning baseline .
Besides the above methods, one may wish to see the performance of more classical approaches such as either dynamic (Saputra et al., 2018) or active (Mu et al., 2016) visual SLAM which has been studied for decades in robotics, even though they are not designed to work in dynamic environments like ours, nor do they study the planning for better construction accuracy.
Therefore, we implement a naive handcrafted policy (pseudo code in Algorithm 1 in the supplementary Section B) with basic localization and planning modules as a non-learning baseline.
The localization module borrows the idea from visual SLAM (Saputra et al., 2018) which relies on ﬁnding the common features through successive images to estimate the robot’s pose.
The planning module simply controls the robot to always build at the nearest possible location.
M OBILE C ONSTRUCTION IN G RID W ORLD 3. T ASK O VERVIEW
We formulate a mobile construction task as a 6-tuple POMDP (cid:104)S , A , T , O , R , D (cid:105) , in which a robot is required to accurately create geometric shapes according to a design D in a grid world.
The state space S is represented as S =
G × P , where G is a space of all possible grid states G storing the number of bricks at each grid, and P is a space of all possible robot positions p in the grid world.
At each time step, the robot takes an action a ∈ A , either moving around or dropping a brick at or near its location.
Moving a robot will change its location according to the unknown probabilistic transition model T ( p (cid:48) | p, G, a ) .
Dropping a brick will change the grid state G at (for 1D & 2D tasks) or near (for 3D tasks) position p without any uncertainty for simplicity.
This is reasonable because, in mobile construction settings, motion uncertainty is often the key challenge to ensure accuracy (Sandy et al., 2016).
The robot can make a local observation o ∈ O of G centering around its current location, with a sensing region deﬁned by a half window size W s .
We pad the constant value − 1 outside the grid world boundary to ensure valid observations in P , which is distinguished from empty ( = 0 ) or ﬁlled
( > 0 ) grid states.
This could help the robot localize itself near the boundary.
Finally, the design
D ∈ G is simply a goal state of the grid world the robot needs to achieve, and R ( s, a ; D ) is the reward function depending on this design.
The aforementioned interdependence challenge in localization and planning is reﬂected via two factors in this setting.
First, partial observability makes robot localization necessary.
Second, the environment uncertainty in T simulates real-world scenarios where the motion control of the mobile robot is imperfect and the odometry is error-prone.
We implement this by sampling the robot’s moving distance d in each simulation time step from a uniform distribution.
3. T ASK V ARIATIONS
Next, we specialize the above formulation into a suite of progressively more challenging tasks which varies in the setup of dimension, design D type, and obstacle mechanism, as summarized in Tableand Figure 2.
[Figure] [Figure]
[Table] [Table]
[Table] [Table]
Dimension.
We vary our tasks by the grid world dimension in 1/2/3D respectively.
The robot is restricted to moving along the x-axis in 1D and x-y plane for 2D and 3D environments.
Constant vs. Variable Design .
The design of our tasks could be either constant or variable.
The constant design task requests a robot to build the same shape D in all episodes in both training and testing.
For the variable design task, the design D will vary for each episode.
For a variable design task, to account for real-world situations where robots should have access to the design, we augment the local observation with additional information such as the design D as a 4-tuple o env = (cid:104) o, N s , N b , D (cid:105) , where N s is the number of actions the robot has taken
[Table]
and N b is the number of bricks used.
Dense vs. Sparse Design.
A dense design is a solid shape, and a sparse design is an unﬁlled shape (Figure A.2e and Figure A.2f).
Obstacles.
Different from 1D and 2D tasks where the robot size is ignored, the robot in the 3D tasks occupies 1 grid so its motion could be obstructed by the built bricks.
Stop criteria.
Each episode ends when N s = N smax or N b = N bmax , where the former is the maximum number of steps and the latter the maximum number of bricks (the integrated area of the design D within each episode).
We set N smax reasonably large to ensure the design completion.
In addition to the same stop criteria as in the 1D/2D cases, the game will stop when the robot is obstructed by the built bricks and cannot move anymore in a 3D environment.
R ECURRENT POSITION ESTIMATION IN DYNAMIC ENVIRONMENTS
After formulating mobile construction tasks, we benchmarked our naive model-free DRL baselines on these tasks.
We found that these DRL policies perform worse than our human baselines, especially on 2D and 3D tasks.
We believe the low performance is due to the difﬁculty in learning meaningful representations and an effective control policy jointly via RL training alone.
Especially, the aforementioned interdependence challenge of mobile construction tasks requires the agent to localize itself in a dynamic environment where the surrounding structure could change after the agent build a new brick.
Inspired by some recent studies (Stooke et al., 2021;
Lample & Chaplot, 2017;
Jaderberg et al., 2016; Mirowski et al., 2016) which decouple representation learning from RL, we propose our method which combines (1) a pre-trained localization network (L-Net) to estimate the current agent position and (2) a DRQN to select the best action based on the predicted positions and observations.
Section 4.1 introduces our proposed method and section 4.2 explains the training process of L-Net.
[Figure]
𝑜 𝑡
[Figure]
𝑦 𝑡
Linear Ƹ𝑝 𝑡− ℎ 𝑡− ℎ 𝑡
LSTM Ƹ𝑝 𝑡 ℎ 𝑡+ 𝑦
𝑡+
Linear Ƹ𝑝 𝑡+
DRQN
Env
L-Net
Overall Pipeline
L-Net .. .
DRQN
Env （ a ） （ b ） ... 𝑜 𝑡− ..
𝐷 𝑡 𝑐𝑟𝑜𝑝 Ƹ𝑝 𝑡 𝐷 𝐷 𝐷 𝐷 𝑡+1𝑐𝑟𝑜𝑝 𝑜 𝑡 𝑜 𝑡 𝑜 𝑡+ 𝑜 𝑡 𝑜 𝑡+ 𝑜 𝑡+ 𝑜 𝑡+ 𝑎 𝑡− 𝑎 𝑡 𝑎 𝑡 𝑎 𝑡 𝑎 𝑡+
[Figure]
4. DRQN WITH L-N ET
Our method is shown in Figure 3, consisting of two main components: (1) an L-Net which is pretrained via supervised learning, using an LSTM to predict position ˆ p t at each time step t ; (2) a standard DRQN which takes predicted positions ˆ p t , observations o t , and cropped goal design D crop as input and outputs the actions a t at current frame.
Let us use the 2D dynamic task as an example to explain the overall pipeline.
Firstly, the raw observations o t − 1 , o t of the previous and current time steps respectively, are ﬂattened to vectors.
Then these two vectors are concatenated with the predicted position ˆ p t − 1 and the action a t − 1 in the previous time step.
This concatenated vector z and hidden state h t − 1 are fed to an LSTM to output a hidden feature y t and its hidden state h t .
Finally, a linear layer is used to map the input hidden feature y t to a predicted position ˆ p t .
The L-Net which is composed of the LSTM and the linear layer is pre-trained through supervised learning.
As shown in Figure 3a, the observation o t and the design D are given to the DRQN to output an action a t in the baseline.
We tried this way of inputting information to DRQN similarly in our pipeline in Figure 3b, and we additionally input the predicted position p t to the DRQN in our method at the beginning.
But the resulting performance is not signiﬁcantly better than our model-free RL baselines.
This suggests that the raw input (the predicted robot position and the complete design) is not an efﬁcient representation for DRQN-based training.
Inspired by our human volunteers who choose to only refer to a local region of the D near the current position to make decisions when solving our tasks, we crop a local design D cropt centering at the predicted position ˆ p t from the complete design D .
Then, the D cropt , ˆ p t , and o t are given to the DRQN-based policy network to output an action.
Note that when training this pipeline, the pre-trained L-Net is frozen and only the initial position p 0 is assumed known.
All other positions ˆ p t | t =1 , 2 ,..., used in each DRQN step are predicted by L-Net based on predicted positions ˆ p t − 1 from its previous time step.
4. L-N ET PRE - TRAINING
Since the L-Net is trained apart from the overall pipeline training by supervised learning, we collect a dataset which contains 30000 episodes of games.
Each game leads to a replay buffer { M i }| i =1 , 2 ,...,N , where M i =
( o i − 1 , a i − 1 , p i − 1 , o i ) .
Each data in such a buffer is collected by letting the agent perform random actions until an episode ends.
When training the L-Net, we randomly sample a chunk of data { M i }| i =
t +1 ,t +2 ,...,t + L , where L is a constant sequence length, from the entire sequence.
As shown in supplementary Figure B.1, we ﬁrstly feed the data M t +1 and a randomly initialized hidden state h t to the L-Net that predicts position ˆ p t +1 and its hidden state h t +1 .
Note that we always use the ﬁrst ground-truth (GT) position p t as input and the input positions of following steps are all outputs from previous steps.
This means we do not perform teacher-forcing to train the
LSTM.
Then, we just repeat L steps and compute L predicted positions
ˆ p i | i
= t +1 ,t +2 ,...,t + L .
Finally, the L-Net is trained by minimizing the L2 loss between the predicted positions and GT positions.
E XPERIMENTS
We conduct comprehensive experiments to test baselines and our proposed method on mobile construction tasks.
In this section, we only pick representative baselines from different classes to show the results.
All experiment results are listed in supplementary Table 7 and the best qualitative results are shown in supplementary Figure C.3.
The evaluation metric, reward function, and training protocol are described in this section.
Baselines.
We choose two Q-learning methods, an actor-critic method, a search-based method, a non-learning method
and human players as representative baselines.
Detailed architecture designs and hyperparameter setups for each algorithm are explained in the supplementary Section B. • DQN .
For constant design tasks, we use an MLP-based Q network.
For variable design tasks, the design D is mapped to a feature vector by a 3 convolutional layers.
• DRQN .
We add one recurrent LSTM layer to the Q network as the DRQN baseline.
• PPO .
We benchmark discrete PPO in the Stable Baselines (Hill et al., 2018).
• DQN+MCTS .
For this model-based method, we simply combine DQN with standard MCTS algorithm, similar to Bapst et al.
• Handcrafted policy.
It consists of basic localization and planning modules.
• Human .
A simple GUI game that allows a human player to attempt our tasks in identical environments with the same limitations of partial observability and step size uncertainty.
They act as the agent, using the ARROW keys to maneuver, and SPACE to drop a brick.
Evaluation metric: we use the Intersection over Union (IoU) score as our evaluation criteria which is measured between the terminal grid state G T and the ground-truth design D .
Here, we use IoU as our evaluation criteria because it is more straightforward for us to evaluate the quality of the built structure to the ground-truth design D .
The IoU is deﬁned as:
[Equation]
Reward function.
The reward function R for 1D and 3D environments is shown in Table 3.
For the 2D environment, R is simply designed as: the agent gets 5/0 for dropping bricks at correct/wrong positions and gets 0 for movement.
Note that in the 2D environment we tried to add a similar penalty as in Table 3 for incorrect brick dropping, but we found this led to more frequent actions of moving instead of dropping bricks because of the higher chance of negative rewards of the latter (especially for sparse plans).
Consequently, this led to bad performance of all tested methods.
Therefore, we removed the penalty from the reward function in all 2D environments.
[Table]
[Figure]
[Table]
To validate the proposed framework and its robustness, all baselines are trained with the same set of 4 random seeds and averaged results are reported.
For the constant design tasks in 1D/2D/3D, we test the trained agent for 500 times for each task and report the average and the standard deviation (stddev) of IoU scores among 500 tests.
For the variable design tasks, we randomly generated 500 ground-truth designs and split them to 8/1/1 for training/validation/testing.
Following the same protocol as constant design tasks, we test all baselines on the testing set for 500 times for each task and report the average and the stddev of IoU scores among 500 tests
[Figure]
5. B ENCHMARK RESULTS
[Figure]
Figure 4 summarizes the quantitative testing results and the best qualitative testing results of the baselines and our method.
In general, we see that the performance of each method drops as the dimension of the environment increases.
This clearly shows that the larger the exploration space is and that the more complex the construction mechanisms are (e.g., obstacles in 3D tasks, further studied in Section 5.2), the worse the tested method performs.
Especially in 3D environments, most of the baselines learn ineffective policies or fail to learn any policies.
Meanwhile, most of the methods consistently perform worse on the sparse design tasks than on the dense ones.
We believe that such poor performance is due to the much sparser reward signals in sparse design tasks.
Moreover, we notice that all RL baselines have more difﬁculties dealing with variable design tasks compared to constant ones.
We posit that the RL baselines lack the capacity to learn effective representations for variable designs.
DRQN is expected to be the best among all RL baselines in most of the tasks because it can learn better from a long period of history.
Compared with all examined RL baselines, our proposed method achieves better performance consistently in all tasks.
In some cases, it can even surpass the human baseline and the handcrafted policy which relies on prior knowledge of the environment setups.
Next, we will compare them in detail.
RL w/ v.s. w/o L-Net .
Besides the proposed method described in section 4, we also include another baseline which combines DQN with L-Net to further test the effectiveness of our method.
We ﬁnd that RL w/ L-Net methods outperform all other RL baselines consistently among all tasks.
For 1D constant tasks, the DRQN w/ L-Net achieves 0.896 ave IoU score, which even surpasses the human baseline (IoU 0.83).
It also performs the best among other learning-based baselines in 1D variable tasks.
As shown in Figure 4(c), DRQN w/ L-Net can perfectly build the Gaussian-shaped design.
In contrast, most of the RL baselines are only able to build a ﬂattened shape (See the ﬁrst column of Figure C.3 in supplementary).
All these suggest the importance of representation learning that enables a reasonable position estimation, even in our dynamic environments.
On the 2D constant dense task, all methods reach similar performance.
However, the gap between
RL w/ L-Net and RL w/o L-Net becomes signiﬁcantly larger in more difﬁcult 2D tasks, such as the 2D constant sparse task.
The DRQN w/ L-Net can achieve a reasonable performance of median
IoU 0.301 and best IoU 0.545 in the 2D variable sparse task where others only learn very limited policies.
The qualitative results in Figure 4(c) show that DRQN w/ L-Net can complete half of the triangle shape in best cases while RL baselines can only drop a few bricks around the goal design (see the 6th column of Figure 4(d)).
This indicates that with the help of position prediction by the pre-trained L-Net, DRQN is able to learn a better construction policy.
Similarly, the DRQN w/ L-Net method outperforms RL baselines in 3D tasks.
From the bar chart of Figure 4(a), we can see that DRQN w/ L-Net can still learn reasonable policies which achieve average IoU 0.3074 and 0.315 on the 3D constant dense and sparse tasks.
Our method is almost able to construct the complete hollow cylinder in Figure 4(c).
Moreover, it is remarkable that DRQN w/
L-Net manages to learn a rough construction policy while other RL baselines fail to learn any useful policies.
We conclude that explicit localization via representation learning is crucial for tackling 3D tasks, though the performance is still limited due to the potential challenges of obstacles.
Handcrafted policy .
Although our handcrafted method performs seemingly well on 1D tasks, one should not draw a hasty conclusion that it must be the ultimate direction for solving mobile construction.
Because the good performance of this method relies on prior knowledge of the distribution parameters of the probabilistic transition model T : our handcrafted localization has a very low error rate (especially in 1D settings) by leveraging this privileged information, which is not accessible in other baseline methods and in reality.
Moreover, from Figure 4, we can see that the obstacle mechanism, the sparse designs, and the environment uncertainty in 2D and 3D are still challenging for this method.
We believe learning-based methods have a good potential to adaptively address these variations, while solving them one by one via different handcrafted methods is less effective.
Human baseline .
We collected 30 groups of human test data, totaling 490 episodes played, and report the average IoU shown in the Figure 4 with dot line.
All humans were required to play the same maximum number of steps for each game as other baselines.
From the feedback, we found that humans could learn effective policies (building landmarks to help localization) very efﬁciently in at most a few hours, which is much sooner than training an RL model.
5. A BLATION STUDY
[Figure]
[Table]
We performed ablation studies to comprehensively analyze the reasons associated with the poor
RL baseline performances on 2D and 3D tasks.
We identify four potential challenges: (1) the obstacles in 3D environments, (2) the lack of localization information, (3) the step size d uncertainty, and (4) the lack of landmarks in the environment for localization.
We use DRQN results on constant tasks as the basis of this ablation study and all the ablation experiments were conducted using the same setup and test criteria described in Section 5.
We also try imitation learning to train DRQN with expert experience and conduct an ablation study for Rainbow (see supplementary Section C).
[Figure]
To explore the inﬂuence of the obstacles, we conducted two experiments: adding the obstacle mechanism in a 2D task and removing it from the 3D one (Figure 5).
As shown in Table 4, the performance of the 3D task increases signiﬁcantly from 0.071 to 0.799 for dense designs.
Similarly, when we add obstacles into 2D, the IoU score drops by more than 40% .
These results suggest that the presence of obstacles is an important reason for the poor performance on 3D tasks.
Localization.
From the previous experiments, we conclude that good position estimation via representation learning could help RL learn better construction policies.
Moreover, it is interesting to test what are RL methods’ performance upper bound with the help of perfect localization.
To test this, we provided GT position p for DRQN on 2D constant tasks.
From Table 4, we can see that with the help of position information, 2D constant sparse is perfectly solved (see Figure 6a).
Environment uncertainty.
Besides the limited sensing range, the random step size could be another reason for the poor performance of baselines.
Therefore, we conduct an experiment where the step size uncertainty is removed.
From Table 4, we can see that the IoU increases by more than 40% compared to the sparse designs (see Figure 6b).
[Figure]
Landmarks.
An empty initial environment lacks landmarks widely used in SLAM methods for the robot to localize itself.
This could be another potential challenge of mobile construction tasks.
We conduct an experiment: randomly adding some landmarks in the initial environment.
From Table 4, we can see that IoU increases dramatically both on dense and sparse tasks (see Figure 6c and 6d).
C ONCLUSION AND F UTURE W ORK
To stimulate the joint effort of robot localization, planning, and deep RL research, we proposed a suite of mobile construction tasks, where we benchmarked the performance of common model-free and model-based deep RL algorithms, a handcrafted policy with basic localization and planning, and human baselines.
Meanwhile, we propose our method which incorporates a pre-trained L-Net and
DRQN to solve these tasks.
The experiment results indicate that learning an explicit position estimation can effectively improve the performance of RL methods on mobile construction tasks.
Although the performance of the proposed method is limited on the hardest tasks, we believe augmenting RL frameworks with representation learning is a promising direction to solve the interdependence challenge in mobile construction tasks.
We believe that the limitations of current methods could be overcome by designing a localization function which is able to localize more precisely in dynamic environments.
In the future, we plan to further extend our mobile construction task suite with more features such as allowing a new action of explicitly placing landmarks, physics-based simulation in continuous worlds, and multi-agent mobile construction.
R EPRODUCIBILITY S TATEMENT
To make all our experiment results reproducible, we submit codes with hyperparameters used for each method on each task in the supplementary material.
We also provide an instruction on how to use our code to reproduce the experiment results.
A CKNOWLEDGEMENT
The research is supported by NSF CPS program under CMMI-1932187.
The authors gratefully thank our human test participants, Dongdong Liu and Armand Jordana for implementing the SAC and PPO baselines and the helpful comments from Bolei Zhou, Zhen Liu and the anonymous reviewers and also Congcong Wen for paper revision
A M OBILE CONSTRUCTION TASKS
Mobile construction robots .
Recently we have seen a rising trend of 3D printing using mobile robots all around the globe for construction and manufacturing (Werfel et al., 2014; Jokic et al., 2014;
Ardiny et al., 2015; Nan, 2015; Marques et al., 2017; Buchli et al., 2018; Zhang et al., 2018;
Melenbrink et al., 2020).
All of those are carefully engineered systems that either assume some global localization ability or only work for speciﬁc scenarios, which restricts their feasibility in large scale.
Moreover, none of them address the aforementioned challenge from a theoretical perspective.
To stimulate both robot localization and planning and deep RL research community to work on this problem, we design an efﬁcient mobile construction simulation environment with a series of tasks in 1/2/3D grid worlds.
Several types of goal design D used in each task of 1/2/3D environments are shown in Figure A.2.
We also show some real world examples of mobile construction tasks in
Figure A.1.
For the 1D constant tasks, we consider three types of shape designs D ∈ R W : sin function curve,
Gaussian curve and Step function curve (see Figure A.2a to Figure A.2c).
For the dynamic tasks, we generate D (Figure A.2d) based on the following equation: D = a sin( bx + c ) ,where a ∼
U (3 , 12) , b ∈ { 1 , 2 , 3 } , and c ∼ U ( − π, π ) .
The coefﬁcients a, b and c are chosen randomly for each episode.
For 2D variable tasks shown in Figures A.2g and A.2h,three vertexes of a triangle are randomly picked within the grid world.
The setup of designs D in 3D is similar to the ones in 2D as shown in Figures A.2i-A.2l.
B B ASELINE SETUP
Several common model-free and model-based RL baselines, and one handcrafted policy are considered in our paper.
Here, we ﬁx the environment constants as follows: (1) for 1D environments, half window size W s = 2 , and the environment width W = 30 ; (2) for 2D and 3D environments W s =and W = H = 20 .
Detailed architecture designs and hyperparameter setups for each baseline are explained as follows.
[Figure]
DQN .
For the constant
, we use with three hidden layers containing [64,128,128] nodes with ReLU activation function for each layer.
Three convolutional layers with kernel size of 3, and ReLU activation functions are used to convert the ground-truth D to feature vector for variable design tasks.
We train DQN on each task for 3,000 episodes.
Batch size is 2,000, and replay buffer size 50,000.
For tuning the learning rate, we try a relatively small value 1e-8 to make sure convergence initially and gradually increase it until ﬁnding a proper value.
Additionally, instead of only using the current frame, we try to stack ten frames of historical observations in the replay buffer.
This is similar to how the original DQN handles history information for Atari games (Mnih et al., 2013), but no signiﬁcant difference was found.
DRQN .
We benchmark DRQN by simply adding one recurrent LSTM layer to the Q network in the DQN.
The hidden state dimension is 256 for LSTM layer for all tasks.
We train it for 10, episodes with batch size of 64 and replay memory size of 1,000.
DRQN+Hindsight .
We augment the DRQN baseline with hindsight experience replay (Andrychowicz et al., 2017) as another baseline.
At the end of each training episode, the transitions (cid:104)
o tenv , a t , R ( s t , a t ; D ) , o t +1 env (cid:105) of each time step t will be relabeled as (cid:104) o tenv , a t , R ( s t , a t ; G T ) ,
o t +1 env (cid:105) , where we change the D to the grid state G at the terminate step
T .
Both transitions are stored into the replay buffer.
We train this DRQN+Hindsight for 10, episodes with batch size of 64 and replay memory size of 1,000.
PPO .
We benchmark PPO in discrete settings using the Stable Baselines implementation (Hill et al., 2018).
We train PPO for 10 million time steps with a shared network of 3 layers of 512 neurons with tanh activation function.
For the hyperparameters, we use the 1D constant environment to tune the learning rate, the batch size, the number of minibatches size, and the clipping threshold.
We found that the most sensitive parameters were the batch size and the minibatch size and chose the following values: 1 × 10 5 for the batch size, 1 × 10 2 for the number of minibatches, 2 . 5 × 10 − 4 for the learning rate and 0 .
1 for the clipping threshold.
Rainbow .
For the Rainbow implementation, we use 3 noisy hidden layers (Fortunato et al., 2017) with 128 nodes in each layer, and ReLU nonlinear activation functions.
Rainbow has a large set of
[Table]
Intel(R) Core(TM) i9-9920X CPU @
3.50GHz using a single thread and report the average, maximum, and standard deviation of the simulation time of each game.
Our environments can be simulated faster by batch processing.
[Figure]
Linear
[Figure]
L-Net
[Figure]
L-Net
[Figure]
𝑝 𝑡
[Figure]
𝐿𝑜𝑠𝑠:෍
𝑖
[Figure]
𝐿 𝐿 2 ො𝑝 𝑡+𝑖 ,𝑝 𝑡
[Figure]
𝑖
[Figure]
Loop
[Figure]
ℎ 𝑡
[Figure]
𝐿−
LSTM
Cell Ƹ𝑝 𝑡+𝐿− 𝑦 𝑡+𝐿
Linear Ƹ𝑝 𝑡+𝐿
L-Net 𝑝 𝑡+𝐿
[Figure]
𝑜 𝑡+ 𝑜 𝑡+ 𝑜 𝑡+ 𝑜 𝑡+𝐿− 𝑜 𝑡+𝐿 ...
L-Net Training
[Figure]
hyperparameters, as each of the six components adds additional hyperparameters.
We used those suggested in (Hessel et al., 2017) as a starting point, but they led to poor results on our speciﬁc task and environment designs.
As a grid search over such a large hyperparameter space was impractical, we used a random search approach.
Based on empirical results, the algorithm was most sensitive to learning rate, as well as V min , V max , and n atoms , which deﬁne the value distribution support predicted by the distributional Q-network.
Generally, a V min value of − 5 , V max of 35 , and n atoms = 101 provided stable performance, and were chosen heuristically based on an approximate range of discounted rewards possible in our environments.
We used a prioritized experience replay buffer of size 1 × 10 4 , with priority exponent ω of 0 . 5 , and a starting importance sampling exponent β of 0 . 4 .
Additionally, we used multi-step returns with n = 3 , and noisy network σ 0 = 0 .
Finally, we used a learning rate of 5 × 10 − 5 for 1D and 2D, and 1 × 10 − 4 for 3D environments.
SAC .
We use the implementation of (Christodoulou, 2019) for SAC in discrete settings which has automatic tuning mechanism for entropy hyperparameters.
We use a learning rate 3 × 10 − 4 for target networks for most plans.
We use ReLU activation functions for the hidden layers and Softmax for the ﬁnal layer of the policy network.
We use interpolation factor τ = 5 × 10 − 3 for target networks and the start steps before running the real policy is 400 with mini batch size 64.
We ﬁrst search the main hyperparameters based on 1D constant case.
Next, we use different network architectures for relatively complex 2D and 3D cases with the same hyperparameters as 1D constant case.
For 1D environment, we use 2 hidden layers with 64 nodes each for both actor and critic networks.
For 2D environment, we use 3 hidden layers with 512 nodes each for the variable design and 3 hidden layers containing [64, 128, 64] nodes for the constant design.
A 5-layer network architecture containing [64,128,256,128,64] nodes is applied in the 3D cases.
DQN+MCTS .
For this model-based method, we simply combine DQN with standard MCTS algorithm, similar to Bapst et al.
In each time step, the tree search method uses the current state s t as the root node and the latest Q network as a guide to evaluate values for each node of the search tree.
When the search process ends, the action a t is selected based on the highest visit counts among all possible actions from the root node.
In all experiments, we use the true environment simulator as our transition model.
We use same architecture design for each task as we use in the DQN experiments.
The rollout of each tree search is 20 and the Upper Conﬁdence Bound (UCB)
(Auer et al., 2002) is used to balance the exploration and exploitation in the search process.
The UCB constant is set to 0.5 for each experiment.
We use learning rate 1 × 10 − 3 , batch size 2000, and replay buffer size 50,000 for all tasks and train the DQN on each task for 3000 episodes.
Handcrafted policy.
Besides these RL baselines, we also consider a handcrafted policy with basic localization and planning modules.
The localization module borrows the idea of SLAM which uses the common features between successive frames to localize the robot.
For the planning modules, we simply let the robot move to the nearest empty grid which should be built.
The detail handcrafted algorithm is shown in Algorithm 1.
The algorithm is divided into two sub-modules, localization and planning.
Localization receives current location l t , current observation o t and next observation o t + to determine next position l t +1 by ﬁnding the common features from two successive observations o t and o t +1 .
We borrow this idea from visual SLAM which uses similar mechanism to localize.
The planning function will ﬁnd all candidate target location L candidate for next step by comparing the current observation with the ground truth design D .
Based on these candidate positions, the agent will decide whether to move to nearest candidate position or build at current position.
The priority action space
A prior is used to decide the action a when the robot does not receive any speciﬁc action commend from the nearest planning policy.
This priority action will change when robot touches the boundary of the environment.
This mechanism helps robot explore more spaces of the unknown environment.
[Figure]
count are shown, while rewards are hidden.
Human .
For assessing human performance, we made a simple GUI game (details and video samples of the game in supplementary material) that allows a human player to attempt our tasks in identical environments with the same limitations of partial-observability and step size uncertainty.
They act as the agent, using the ARROW keys to maneuver, and SPACE to drop a brick.
The human baseline GUI game is shown in Figure B.2.
In constant task environments, human players are required to complete the task without the access to the ground-truth design, while in variable task environments, they can reference the current ground truth design.
Additionally, players can toggle between training or evaluation mode.
In training mode, they can view per-step reward as well as their cumulative reward over the episode, whereas in evaluation mode, they can only see the number of bricks used and steps taken.
For each episode played, players reported their episode-IoU.
C B ENCHMARK RESULT
In this section, we show the quantitative results of each baseline on all tasks in Table 7.
We also explain imitation learning and the Rainbow method ablation study here.
[Table] [Table]
Learning from expert experience.
Because most of the model-free and model-based RL methods cannot perform well on 2D and 3D tasks due to the aforementioned challenges, we are curious to study whether learning from expert experience could be one potential solution to mobile construction tasks.
Therefore, we collect expert experiences on 2D constant sparse and variable dense tasks via our handcrafted policy which performs better than the RL methods.
Then we train the DRQN baseline with these (instead of random) experiences in the initial replay buffer.
Table 6 shows that learning from expert can indeed boost the performance of DRQN, especially for the worst performing cases.
Rainbow method ablation study.
We observed similar performance across most tasks when comparing the base DQN algorithm with DQN plus Rainbow.
To better understand the effect of each individual extension on the whole, we conducted six separate runs: in each, we removed one extension from the complete Rainbow algorithm, similar to (Hessel et al., 2017).
Figure C.1 displays the IoU running average over training in the 2D variable environment for each of these pruned conﬁgurations.
Overall, we ﬁnd that the complete Rainbow algorithm initially learns faster than all but one of the pruned conﬁgurations, suggesting that the combination of optimizations indeed leads to better sample-efﬁciency in the learning process.
Removing noisy networks (reverting to an epsilon-greedy exploration approach) led to the largest decrease in learning efﬁciency.
Over a longer horizon, the top testing performance plateaus to nearly the same across all conﬁgurations.
In the case of distributional learning, multi-step learning, and double DQN, removing each individual actually improved top test performance, from 0.676 to 0.685, 0.701, and 0.714 respectively.
While this study is not exhaustive, it suggests that the Rainbow algorithm does not easily generalize from the Arcade
Learning Environment (Bellemare et al., 2013) (which it was designed for) to our speciﬁc tasks.
Construction accuracy with position tion error.
We also conduct an experiment to test how the position prediction precision of L-Net could effect the construction performance of DRQN agent.
We test
DRQN w/ L-Net on 2D variable sparse task for 500 times and get average L2 distance between predicted and GT positions and IoU of each game.
As shown in the Figure C.2, we ﬁnd that the construction performance of the policy is expected to be effected by the accuracy of position estimation by L-Net.
With higher localization error, the construction will also be poorer.
This indicates that designing an appropriate localization module is key path to solving mobile construction tasks.
[Table]
1: function M AIN 2:
Initialize environment and get initial observation o t = 3:
Get initial location p t = 4:
Initialize priority action space
A prior 5: for t ← 0 , N smax do 6: a t , A prior =P LANNING ( p t , o t ,
A prior ) 7:
Execute a t and obtain next observation o t +1 , reward, and terminal sign Done from environment.
8: p t +1 =
L OCALIZATION ( p t , o t , o t +1 , a t ) 9: if Done then 10:
Break 11: function L OCALIZATION ( p t , o t , o t +1 , a t ) 12:
Common Feature = F EATURE M ATCHING ( o t , o t +1 ) 13: if Common Feature is empty or o t =
o t +1 then 14:
Determine p t +1 using Odometry, here we assume step size is always 1.
return p t + 15: else 16:
Determine step size using Common Feature.
Calculate p t +1 using step size and a t .
return p t + 18: function P LANNING ( p t , o t , A prior ) 19:
P candidate =
C OMPARE ( o t , D ( p t ) ) 20:
Update A prior based on the current boundary condition 21: if P candidate is empty then 22: a t is random sampled from A prior return a t , A prior 23: else 24:
Find nearest location p near from P candidate 25: if p near = p t then return a =Drop brick, A prior 26: else 27:
Determine action a t based on the corresponding direction of p near to p t return a t , A prior