HTRON : Efﬁcient Outdoor Navigation with Sparse Rewards via Heavy Tailed Adaptive Reinforce Algorithm
Abstract: We present a novel approach to improve the performance of deep reinforcement learning (DRL) based outdoor robot navigation systems.
Most, existing
DRL methods are based on carefully designed dense reward functions that learn the efﬁcient behavior in an environment.
We circumvent this issue by working only with sparse rewards (which are easy to design), and propose a novel adaptive H eavy-
T ailed R einforce algorithm for O utdoor N avigation called HTRON .
The key idea in this work is to utilize heavy-tailed policy parametrizations which implicitly induces exploration in sparse reward settings.
We evaluate the performance of HTRON against Reinforce, PPO and TRPO algorithms in three different outdoor scenarios: goal-reaching, obstacle avoidance, and uneven terrain navigation.
We observe in average an increase of 34 .
41% in terms of success rate, 15 .
15% decrease in the average time steps taken to reach the goal, and 24 .
9% decrease in the elevation cost compared to the navigation policies obtained by the other methods.
Further, we demonstrate that our algorithm can be transferred directly into a Clearpath Husky robot to perform real-world outdoor terrain navigation.
Introduction
Autonomous robot navigation in complex outdoor environments has been an active area of research.
The resulting navigation systems are used for different applications, including delivery [1], search and rescue [2], planetary explorations [3], etc.
Such applications require robots to gather useful information efﬁciently from the environment to make intelligent navigation decisions [4].
To this end, Deep Reinforcement Learning (DRL) techniques have been widely employed in recent robotic systems due to their inherent exploration and exploitation capabilities to gather necessary information from the learning environments [5, 6, 7, 8, 9, 10].
Nevertheless, a major challenge for DRL based algorithms is dealing with sparse rewards in continuous state and actions spaces.
In particular, the rewards are sparse in navigation settings because it’s only available on reaching a goal (positive reward) or hitting an obstacle (negative rewards) [11].
Hence, training DRL policies under sparse reward settings oftentimes lead to instabilities and convergence issues that can signiﬁcantly degrade consistent performance, especially in navigation applications [12, 13].
To improve the training efﬁciency and deal with sparse rewards, two popular methods in the literature are reward shaping [14] and demonstration-guided learning [15, 16].
The primary objective of rewards shaping-based methods is to induce an exploration strategy using curiosity-driven methods which provide an additional pseudo reward for exploration in the environment [17].
However, such methods require carefully designed intrinsic rewards which can introduce expert-speciﬁc bias to the learning systems hindering the overall performance.
Similarly, demonstration-guided methods have a high dependence on expert supervision which could be difﬁcult to obtain in practice for many outdoor navigation tasks [18].
Instead of modifying the rewards, a more pragmatic approach is to try to work directly with the sparse rewards [19] and make the DRL based policies to train for the outdoor navigation task at hand.
But the major challenge is how to induce inherent exploration into the training without reshaping the rewards.
Recent work in literature[20, 21] suggest that one way to handle these is to use heavy-tailed policy (such as Cauchy) parametrization techniques.
Motivated by these factors, in this work, we try to ﬁnd optimal behaviors for outdoor navigation tasks while directly operating under sparse reward settings.
Main Contributions:
We present a heavy-tailed policy formulation with adaptive gradient tracking to address three navigation challenges encountered in complex outdoor environments (goal-reaching, obstacle avoidance, and navigating on uneven terrains).
Our proposed algorithm HTRON outperforms three state-of-the-art algorithms: Reinforce [22], PPO [23]
and TRPO [24] in terms of cumulative reward return with improved sample efﬁciency for faster convergence.
We propose a set of novel sparse reward functions that do not require careful reward designs yet are capable of successfully navigating a differential drive robot using an improved stochastic policy gradient method.
We evaluate the navigation performance of HTRON in both simulated and real-world complex outdoor environments using a Clearpath Husky robot.
HTRON results in an increase of up to 34 .
41% in terms of navigation success rate, 15 .
15% decrease in the average time steps taken to reach the goal, and 24 .
9% decrease in the elevation cost compared to the navigation policies obtained by the other methods.
Related Work
Learning-based approaches, in particular DRL have pushed the boundaries of mobile robot navigation for unstructured and dynamically changing environments [25, 26, 8].
Nevertheless, these methods suffer from sample efﬁciency under sparse rewards settings because dense rewards play an important role in learning for DRL methods.
We summarize the related work next.
Outdoor Robot Navigation: Navigation, path planning and obstacle avoidance are extensivelystudied fundamental research problems in the ﬁeld of robotics [27, 28].
Number of deterministic [29, 30] and stochastic [31, 32] algorithms have been proposed in the past three decades.
However, robot navigation in complex outdoor environments still remains a challenging problem given the complexity and diversity of the setting.
To perform robot-centric decision-making, deep reinforcement learning (DRL) strategies have been employed in recent studies for outdoor navigation [33, 34].
For example, deterministic policies such as DDPG[26], A3C[35], and DQN[36] with dense rewards have been incorporated for robot navigation on uneven terrains.
An adaptive model that directly learns control and environmental dynamics is presented in [8].
Moreover, segmentation[37] and self supervised [25] methods for identifying navigable regions have also been utilized with a combination of DRL based navigation methods to ensure stable and efﬁcient navigation.
Sparse Reward Settings: In sparse reward settings, an environment rarely produces a useful reward which signiﬁcantly affects the reinforcement policy leaning [11].
To deal with this issue, several reward shaping methods based on intrinsic curiosity and information gain-based shaping have been proposed [38].
For example, reward shaping based methods such as [39, 17] encourage the agent to explore unvisited states by modifying the actual reward output.
However, such approaches require additional effort and expertise in reward function design.
Imitation learning and learning from demonstration methods are another approach for dealing with sparse rewards in practice [40].
However, obtaining reliable expert demonstrations for continuous control tasks is practically infeasible and challenging [41, 15].
Heavy-tailed Policy Parametrization: The possibility of incorporating heavy-tailed distributions to RL tasks is theoretically analyzed in recent literature [42, 21].
For example, in [43], beta distribution is utilized with dense rewards for stochastic policy training to enhance the state exploration capabilities.
Further, a heavy-tailed policy gradient method is proposed in [42] to ﬁnd convergence to global maxima while minimizing the risk of local maxima convergence.
Inspired by these ideas, we propose a heavy-tailed adaptive reinforce algorithm to deal with sparse rewards in complex outdoor environments.
Problem Formulation and Our Approach 3.1 Outdoor Navigation via Reinforcement Learning
Mathematically, we formulate the outdoor navigation problem as a Markov Decision Process (MDP) in continuous state and actions spaces:
[Equation] where S denotes the state space including distance from the goal, heading, roll and pitch angle; A is the actions space (linear, angular velocity); P ( s (cid:48) | s, a ) is the transition kernel; r ( s, a ) is the reward and γ ∈ (0 , 1) denotes the discount factor.
The objective of the robot is to learn a navigation policy π θ ( a | s ) parameterized by θ (which controls the probability of taking a particular action a in given state s ) to perform efﬁcient outdoor navigation.
This is achieved by solving the following optimization problem
[Equation]
where V π θ ( s 0 ) =
E (cid:2) (cid:80) ∞ t =0 γ t r ( s t , a t ) | s
= s, a t ∼ π θ ( ·| s t ) (cid:3) is the average cumulative reward (or value function), and s 0 denotes the initial state along a trajectory { s t , a t , r ( s t , a t ) } ∞ u[-=
-]
=0 .We note that since the goal of the outdoor navigating robot is to reach a goal and to make sure that the problem (2) is actually achieving that, we need to design the reward function r ( s, a ) such that upon maximizing its cumulative value, our goal is achieved.
This is one of the major challenges in applying RL ideas to outdoor navigation.
The majority of the RL results revolve around learning in standard environments and platforms like Gym, Mujoco [44] or other simulated environments where the rewards structures are already deﬁned.
However, in practical outdoor navigation scenarios, it is extremely hard to generate dense rewards due to the possibility of huge unknown trajectories a robot can take while navigating in the environment [34, 8].
The problem is exacerbated in continuous state-action spaces where one needs to deﬁne rewards even for inﬁnite possibilities, which could be impractical.
3.2 Our Approach
To deal with this issue and empower the application of RL to outdoor navigation, we take a different route and advocate the use of sparse rewards.
The advantage of sparse rewards is the simplicity of their design as they need to be deﬁned only at speciﬁc goals/sub-goals to the robot.
However, the sparse reward makes the learning problem hard due to the non-trivial estimation of value functions over the continuous state space, which can be mitigated through the use of heavy-tailed policy gradient algorithms[42, 21].
In this work, we start with a construction of an experimental sparse reward design methodology for outdoor navigation scenarios and proceed with the description of our algorithm.
3.2.1 Outdoor Navigation using Sparse Rewards
We categorize the level of complexity in the reward structure using three navigation scenarios in outdoor environments.
In all scenarios, our agent is modeled as a differential drive robot with a two-dimensional continuous action space a = ( v, ω ) (i.e. linear and angular velocities), whereas the state space dimension varies for each scenario.
The state inputs are obtained in real-time from the robot’s odometry, and LiDAR sensors.
The list of state inputs used throughout the three scenarios is as follows
: • d goal ∈ R + - current distance between the robot and its goal; • α goal ∈ [0 , π ] - current angle between the robot’s heading direction and the goal; • ( v t − 1 , ω t − 1 ) ∈ [ − 1 , 1] - actions from the previous time step (i
e. linear and angular velocities); • ( θ roll , θ pitch ) ∈ [0 , π ] - roll and pitch angle of the robot respectively; • D obs ∈ [0 , 10] - laser scan vector that includes distance to the obstacles around the robot (i
e. 360 ◦ laser scan data as a vector with ∼ 720 elements) at a given time.
Scenario 1 (Goal Reaching Baseline):
In this scenario, the robot is placed in an obstacle-free outdoor terrain with an objective to reach a given goal location.
We incorporate distance to the goal, heading angle to the goal and previous actions to deﬁne a four-dimensional state space s =
[ d goal , α goal , v t − 1 , ω t − 1 ] .
We utilize a set of sparse rewards that are easy to implement for policy training.
For the goal reaching task, we deﬁne r heading and r dist rewards to maintain the robot’s heading direction towards the goal and to encourage reaching the goal respectively.
Hence,
[Equation]
where β g is a constant used to adjust the reward amplitude for goal reaching, σ g ∈ [0 , 0 . 1] is the variance of the normal distribution N ( ., σ 2 g ) which we maintain at a very low range.
This ensures, the robot receives a reward r dist only when it reaches either d goalor the deﬁned goal.
Otherwise, r dist is zero.
These reward formulations intuitively motivate the robot to reach the goal by rewarding it when the halfway or the complete distance to the goal is traveled.
The total reward is obtained as, r tot = r heading + r dist .
The reward distribution w.r.t the state parametric space is visualized in
Figure 1(a) to highlight the reward sparseness.
Scenario 2 (Obstacle Avoidance):
In this scenario, the robot is placed in an outdoor area with static obstacles such as trees, walls, and sharp hills.
The objective is to reach the goal while avoiding collisions with these obstacles.
Additionally laser scan vector is taken into account in the state space s =
[ d goal , α goal , v t − 1 , ω t − 1 , D obs ] .
In addition to the two rewards deﬁned in the previous scenario (i.e. r heading and r dist ), we introduce an additional sparse reward r obs as,
[Equation]
to penalize collisions.
Here, d collision is the minimum safety clearance that the robot should maintain with obstacles.
The modiﬁed total reward is obtained as, r tot =
r heading + r dist + r obs .
Scenario 3 (Navigating on Uneven Terrains):
In this scenario, the robot is placed in a highly elevated outdoor terrain where it might ﬂip over or experience instability in some regions.
The objective is to navigate to a goal location while avoiding steep elevations in the environment.
The corresponding state space is s =
[ d goal , α goal , v t − 1 , ω t − 1 , θ roll , θ pitch ] .
To minimize navigating on steep elevations, we incorporate another sparse reward term ( r stable ) with the goal reaching reward pair.
We consider the states where the robots roll ( θ roll ) or pitch angle ( θ pitch ) exceeds ± π 4 as steep elevations (i.e. unstable robot orientations).
Hence, the stability sparse reward is deﬁned as,
[Equation]
Hence, the total reward function for this scenario is obtained as, r tot =
r heading + r dist + r stable .
3.2.2 Heavy-Tailed Reinforce Algorithm for Outdoor Navigation ( HTRON )
Just operating with sparse rewards is not sufﬁcient to solve the outdoor navigation problem because the learning procedure won’t be able to explore the environment in a required manner.
For instance, continuous control robotics research primarily relies on Gaussian policy parametrization given by π θ ( a | s ) =
N ( a | ϕ ( s ) (cid:62) θ, σ 2 ) where θ controls the mean of the Gaussian, ϕ ( s ) denotes the states feature representation ϕ :
S → R d with d (cid:28) q , and σ 2 is variance.
However, the performance of Gaussian policy parametrization under sparse rewards is shown to suffer badly in the latest research
The primary reason for failure is due to the light-tailed nature of Gaussian distribution which restricts the policy to take action closer to its mean value and thereby restricts exploration.
Typical methods to deal with exploration includes adding intrinsic curiosity [45] or entropy based exploration [17] which either requires learning the global dynamics model or estimation of the occupancy density function which is extremely expensive for continuous control robotics problem.
In this work, we leverage an alternate approach motivated by the latest research by [42, 21] and focus on developing a heavy-tailed parametrization based policy for outdoor navigation problems with additional robotics-driven constraints as detailed in Algorithm 1.
We parameterize the policy by a heavy-tailed Cauchy distribution given by
[Equation]
where σ is the ﬁxed variance.
Now, ﬁnally we write the stochastic policy gradient [20] as
[Equation]
where ∇ J ( θ k , ξ k ( θ k )) denotes the unbiased estimator of gradient ∇ J ( θ k ) at θ k , and T k ∼
Geom (1 − γ 1 / 2 ) .
Although heavy-tailed parametrization induces exploration which has proven to be extremely beneﬁcial in sparse reward settings [42, 21], it also induces instability in behaviour due to the high probability of taking extreme action even near optimal regions.
Hence, [42] proposed gradient tracking to mitigate the above issue.
However, instead of momentum tracking with mirror ascent type update as proposed in [42], we use an adaptive moment estimation based method as an optimizer with gradient clipping to stabilize the instability induced owing to the heavy-tailed parametrization which improves the time taken by gradient tracking based methods.
[Equation]
where m k , v k are the ﬁrst and second moments respectively and β 1 , β 2 , (cid:15) are hyperparamters for the optimizer and g k = ∇ J ( θ k , ξ k ( θ k )) with the added constraint of clipping given as (cid:107)∇ J ( θ k , ξ k ( θ k ))
(cid:107) ∞
≤ φ .
Another important challenge is to incorporate practical constraints into the learning system which is speciﬁc to robotics based problems where the action is bounded.
To solve the same, we use a simple projection P ∞ ( · ) based technique where we project the output from the policy π θ ( a | s ) into an inﬁnite norm ball which constraints the action space within the bound and can be shown as a ∼ π θ ( a | s ) such that (cid:107) a (cid:107) ∞ ≤ δ .
Experiments
We conduct a detailed analysis and performance comparison of our proposed method against three state-of-the-art stochastic policy gradient algorithms: TRPO [24], PPO [23], and Reinforce [22].
We used identical policy distribution parameters (e.g. distribution standard deviation σ ) and neural network architectures with all four algorithms during training and evaluation.
The heavy-tailed policies are implemented and trained with Pytorch using a Unity based realistic outdoor simulator, Clearpath
Husky robot model, and ROS Melodic platform.
The Unity simulator includes diverse terrains and elevations for training and testing.
Further, we deploy HTRON on a real Clearpath Husky robot on outdoor terrains.
4.1 Policy Learning Performance
We evaluate the performance of all algorithms through the policy convergence rate (see Figure 2).
In the goal-reaching baseline scenario, we observed that standard Gaussian policy-based algorithms cannot achieve the highest reward return when we incorporate the sparse rewards.
In contrast, our heavy-tailed policy is able to obtain maximum rewards in a sample-efﬁcient manner as shown in
Figure 2(a).
This is primarily due to the improved exploratory behavior achieved by the heavytailed policy in comparison to the standard Gaussian policies.
In Scenario 2 with static obstacles, our experiments showed that the random and sparse obstacle penalties signiﬁcantly affect policy convergence and stability.
In particular, PPO converges to the halfway point towards the goal with multiple collisions.
However, our method successfully learns policies to reach the goal within ∼ 40 episodes (see Figure 2(b)).
This demonstrates the accelerated learning capabilities provided by the heavy-tailed formulation compared to the traditional stochastic policy gradient methods.
In Scenario 3, we include r stable to penalize the actions that could navigate the robot towards steep elevations.
We observe that Reinforce and TRPO based policies struggle to even reach the goal under this setting while PPO manages to reach the goal in only a few episodes without proper convergence.
Further, we performed additional experiments on PPO, TRPO, and Reinforce algorithms by increasing the standard deviation ( σ = 0 . 5 ) of their Gaussian policy distribution(see Fig. 3).
We observe that even with increasing variance, our algorithm (HTRON) outperforms all the baselines including
PPO, TRPO, Reinforce, etc which demonstrates the superiority of our algorithm.
Additionally, We observe that the Gaussian policies cannot achieve faster or better policy convergence by simply in- creasing the standard deviation of the policy distribution.
Instead, the Gaussian policies demonstrate higher instability during training.
Hence, we restricted to σ = 0 . 25 as presented in Fig. 2 during our experiments. 4.2 Navigation Performance
We compare our method’s navigation performance qualitatively in Figure 4 and quantitatively in Table 1.
The quantitative evaluations of the navigation trajectories are based on the following metrics:
• Success Rate: The percentage of successful goal reaching attempts without any collisions or ﬂip-overs out of the total experiments.
• Avg.
Trajectory Length:
The average number of time steps taken by the robot to reach the goal. • Elevation Cost: Norm of the elevation gradient experienced by the robot throughout a trajectory (i
e. ||∇ z r || , where z r is the vector that includes vertical motions of the robot’s along a trajectory).
We would like to note that in this work rather than proposing a competitive navigation algorithm, the primary objective is to highlight the importance of heavy-tailed distributions for faster policy convergence under practical sparse reward settings.
However, in addition to comparing against the
DRL-based methods
, we evaluate the navigation performance of two classical algorithms: Dynamic
Window Approach(DWA)[30] and Ego-graph[47]) under the same test conditions.
We observe that DWA and Ego-graph outperform all the DRL based methods in terms of success rate during goal reaching and obstacle avoidance. However, HTRON demonstrates comparable or better performance during uneven terrain navigation in terms of all the evaluation metrics. We further notice that our method maintains the highest success rate in all three scenarios while other DRL policies demonstrate a signiﬁcantly low performance in successful goal-reaching.
This is primarily due to the poor convergence in Gaussian policies under sparse reward settings.
Further, HTRON reaches the goal faster using fewer time steps than the other three methods.
This indicates that our policy has optimized to collect more goal completion rewards by reaching the goal location quickly.
Policy convergence plots presented in Figure 2 further validates this argument.
Navigation scenarios 1 and 2 are common in both indoor and outdoor environments.
Hence, we further investigate the importance of our algorithm to handle uneven terrains encountered in complex outdoor environments.
We observe that our method is capable of avoiding steep elevations from the [Table] changing roll and pitch angles before leading to robot ﬂip-overs (see Figure 2(c) and 4(c) ).
Hence, trajectories generated by our algorithm signiﬁcantly minimize the elevation cost experienced by the robot while other methods navigate along steep terrains with higher elevation costs.
Finally, we integrate our algorithm into a real Clearpath Husky robot to demonstrate navigation capabilities in real outdoor settings.
We observe that our method can successfully perform the navigation tasks we trained in the simulator.
Sample navigation trajectories from three real outdoor scenarios are presented in Figure 5.
Limitations and Analysis
We discuss the limitations of our method that we expect to address in the future.
The robot cannot identify the uneven terrains without experiencing the uneven region due to the orientation based elevation sensing.
A 3D LiDAR pointcloud based elevation map could be utilized in state space to identify the elevation changes in the robot’s vicinity without explicitly experiencing elevation changes from the orientation data.
Moreover, the heavy-tailedness of the policy distribution could lead to convergence instabilities.
Especially, we observe considerable instability and non-converging behavior when we try to use PPO and TRPO with heavy-tailed distributions.
Hence, further investigations need to be conducted to develop robust and stable heavy-tailed policy models for higher dimensional state spaces and more complex sparse reward settings.
Conclusion
In this work, we presented a novel approach for adapting heavy-tailed policy based control strategy for efﬁcient outdoor mobile robot navigation – HTRON .
The proposed method is able to overcome the major issue of sparse rewards inherent to DRL based approaches.
We show that without handcrafted reward shaping methods we are able to accelerate learning capabilities and produce a reliable navigation policy.
We have assessed the performance of navigation policies in three different scenarios based on their complexity and corresponding objectives.
All policies have been trained in a ROS-based high-ﬁdelity Unity simulator.
We report a performance comparison of our method against REINFORCE, PPO, and TRPO algorithms in terms of learning rate and navigation quality.
We also deployed HTRON on real Clearpath Husky in outdoor terrains.
And ﬁnally, we report the limitations, possible improvements, and future work that this approach can lead to robust outdoor navigation. Acknowledgments
This work was supported in part by ARO Grants W911NF1910069, W911NF and Army
Cooperative Agreement W911NF2120076.
We acknowledge the support of the Maryland Robotics
Center.