Group-based Motion Prediction for Navigation in Crowded Environments
Anonymous Author(s)
AfﬁliationAddress email
Abstract: We focus on the problem of planning the motion of a robot in a dynamic multiagent environment such as a pedestrian scene.
Enabling the robotto navigate safely and in a socially compliant fashion in such scenes requires arepresentation that accounts for the unfolding multiagent dynamics.
Existing approaches to this problem tend to employ microscopic models of motion prediction that reason about the individual behavior of other agents.
While such models mayachieve high tracking accuracy in trajectory prediction benchmarks, they oftenlack an understanding of the group structures unfolding in crowded scenes.
Inspired by the Gestalt theory from psychology, we build a Model Predictive Controlframework (G-MPC) that leverages group-based prediction for robot motion planning.
We conduct an extensive simulation study involving a series of challengingnavigation tasks in scenes extracted from two real-world pedestrian datasets.
Weillustrate that G-MPC enables a robot to achieve statistically signiﬁcantly highersafety and lower number of group intrusions than a series of baselines featuringindividual pedestrian motion prediction models.
Finally, we show that G-MPCcan handle noisy lidar-scan estimates without signiﬁcant performance losses.
Introduction
Over the past three decades, there has been a vivid interest in the area of robot navigation in pedestrian environments [1, 2, 3, 4, 5].
Planning robot motion in such environments can be challengingdue to the lack of rules regulating trafﬁc, the close proximity of agents and the complex emergingmultiagent interactions.
Further, accounting for human safety and comfort as well as robot efﬁciencyadd to the complexity of the problem.
To address such speciﬁcations, a common [6, 4, 3, 7, 8] paradigm involves the integration of abehavior prediction model into a planning mechanism.
Recent models tend to predict the individualinteractions among agents to enable the robot to determine collision-free candidate paths [3, 4,9].
While this paradigm is well-motivated, it tends to ignore the structure of interaction in suchenvironments.
Often, the motion of pedestrians is coupled as a result of social grouping.
Further,the motion of multiple agents can often be effectively grouped as a result of similarity in motioncharacteristics.
Lacking a mechanism for understanding the emergence of this structure, the robotmotion generation mechanism may yield unsafe or uncomfortable paths for human bystanders, oftenviolating the space of social groups.
Motivated by such observations, we draw inspiration from human navigation to propose the use ofgroup-based prediction for planning in crowd navigation domains.
We argue that humans do notemploy detailed individual trajectory prediction mechanisms.
In fact, our motion prediction capabilities are short-term and do not scale with the number of agents.
We do however employ effectivegrouping techniques that enable us to discover safe and efﬁcient paths among motions of crowdnetworks.
This anecdotal observation is aligned with gestalt theory from psychology [10] whichsuggests that organisms tend to perceive and process formations of entities , rather than individualcomponents.
Such techniques have recently led to advances in computer vision [11] and computational photography [12].
Similarly, we envision that a robot could reason the formation of effectivegroups in a crowded environment and react to their motion as an effective way to navigate safely.
Related Work
Over the recent years, a considerable amount of research has been placed to the problem of robotnavigation in crowded pedestrian environments [4, 3, 18, 8, 19, 20, 21, 22, 23].
Such environmentsoften comprise groups of pedestrians, navigating as coherent entities.
This has motivated recentwork on group detection and group motion modeling.
Groups are often perceived as sub-modular entities that collectively deﬁne the behavior of the crowd.ˇ Sochman and Hogg [24] suggests that 50 - 70% of pedestrians walk in groups.
Many works exist ingroup detection.
One popular area in such domain is static group detection, often leveraging Fformation theories [25].
However, dynamic groups often dominate pedestrian rich environmentsand they exhibit different spatial behavior [26].
Among dynamic group detection, the most commonapproach treats grouping as a probabilistic process where groups are a reﬂection of close probabilistic association of pedestrian trajectories [27, 28, 29, 30, 31].
Others use graph models to buildinter-pedestrian relationships with strong graphical connections indicating groups [32, 33].
The social force model [34] also inspires Mazzon et al.
[35], ˇSochman and Hogg [24] to develop featuresthat indicate groups.
Clustering is another common group of technique to group pedestrians withsimilar features into groups [36, 37, 38, 39].
In this paper, we do not intend to explore the state-ofthe-art grouping practice.
For our formulation, it is sufﬁcient to employ a simple clustering basedgrouping method proposed by Chatterjee and Steinfeld [39].
Applications on groups often focus on a speciﬁc behavior aspect.
In terms of interacting with pedestrians, a major focus in this area is how a robot should behave as part of the group formation [40].
On dyad groups involving a single human and a robot, some researchers examined socially appropriate following behavior [41, 42, 43, 44] and conversely, guiding behavior [45, 46, 47].
In worksthat do not include robots as part of the pedestrian groups, some research teams studied how a robotshould guide a group of pedestrians [48, 49, 50].
From navigation perspective, Yang and Peters [26]leverage groups as obstacles, but their group space is a collection of individual personal spaces withoccasional O-space modeling from F-formation theories.
Without the engineered occasional occurrence of O-space, their representation reduces to one of our baselines.
Katyal et al.
[51] introduce an additional cost term that leverages robot’s distance to the closest group in a reinforcement learningframework.
They model groups using convex hulls directly generated from pedestrian coordinatesinstead of taking personal spaces into consideration.
This less principled approach often leads to therobot approaching dangerously close to pedestrians.
In our work, we additionally explore the capabilities of groups in handling imperfect sensor inputs.
While our focus is on analysing the beneﬁts ofgroups, our group based formulation can be easily incorporated into Katyal et al.
[51]’s framework.
Problem Statement
Consider a robot navigating in a workspace W ⊆ R 2 amongst n other dynamic agents.
Denote bys ∈ W the state of the robot and by s
i ∈
W the state of agent i ∈ N = { 1 , . . . , n } .
The robot isnavigating from a state s 0 towards a destination s T by executing a policy π : W n +1 × U
→ U thatmaps the assumed fully observable world state S
=
s ∪ i =1: n s i to a control action u ∈ U , drawnfrom a space of controls
U ⊆ R 2 .
We assume that the robot is not aware of agents’ destinations s iTor policies
π i : W n +1 × U i
→ U i , i ∈ N .
In this paper, our goal is to design a policy π that enablesthe robot to navigate from s 0 to s T safely and socially.
Group-based Prediction
We introduce a framework for group-based representations based on [13] and a model for groupbased prediction that is amenable for use in decentralized multiagent navigation.
4.1 Group Representation
Deﬁne as θ i ∈ [0 , 2 π ) the orientation of agent i ∈ N which is assumed to be aligned with thedirection of its velocity u i , extracted via ﬁnite differencing of its position over a timestep dt anddenote by v i =
||
u
i || ∈ R + its speed.
We deﬁne an augmented state for agent i as q i =
( s i , θ i , v i ) .
We treat a social group as a set of agents who are in close proximity and share similar motioncharacteristics.
Assume that a set of J groups, J = { 1 , . .
, J } navigate in a scene.
Deﬁne byg i ∈
J a label indicating the group membership of agent i .
We then deﬁne a group j ∈ J as a set G j = { i ∈ N | g i = j } and collect the set of all groups in a scene into a set G = { G j | j ∈ J } .
Extracting Group Membership.
We deﬁne the combined augmented state of all agents as q =∪ i =1: n q i .
To obtain group memberships for a set of agents N , we apply the Density-Based Spatial
Clustering of Applications with Noise algorithm (DBSCAN) [52] on agent states:
[Equation]
Where (cid:15) s , (cid:15) θ , (cid:15) v are respectively threshold values on agent distances, orientation and speeds.
Extracting the Social Group Space.
For each group G j , j ∈ J , we deﬁne a social group space asa geometric enclosure G j around agents of the group.
For each agent i ∈
G j , we deﬁne a personalspace P i as a two-dimensional asymmetric Gaussian based on the model introduced by Kirby [53].
Refer to Appendix A for detailed descriptions.
Given the personal spaces P i , i ∈ G j , of all agents in a group j , we extract the social group spaceof the whole group as a convex hull:
[Equation]
The shape described by G j represents an obstacle space representation of a group containing agentsin close proximity with similar motion characteristics.
For convenience, let us collect the spaces ofall groups in a scene into a set G = {G j | j ∈ J } .
4.2 Group Space Prediction Oracle
Based on the group-space representation of Sec. 4.1, we describe a prediction oracle that outputs anestimate of the future spaces occupied by a set of groups G t : t f up to a time t f = t + f , where fis a future horizon given a past sequence of group spaces G t h : t from time t
h = t − h where h is awindow of past observations:
[Equation]
[Table]
where O j is a model generating a group space prediction for group G j .
Refer to Appendix B fordetailed description of partial input handling.
We implement the oracle O j of eq.
(3) using a simple autoencoder.
The encoder follows the 3Dconvolutional architecture in [54] whereas the decoder mirrors the model layout of the encoder.
Theautoencoder takes as input a sequence 1 G t h : t and outputs a sequence G t +1: t f which we pass througha sigmoid layer.
We supervise the autoencoder’s output using the binary cross entropy loss.
We veriﬁed the effectiveness of our autoencoder on the 5 scenes of our experiments by conductinga cross-validation comparison against a baseline.
The baseline predicts the future shapes by linearlytranslating the last social group shape using its geometric center velocity.
We use Intersection over
Union (IoU) as our metric.
Between the ground truths and the predictions, this metric divides thenumber of overlapped pixels by the number of pixels occupied by either one of them.
As shown in
Table 1, our autoencoder outperforms the baseline.
Model Predictive Control with Group-based Prediction
We describe G-MPC, a model predictive control (MPC) framework for navigation in multiagentenvironments that leverages the group-based prediction oracle of Sec. 4.
We describe our group-prediction informed MPC, or G-MPC.
At planning time t , given a (possiblypartial) augmented world state history Q t ˆ h : t , we ﬁrst extract a sequence of group spaces G t h : t basedon the method of Sec. 4.1.
Given these, the robot computes an optimal control trajectory u ∗ =
u ∗ 1: Kof length K by solving the following optimization problem:
[Equation]
where γ is the discount factor and J represents a cost function, eq.
(5) initializes the group spacehistory ( k = 2 − h is the timestep displaced a horizon h in the past from the ﬁrst MPC-internaltimestep k = 1 ), eq.
(6) initializes the robot state to the current robot state s t , eq.
(7) is an updaterule recursively generating a predicted future group sequence up to timestep k
f = k + f givenhistory from time
k h =
k − h up to time k , O represents a group-space prediction oracle based on
Sec. 4, and eq.
(9) is the robot state transition assuming a ﬁxed time parametrization of step size dt .
We employ a weighted sum of costs J g and J d , penalizing respectively distance to the robot’s goaland proximity to groups:
[Equation]
where λ is a weight representing the balance between the two costs and
[Equation] [Equation] [Table]
penalizes a rollout according to the distance of the last collision-free waypoint to the robot’s goal.
Further, we deﬁne J d as:
[Equation]
12) where
[Equation]
[Equation]
where D ( s k − G jk ) returns the minimum distance between the robot state and the space occupied bygroup j at time k .
Using D , function D computes the minimum distance to any group for a giventime.
In most cases, the robot lies outside of groups, i.e., s k / ∈ G jk –therefore, the cost J d tries tomaximize the distance D .
Sometimes, the robot might end up entering the group space G –in thosecases, J d tries to minimize D , to steer the robot towards the direction of quickest escape from thegroup.
In case that the robot is inside a group to begin with, we shrink the group sizes in Sec.
4.1until
the robot is outside the groups again.
To solve eq.
(4), we search over a ﬁnite set U of control trajectories of horizon K .
With the assumption that the robot is holonomic and is not under any kinematic constraints, we use a set of R controlrollouts U
= { u 1 , ..., u R } with three levels of tangential speeds and a set of turning speed, i.e.,
[Equation]
To ensure compatibility between our group-based prediction model and our MPC formulation, weset the control rollout time horizon to be the prediction model’s prediction horizon, or K = f .
Evaluation
We evaluate our framework through a simulation study in which the robot performs a navigationtask (a transition between two points) within a crowds of dynamic agents in a set of scenes.
6.1 Experimental Setup
We consider a set of realistic pedestrian scenes, drawn from the ETH [14] (ETH and HOTEL scenes)and UCY [15] (ZARA1, ZARA2 and UNIVERSITY scenes) datasets, which often serve as benchmarking testbeds in the motion prediction and social navigation literature [55, 17, 56, 57].
In eachscene, we deﬁne two navigation tasks (see Fig. 2): Flow : in which the robot navigates along thecrowd ﬂow and Cross in which the robot intersects vertically with the trafﬁc ﬂow.
For each task, wegenerate a set of trials by segmenting the scene recording into blocks involving challenging interactions.
We deﬁne a challenging interaction to be a segment involving at least 5 pedestrians inside thetest region drawn in black in Fig.
This process provided us with a distribution of trials as shownin table Table 2.
Across all trials, we keep the robot’s maximum speed at 1 .
75 m/s .
We consider two experimental conditions: an Ofﬂine and an Online one.
In the Ofﬂine one, therobot navigates among a crowd moving according to a recording of a human crowd.
Under thiscondition, pedestrians act as dynamic obstacles that do not react to the robot, a situation whichcould arise in cases where robots are of shorter size and could thus be easily missed by navigatingpedestrians.
In the Online one, the robot navigates among a crowd 2 moving by running ORCA [16],a policy that is frequently used as a simulation engine for benchmarking in the social navigationliterature[57, 8, 58].
To investigate the value of G-MPC, we develop three variants of it.
group-auto is a G-MPC inwhich the autoencoder has a history h
= 8 and a horizon f = 8 .
group-nopred is a variant that features no prediction at all –it just reacts to observed groups at every timesteps and it is equivalentto the framework of Yang and Peters [26].
Finally, laser-group-auto is identical to group-autobut instead of using ground-truth pose information, it takes as input noisy lidar scan readings.
Wesimulate this by modeling pedestrians as 1 m -diameter circles and lidar scans as rays projecting fromthe robot.
We refer to the spec sheet of a SICK LMS511 2D lidar for simulation parameters.
Wefurther inject noise into the readings according to the spec sheet.
Under this simulation, pedestriansmay only be partially observable or even completely occluded from the robot.
We compare the performance of these policies against a set of MPC variants using mechanismsfor individual motion prediction.
ped-nopred is a vanilla MPC that reacts to the current states ofother agents without making predictions about their future states.
ped-linear is a vanilla MPC thatestimates future states of agents by propagating agents’ current velocities forward.
This baselineis motivated by recent work showing that constant-velocity models yield competitive performancein pedestrian motion prediction tasks [59].
Finally, ped-sgan is an MPC that uses S-GAN [17] toextract a sequence of future state predictions for agents based on inputs of their past states.
Weselected S-GAN because it is a recent highly performing model.
To ensure a fair comparison, all the MPC policy variants are integrated with the same MPC controller evaluated at dt = 0 .
We measure the performance of the policies with respect to four different metrics: a) Success rate ,deﬁned as the ratio of successful trials over total number of trials; b) Comfort , deﬁned as the ratioof trials in which the robot does not enter any social group space over the total number of trials; c) Minimum distance to pedestrians , deﬁned as the smallest distance between the robot and any agentper trial; d) Path length , deﬁned as the total distance traversed by the robot in a trial.
To track the performance of G-MPC, we design a set of hypotheses targeting aspects of safety andgroup space violation which we investigate under both experimental conditions, i.e., ofﬂine andonline:
H1 :
To explore the beneﬁts of group based representations alone, we hypothesize that group-nopredis safer than ped-nopred while achieving similar success rates but worse efﬁciency.
H2 : To explore the full beneﬁt of group based formulation, we hypothesize that group-auto is saferthan ped-linear and ped-sgan while achieving similar success rates but worse efﬁciency.
H3 :
To explore how our formulation handles imperfect inputs, we hypothesize that laser-groupauto achieves similar safety to group-auto while achieving similar success rate and efﬁciency.
H4 :
To check that our formulation is socially compliant, we hypothesize that group-nopred , groupauto and laser-group-auto violate agents’ group space less often than the baselines.
6.2 Results
Quantitative Analysis.
Fig.
3 and Fig.
4 contain bar charts representing the performance of G-MPCcompared with its baselines under Ofﬂine and Online settings respectively.
Bars indicate means,errorbars indicate standard deviations, “F” and “C” are ﬂow and cross scenarios respectively, andthe number of asterisks indicates increasing signiﬁcance levels: α = 0 .
according totwo-sided Mann-Whitney U-tests.
H1 :
We can see from both Fig. 3 and Fig.
4 that G-MPC achieves statistically signiﬁcantly largerminimum distances to pedestrians across all scenarios, often with p < 0 .
This illustrates thatthe group representation is in itself capable of upgrading a simple MPC with no prediction.
Asexpected, we observe that the price G-MPC pays for that is a larger average path length.
We also seethat success rates are comparable.
Overall, we conclude that H1 holds.
H2 : When future state predictions are considered, G-MPC obtains statistically signiﬁcant results inmost scenes supporting its attributes of being safer at the cost of worse efﬁciency.
Thus H2 is partially conﬁrmed.
In ofﬂine scenarios, G-MPC has lower success rates in crossing scenarios.
Uponcloser inspection, most failure cases are due to timeouts from G-MPC’s conservative behavior.
However, in online scenarios where pedestrians react to the robot, G-MPC achieves high success rates.
In real-world situations, to cross dense trafﬁc, the robot needs to plan its actions with expectations ofreactive pedestrians.
Otherwise, the robot will most probably run into the freezing robot problem [4].
H3 :
Group-based representations have the potential to robustly account for imperfect stateestimates.
Overall, we observe that with simulated imperfect states, G-MPC does not performstatistically signiﬁcantly worse in terms of safety, but in dense crowds of the UNIV scenes it hasworse efﬁciency and worse success rates in online cases.
This shows that H3 holds in terms ofsafety and, in moderately dense human crowds, holds in terms of efﬁciency.
Future work on better group representation is needed to achieve better efﬁciency in high-density human crowds givenimperfect states.
H4 :
From Fig.
3 and Fig.
4, we can see that G-MPC often has fewer group-space intrusions thanits baselines.
While this relationship is not always statistically signiﬁcant, we do see a general trendof the group-based approaches to respect group spaces more often than individual ones.
Thus, weconclude that H4 is partially conﬁrmed.
We additionally observe a general trend that group-auto is better than group-nopred in terms ofhigher success rates, lower chances of group intrusions, longer minimum distances to pedestriansand shorter path lengths.
This shows that our group prediction model offers beneﬁts to the robot’snavigation.
However, in a few scenarios group-nopred performs better.
We largely attribute this tothe ﬁnite inaccuracies of future group predictions and the freezing robot problem that accompaniesthe robot’s more conservative behavior in group-auto than in group-nopred .
Qualitative Analysis.
Qualitatively, it is a more common occurrence for regular MPC to performaggressive and socially inappropriate maneuvers than G-MPC.
As shown in the two examples in
Fig.
5 executed by ped-sgan and group-auto agents, we can see that in ofﬂine conditions, the MPCagent aggressively cuts in front of the two pedestrians to the left before proceeding headlong intothe cluster of pedestrians, only managing to avoid the deadlock by escaping through the narrow gapthat opens up.
While for G-MPC, it tracks the movements of the two pedestrian groups coming fromthe left.
When the two pedestrian groups merge, the agent turns around and reevaluates its approachto cross.
In the online condition, we observe that the MPC agent cuts through a pedestrian group toreach the other side, forcing a member of the group to stop and yield as indicated by the pedestrian’sshrinking personal space, which is proportional to its speed.
In the same situation, the G-MPC agentchooses to circumvent behind the social group.
Conclusion
We introduced a methodology of generating group-based representations and predicting their futurestates.
Through an extensive evaluation over the ﬂow and crossing scenarios drawn from 10 differentreal-world scenes from 2 different human datasets with both reactive and non-reactive agents, wedemonstrate that our approach is safer and more socially compliant.
Through experimentation withsimulated laser scans, our model displays promising potential to process noisy sensor inputs withoutmuch performance downgrade.
Various improvements to our control framework are possible.
For example, we could incorporatestate-of-the-art oracles in the form of advanced video prediction models [60] or incorporate intergroup interaction modeling.
Additional considerations such as the set of rollouts or the cost functionscould possibly increase performance.
Finally, alternative control frameworks such as reinforcementlearning approaches could be applicable.
However, our goal in this paper was to illustrate the valueof group-based representations for navigation tasks.
Future work will involve improving both theprediction and the control components of our framework.
Finally, we plan on validating our ﬁndings on a real-world robot to fully test the capability of GMPC to handle noisy sensor inputs.
We also plan to investigate ways to improve computation timeto enhance our approach’s real-world applicability, which includes simplifying group representationgeometry and predicting future group states in metric space instead of in image space.