  We propose to predict the future trajectories of observed agents (e.g.,
pedestrians or vehicles) by estimating and using their goals at multiple time
scales. We argue that the goal of a moving agent may change over time, and
modeling goals continuously provides more accurate and detailed information for
future trajectory estimation. In this paper, we present a novel recurrent
network for trajectory prediction, called Stepwise Goal-Driven Network (SGNet).
Unlike prior work that models only a single, long-term goal, SGNet estimates
and uses goals at multiple temporal scales. In particular, the framework
incorporates an encoder module that captures historical information, a stepwise
goal estimator that predicts successive goals into the future, and a decoder
module that predicts future trajectory. We evaluate our model on three
first-person traffic datasets (HEV-I, JAAD, and PIE) as well as on two bird's
eye view datasets (ETH and UCY), and show that our model outperforms the
state-of-the-art methods in terms of both average and final displacement errors
on all datasets. Code has been made available at:
https://github.com/ChuhuaW/SGNet.pytorch.
