  Predicting the behavior of road users, particularly pedestrians, is vital for
safe motion planning in the context of autonomous driving systems.
Traditionally, pedestrian behavior prediction has been realized in terms of
forecasting future trajectories. However, recent evidence suggests that
predicting higher-level actions, such as crossing the road, can help improve
trajectory forecasting and planning tasks accordingly. There are a number of
existing datasets that cater to the development of pedestrian action prediction
algorithms, however, they lack certain characteristics, such as bird's eye view
semantic map information, 3D locations of objects in the scene, etc., which are
crucial in the autonomous driving context. To this end, we propose a new
pedestrian action prediction dataset created by adding per-frame 2D/3D bounding
box and behavioral annotations to the popular autonomous driving dataset,
nuScenes. In addition, we propose a hybrid neural network architecture that
incorporates various data modalities for predicting pedestrian crossing action.
By evaluating our model on the newly proposed dataset, the contribution of
different data modalities to the prediction task is revealed. The dataset is
available at https://github.com/huawei-noah/PePScenes.
