  Reasoning about the future behavior of other agents is critical to safe robot
navigation. The multiplicity of plausible futures is further amplified by the
uncertainty inherent to agent state estimation from data, including positions,
velocities, and semantic class. Forecasting methods, however, typically neglect
class uncertainty, conditioning instead only on the agent's most likely class,
even though perception models often return full class distributions. To exploit
this information, we present HAICU, a method for heterogeneous-agent trajectory
forecasting that explicitly incorporates agents' class probabilities. We
additionally present PUP, a new challenging real-world autonomous driving
dataset, to investigate the impact of Perceptual Uncertainty in Prediction. It
contains challenging crowded scenes with unfiltered agent class probabilities
that reflect the long-tail of current state-of-the-art perception systems. We
demonstrate that incorporating class probabilities in trajectory forecasting
significantly improves performance in the face of uncertainty, and enables new
forecasting capabilities such as counterfactual predictions.
