INFER: INtermediate representations for FuturE pRediction In urban driving scenarios, forecasting future trajectories of surrounding vehicles is of paramount importance. While several approaches for the problem have been proposed, the bestperforming ones tend to require extremely detailed input representations (e.g. image sequences). As a result, such methods do not generalize to datasets they have not been trained on. In this paper, we propose intermediate representations that are particularly wellsuited for future prediction. As opposed to using texture (color) information from images, we condition on semantics and train an autoregressive model to accurately predict future trajectories of traffic participants (vehicles) (see Fig. above). We demonstrate that semantics provide a significant boost over techniques that operate over raw pixel intensitiesdisparities. Uncharacteristic of stateoftheart approaches, our representations and models generalize across different sensing modalities (stereo imagery, LiDAR, a combination of both), and also across completely different datasets, collected across several cities, and also across countries where people drive on opposite sides of the road (lefthanded vs righthanded driving). Additionally, we demonstrate an application of our approach in multiobject tracking (data association). To foster further research in transferable representations and ensure reproducibility, we release all our code and data.33More qualitative and quantitative results, along with code and data can be found at https:rebrand.lyINFERresults

Understanding PedestrianVehicle Interactions with Vehicle Mounted Vision: An LSTM Model and Empirical Analysis Pedestrians and vehicles often share the road in complex inner city traffic. This leads to interactions between the vehicle and pedestrians, with each affecting the otheru0027s motion. In order to create robust methods to reason about pedestrian behavior and to design interfaces of communication between selfdriving cars and pedestrians we need to better understand such interactions. In this paper, we present a datadriven approach to implicitly model pedestriansu0027 interactions with vehicles, to better predict pedestrian behavior. We propose a LSTM model that takes as input the past trajectories of the pedestrian and egovehicle, and pedestrian head orientation, and predicts the future positions of the pedestrian. Our experiments based on a realworld, inner city dataset captured with vehicle mounted cameras, show that the usage of such cues improve pedestrian prediction when compared to a baseline that purely uses the past trajectory of the pedestrian.