  We are interested in understanding whether retrieval-based localization
approaches are good enough in the context of self-driving vehicles. Towards
this goal, we introduce Pit30M, a new image and LiDAR dataset with over 30
million frames, which is 10 to 100 times larger than those used in previous
work. Pit30M is captured under diverse conditions (i.e., season, weather, time
of the day, traffic), and provides accurate localization ground truth. We also
automatically annotate our dataset with historical weather and astronomical
data, as well as with image and LiDAR semantic segmentation as a proxy measure
for occlusion. We benchmark multiple existing methods for image and LiDAR
retrieval and, in the process, introduce a simple, yet effective convolutional
network-based LiDAR retrieval method that is competitive with the state of the
art. Our work provides, for the first time, a benchmark for sub-metre
retrieval-based localization at city scale. The dataset, additional
experimental results, as well as more information about the sensors,
calibration, and metadata, are available on the project website:
https://uber.com/atg/datasets/pit30m
