Monocular Plan View Networks for Autonomous Driving Convolutions on monocular dash cam videos capture spatial invariances in the image plane but do not explicitly reason about distances and depth. We propose a simple transformation of observations into a birdxe2x80x99s eye view, also known as plan view, for endtoend control. We detect vehicles and pedestrians in the first person view and project them into an overhead plan view. This representation provides an abstraction of the environment from which a deep network can easily deduce the positions and directions of entities. Additionally, the plan view enables us to leverage advances in 3D object detection in conjunction with deep policy learning. We evaluate our monocular plan view network on the photorealistic Grand Theft Auto V simulator. A network using both a plan view and front view causes less than half as many collisions as previous detectionbased methods and an order of magnitude fewer collisions than pure pixelbased policies.

Realtime 3D Traffic Cone Detection for Autonomous Driving. Considerable progress has been made in semantic scene understanding of road scenes with monocular cameras. It is, however, mainly related to certain classes such as cars and pedestrians. This work investigates traffic cones, an object class crucial for traffic control in the context of autonomous vehicles. 3D object detection using images from a monocular camera is intrinsically an illposed problem. In this work, we leverage the unique structure of traffic cones and propose a pipelined approach to the problem. Specifically, we first detect cones in images by a tailored 2D object detector; then, the spatial arrangement of keypoints on a traffic cone are detected by our deep structural regression network, where the fact that the crossratio is projection invariant is leveraged for network regularization; finally, the 3D position of cones is recovered by the classical Perspective nPoint algorithm. Extensive experiments show that our approach can accurately detect traffic cones and estimate their position in the 3D world in real time. The proposed method is also deployed on a realtime, critical system. It runs efficiently on the lowpower Jetson TX2, providing accurate 3D position estimates, allowing a racecar to map and drive autonomously on an unseen track indicated by traffic cones. With the help of robust and accurate perception, our racecar won both Formula Student Competitions held in Italy and Germany in 2018, cruising at a topspeed of 54 kmph. Visualization of the complete pipeline, mapping and navigation can be found on our project page.