Talk to the Vehicle: Language Conditioned Autonomous Navigation of Self Driving Cars We propose a novel pipeline that blends encodings from natural language and 3D semantic maps obtained from visual imagery to generate local trajectories that are executed by a lowlevel controller. The pipeline precludes the need for a prior registered map through a local waypoint generator neural network. The waypoint generator network (WGN) maps semantics and natural language encodings (NLE) to local waypoints. A local planner then generates a trajectory from the ego location of the vehicle (an outdoor car in this case) to these locally generated waypoints while a lowlevel controller executes these plans faithfully. The efficacy of the pipeline is verified in the CARLA simulator environment as well as on local semantic maps built from realworld KITTI dataset. In both these environments (simulated and realworld) we show the ability of the WGN to generate waypoints accurately by mapping NLE of varying sequence lengths and levels of complexity. We compare with baseline approaches and show significant performance gain over them. And finally, we show real implementations on our electric car verifying that the pipeline lends itself to practical and tangible realizations in uncontrolled outdoor settings. In loop execution of the proposed pipeline that involves repetitive invocations of the network is critical for any such languagebased navigation framework. This effort successfully accomplishes this thereby bypassing the need for prior metric maps or strategies for metric level localization during traversal.

Crowdsourced Semantic Edge Mapping for Autonomous Vehicles Highly accurate maps of the road infrastructure are a crucial cornerstone for selfdriving cars to enable navigation in complex traffic scenarios. Traditional methods for creating detailed maps of road environments involve expensive survey vehicles that cannot keep up with the frequent changes in the road network. In this paper, we propose a novel method to derive detailed highdefinition maps by crowd sourcing data using commodity sensors. Our system uses multisession featurebased visual SLAM to align submaps recorded by individual vehicles on a central backend server. We reconstruct 3D boundaries of road infrastructure elements such as road markings and road boundaries from semantic object contours detected in keyframes by a neural network. The result is a concise map of semantically meaningful objects suitable both for localization and higherlevel planning tasks of automated vehicles. We evaluate our method on realworld data against a globally referenced groundtruth map demonstrating a high level of detail and metric accuracy.