IVOA: Introspective Vision for Obstacle Avoidance Vision, as an inexpensive yet information rich sensor, is commonly used for perception on autonomous mobile robots. Unfortunately, accurate visionbased perception requires a number of assumptions about the environment to hold xe2x80x93 some examples of such assumptions, depending on the perception algorithm at hand, include purely lambertian surfaces, texturerich scenes, absence of aliasing features, and refractive surfaces. In this paper, we present an approach for introspective vision for obstacle avoidance (IVOA) xe2x80x93 by leveraging a supervisory sensor that is occasionally available, we detect failures of stereo visionbased perception from divergence in plans generated by vision and the supervisory sensor. By projecting the 3D coordinates where the plans agree and disagree onto the images used for visionbased perception, IVOA generates a training set of reliable and unreliable image patches for perception. We then use this training dataset to learn a model of which image patches are likely to cause failures of the visionbased perception algorithm. Using this model, IVOA is then able to predict whether the relevant image patches in the observed images are likely to cause failures due to vision (both false positives and false negatives). We empirically demonstrate with extensive realworld data from both indoor and outdoor environments, the ability of IVOA to accurately predict the failures of two distinct vision algorithms.

Predicting Unobserved Space for Planning via Depth Map Augmentation Safe and efficient path planning is crucial for autonomous mobile robots. A prerequisite for path planning is to have a comprehensive understanding of the 3D structure of the robotu0027s environment. On Micro Air Vehicles (MAVs) this is commonly achieved using lowcost sensors, such as stereo or RGBD cameras. These sensors may fail to provide depth measurements in textureless or IRabsorbing areas and have limited effective range. In path planning, this results in inefficient trajectories or failure to recognize a feasible path to the goal, hence significantly impairing the robotu0027s mobility. Recent advances in deep learning enables us to exploit prior experience about the shape of the world and hence to infer complete depth maps from color images and additional sparse depth measurements. In this work, we present an augmented planning system and investigate the effects of employing stateoftheart depth completion techniques, specifically trained to augment sparse depth maps originating from RGBD sensors, semidense methods and stereo matchers. We extensively evaluate our approach in online path planning experiments based on simulated data, as well as global path planning experiments based on real world Micro Air Vehicle (MAV) data. We show that our augmented system, provided with only sparse depth perception, can reach onpar performance to ground truth depth input in simulated online planning experiments. On real world MAV data the augmented system demonstrates superior performance compared to a planner based on very dense RGBD depth maps.