  We study the task of embodied visual active learning, where an agent is set
to explore a 3d environment with the goal to acquire visual scene understanding
by actively selecting views for which to request annotation. While accurate on
some benchmarks, today's deep visual recognition pipelines tend to not
generalize well in certain real-world scenarios, or for unusual viewpoints.
Robotic perception, in turn, requires the capability to refine the recognition
capabilities for the conditions where the mobile system operates, including
cluttered indoor environments or poor illumination. This motivates the proposed
task, where an agent is placed in a novel environment with the objective of
improving its visual recognition capability. To study embodied visual active
learning, we develop a battery of agents - both learnt and pre-specified - and
with different levels of knowledge of the environment. The agents are equipped
with a semantic segmentation network and seek to acquire informative views,
move and explore in order to propagate annotations in the neighbourhood of
those views, then refine the underlying segmentation network by online
retraining. The trainable method uses deep reinforcement learning with a reward
function that balances two competing objectives: task performance, represented
as visual recognition accuracy, which requires exploring the environment, and
the necessary amount of annotated data requested during active exploration. We
extensively evaluate the proposed models using the photorealistic Matterport3D
simulator and show that a fully learnt method outperforms comparable
pre-specified counterparts, even when requesting fewer annotations.
