  Vision-and-Language Navigation (VLN) requires an agent to navigate to a
remote location on the basis of natural-language instructions and a set of
photo-realistic panoramas. Most existing methods take words in instructions and
discrete views of each panorama as the minimal unit of encoding. However, this
requires a model to match different textual landmarks in instructions (e.g.,
TV, table) against the same view feature. In this work, we propose an
object-informed sequential BERT to encode visual perceptions and linguistic
instructions at the same fine-grained level, namely objects and words, to
facilitate the matching between visual and textual entities and hence "know
what". Our sequential BERT enables the visual-textual clues to be interpreted
in light of the temporal context, which is crucial to multi-round VLN tasks.
Additionally, we enable the model to identify the relative direction (e.g.,
left/right/front/back) of each navigable location and the room type (e.g.,
bedroom, kitchen) of its current and final navigation goal, namely "know
where", as such information is widely mentioned in instructions implying the
desired next and final locations. Extensive experiments demonstrate the
effectiveness compared against several state-of-the-art methods on three indoor
VLN tasks: REVERIE, NDH, and R2R.
