Retouchdown: Releasing Touchdown on S treet L earn as a Public Resource for Language Grounding Tasks in Street View (2020)

Abstract: The Touchdown dataset (Chen et al., 2019) provides instructions by human annotators for navigation through New York City streets and for resolving spatial descriptions at a given location. To enable the wider research community to work effectively with the Touchdown tasks, we are publicly releasing the 29k raw Street View panoramas needed for Touchdown. We follow the process used for the StreetLearn data release (Mirowski et al., 2019) to check panoramas for personally identifiable information and blur them as necessary. These have been added to the StreetLearn dataset and can be obtained via the same process as used previously for StreetLearn. We also provide a reference implementation for both Touchdown tasks: vision and language navigation (VLN) and spatial description resolution (SDR). We compare our model results to those given in (Chen et al., 2019) and show that the panoramas we have added to StreetLearn support both Touchdown tasks and can be used effectively for further research and comparison.

## introduction following natural language navigation instructions in visual environments requires addressing multiple challenges in dynamic, continuously changing environments, including language understanding, object recognition, grounding and spatial reasoning. until recently, the most commonly studied domains were map-based @xcite or game-like @xcite @xcite @xcite . these environments enabled substantial progress, but the complexity and diversity of the visual input they provide is limited. this greatly simplifies both the language and vision challenges. to address this, recent tasks based on simulated environments include photo-realistic visual input, such as roomto-room (r2r; @xcite , @xcite and touchdown @xcite , all of which rely on panorama photos. a major challenge of creating simulations that use real-world photographs is they at times capture bystanders and their property. this raises privacy concerns and requires additional care to check for and ensure personally identifiable information (pii) is removed from research resources that are made publicly available. existing resources adopt different strategies to address this. the matterport3d dataset @xcite , which underlies the r2r task, is focused on real-estate data that is curated to exclude pii. this approach is limited to environments of a specific type: houses that are for sale. academic resources that focus on urban street scenes opted to manually collect panoramas from scratch and scrub them for pii @xcite . this is laborious and costly-especially the first stage of collecting the panoramas. as a result, such resources cover relatively small areas. google street view has world-wide scale coverage of street scenes. each panorama in street view has gone through a process to protect the privacy of bystanders and their property. individuals can also request specific panoramas to be removed. as such, it is a resource with the potential to transform the research community's ability to study problems such as street scene understanding and navigation. touchdown relies on 29,641 panoramas from street view; however, because raw images cannot be distributed according to the street view terms-of-service, foot_0 these are not provided with the touchdown data. instead, only image feature vectors are available for direct download with the data, and access to the raw panoramas is subject to availability through apis governed by street view's terms of service. research can be done within a company and shared via publication without releasing data; for example, @xcite discussed models for instruction-conditioned navigation in street view. however, the full impact of the data and research about it can be better realized by making at least some portion of such resources available to the broader research community. in this context, streetlearn @xcite stands out as a publicly available resource of street view data that has been approved for dissemination and use for academic research. 2 streetlearn contains 114k panoramas from new york city and pittsburgh that have been manually checked for pii, ensuring, for example, that faces and license plates are blurred. the dataset can be easily accessed. researchers interested in working with the data simply fill a form stating their goals and commit to update the data periodically with newer versions as they are released. this process balances the ability of researchers to use the data with preserving the privacy and rights of individuals impacted by the data. for example, periodic updates allow google to respond to user takedown requests. to increase the accessibility of touchdown and provide an example of how important data can be responsibly released, we integrate the touchdown task and its corresponding street view data into a new version of streetlearn. this paper reconciles touchdown's mode of dissemination with streetlearn's, which was designed to adhere to the rights of google and individuals while also simplifying access for researchers and improving reproducibility. we also provide open source implementations foot_2 of both the vision-and-language navigation and spatial description resolution tasks, which we show to have a consistent performance with the results in the original touchdown paper. we hope that this release of data and code will enable the entire research community to make further progress on these problems and to consider new questions and tasks enabled by this limited but significant slice of street view data. ## process touchdown includes tasks for natural language navigation and spatial reasoning in realistic urban environments. touchdown uses street view panoramas unfortunately, the development and release of touchdown introduced several challenges that complicate working with the data. even though touchdown itself does not contain street view data, it references specific street view panoramas and depends on access to them via the street view api. this requires any researcher that wishes to work on the data to download large amounts of data using the api, which is inconvenient, error-prone and not aligned with the current google maps termsof-service. also, the panoramas available through the api periodically change, potentially making parts of the data unavailable. this means there is no hope for consistent versioning (which hurts reproducibility) regarding panorama availability because the data collected by each researcher is dependent on the particular time they access it. finally, individual researchers or research groups cannot themselves comply with takedown requests. instead, having the panoramas available as part of streetlearn allows for necessary updates and consistent sharing of the panoramas. to address these challenges, we collect, check and release the touchdown panoramas as part of an update to the 114k existing streetlearn panoramas, which cover regions of new york city and pittsburgh. as shown in figure 1 , streetlearn encompasses the entire region of new york city contained in touchdown; however, the streetlearn panoramas themselves are not sufficient for supporting the touchdown tasks themselves. this is for several reasons. in all, the touchdown tasks encompass 29,641 panoramas. all of these went through extensive manual review by annotators to check for personally identifiable information (pii), such as faces and license plates. regions containing pii were marked as bounding boxes by annotators, and we blurred all of these regions for the final images. ## experiments we re-implement the best-reported models on the navigation and spatial description resolution tasks from @xcite to compare performance with our data release to the original touchdown paper. the key difference between the two settings is that our released panoramas contain additional blurred patches (section 2). another minor difference is that we use a word-piece tokenizer @xcite instead of a full-word tokenizer. ## conclusion the research community is interested in using largescale resources such as street view for work on computer vision and navigation. in order to comply with street view's terms-of-service (which allow for only limited use of its data and apis) and with its data restrictions, we have enriched streetlearn with panoramas from the touchdown study. takedown requests that respect individuals' privacy preferences can be managed through the streetlearn package. we encourage the research community to use only vetted and approved resources like streetlearn, including our new release of the touchdown panoramas, for their street view oriented work. the addition of touchdown to streetlearn (a.k.a. retouchdown) boosts the total panorama count for the streetlearn dataset from 114k to 144k. furthermore, it contains multiple panoramas from the same neighborhoods, which supports work on learning to navigate in a region and testing in that same region using panoramas from a different time. our code for training and evaluating vision-andlanguage navigation agents and spatial description resolution models are publicly available as part of the valan framework @xcite . 4