We present a novel place recognition system that adapts state-of-the-art object proposal techniques to identify potential landmarks within an image. The proposed system utilizes convolutional network features as robust landmark descriptors to recognize places despite severe viewpoint and condition changes, without requiring any environment-specific training. The colored boxes in the images above show ConvNet landmarks that have been correctly matched between two significantly different viewpoints of a scene, thus enabling place recognition under these challenging conditions. Abstract-Place recognition has long been an incompletely solved problem in that all approaches involve significant compromises. Current methods address many but never all of the critical challenges of place recognition -viewpoint-invariance, condition-invariance and minimizing training requirements. Here we present an approach that adapts state-of-the-art object proposal techniques to identify potential landmarks within an image for place recognition. We use the astonishing power of convolutional neural network features to identify matching landmark proposals between images to perform place recognition over extreme appearance and viewpoint variations. Our system does not require any form of training, all components are generic enough to be used off-the-shelf. We present a range of challenging experiments in varied viewpoint and environmental conditions. We demonstrate superior performance to current state-of-theart techniques. Furthermore, by building on existing and widely used recognition frameworks, this approach provides a highly compatible place recognition system with the potential for easy integration of other techniques such as object detection and semantic scene interpretation. Visual place recognition research has been dominated by sophisticated local feature-based techniques such as SIFT and SURF keypoints, hand-crafted global image descriptors such as GIST and bag-of-words approaches. However, as robots operate for longer periods of time in real-world environments, the problem of changing environmental conditions has come to the fore, where conventional recognition approaches fail. To address this problem, a number of techniques have been adopted -matching image sequences [27, 29, 33, 28] , creating shadow-invariant images [6, 43, 25, 24] , learning salient image regions [26] or learning temporal models that allow the prediction of occurring changes [31] . Recent research has also demonstrated how generic deep learning-based features trained for object recognition can be successfully applied in the domain of place recognition [41, 3] . However, all current approaches have introduced at least one significant performance or usability compromise, whether it be a lack of invariance to camera viewpoint changes [27, 28] , extensive environment-specific training requirements [26] , or the lack of appearance change robustness [7] . If visual place recognition is to be truly robust, it must simultaneously address three critical challenges: 1) condition invariance; 2) viewpoint invariance; and 3) generality (no environment-specific training requirements). In this paper, we present a unified approach that addresses all three of these challenges. We use a state-of-the-art object proposal method to discover potential landmarks in the images. A convolutional network (ConvNet) is then used to extract general purpose features for each of these landmark proposals. We show that the ConvNet features are robust to both appearance and viewpoint change; the first two critical challenges. We also emphasize that landmark proposals require no training and the ConvNet is pre-trained on ImageNet, a generic image database; the third critical challenge. By conducting experiments on a number of datasets we show that our system is training-free in that no task-specific or even site-specific training is required. We also highlight that only single images are required for matching and the system does not require image sequences. We demonstrate the generality of our system on a number of existing datasets and introduce new challenging place recognition datasets, while comparing to state of the art methods. The novel contributions of this paper are: 1) A place recognition system that is robust to viewpoint and appearance variation, requiring no environment specific training, and 2) The introduction of new challenging datasets exhibiting extreme viewpoint and appearance variation. The paper proceeds as follows. Section II provides a brief overview of related work. The method is described in detail in Section III followed by an overview of the four sets of experiments. We present results in Section V before concluding with a discussion and outlining future work. The focus of research in place recognition has recently moved from recognizing scenes without significant appearance changes to more challenging, but also more realistic changing environments. Place Recognition: Methods that address the place recognition problem span from matching sequences of images [27, 17, 40, 33, 29] , transforming images to become invariant against common scene changes such as shadows [6, 43, 25, 24, 21] , learning how environments change over time and predicting these changes in image space [30, 21, 31], particle filterbased approaches that build up place recognition hypotheses over time [23, 39, 22] , or build a map of experiences that cover the different appearances of a place over time [5] . Learning how the appearance of the environment changes generally requires training data with known frame correspondences. [17] builds a database of observed features over the course of a day and night. [30, 31] presents an approach that learns systematic scene changes in order to improve performance on a seasonal change dataset. [26] learns salient regions in images of the same place with different environmental conditions. Beyond the limitation of requiring training data, the generality of these methods is also currently unknown; these methods have only been demonstrated to work in the same environment and on the same or very similar types of environmental change to that encountered in the training datasets. Although point feature-based methods were shown to be robust against viewpoint changes [7, 8, 38] , to the authors' knowledge, significant changes in both viewpoint and environmental conditions have not been addressed in the literature. We show that robustness to variation in both cases can be addressed without site-specific training. Feature-based Approaches: SIFT [20], SURF [1] and a number of subsequent feature detectors have been demonstrated to display a significant degree of pose invariance but only a limited degree of condition-invariance (illumination, atmospheric conditions, shadows, seasons). Perceptual change as drastic as that illustrated in Fig. 1 has been shown to be challenging for conventional feature detectors [27, 44] and while FAB-MAP [7] is robust with respect to viewpoint changes, it is known to fail in conditions with severe appearance changes [29, 31, 13] . Furthermore, [11, 34] argued that FAB-MAP does not generalize well to new environments without learning a new site-specific vocabulary. [26] shows that patches and region-based methods within an image can exhibit the same robustness as whole-image techniques while retaining some robustness to scale variation, and thus achieve some of the advantages of both point and wholeimage features. However, extensive site-specific training was required. In this research we extend the advantages of regionbased methods to address both viewpoint and environmental changes without the requirement for site-specific training. A commonality between all these approaches is that they rely on a fixed set of hand-crafted traditional features or operate on the raw pixel level. A recent trend in computer vision, and especially in the field of object recognition and detection, is to exploit learned features using deep convolutional networks (ConvNets). The most prominent example of this trend is the annual ImageNet Large Scale Visual Recognition Challenge where for the past two years many of the participants have used ConvNet features [36] . Several research groups have shown that ConvNets outperform classical approaches for object classification or detection that are based on hand-crafted features [19, 37, 10, 12, 35] . The availability of pre-trained network models makes it easy to experiment with such approaches for different tasks: the software packages Overfeat [37] and Decaf [10] or its successor Caffe [16], provide similar network architectures that were pre-trained on the ImageNet ILSVRC dataset [36] . Recent studies have shown that state-of-the-art performance in place recognition can be achieved with networks trained using generic data: [41] demonstrated that ConvNet features representing the whole image outperform current methods for changing environmental conditions. However, whole-image features suffer from sensitivity to viewpoint change. We show that by combining the power of ConvNets and region-based features rather than using whole-image representations, a large degree of robustness to viewpoint change can be achieved. Consequently in this research we build on the best performing aspects of the state of the art; the recognition performance of ConvNet approaches [41] , and the robustness of regionbased methods to viewpoint change [26] . III. PROPOSED SYSTEM In this section we describe the five key components of our proposed place recognition system: 1) landmark proposal extraction from the current image 2) calculation of a ConvNet feature for each proposal 3) projection of the features into a lower dimensional space 4) calculation of a matching score for each previously seen image 5) calculation of the best match Fig. 2 illustrates our system. The approach has several properties that distinguishes it from previous work: • The system does not require any task-specific or sitespecific training. It uses an off-the-shelf pre-trained con-
An outdoor and indoor place recognition approach presented by Sunderhauf et al. REF , employs a reliable landmark proposal method together with strengths in Convolutional Neural Networks(CNN).
Place Recognition with ConvNet Landmarks: Viewpoint-Robust, Condition-Robust, Training-Free
Robotics: Science and Systems
Engineering Computer Science