From: ~Konstantin_Klemmer1
comment: Thanks for that clarification! No further questions for now.
title: Some comments and questions

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper20/Authors
title:  
comment: We’re sorry we forgot to respond. Indeed we agree that autocorrelation is present, and this is exactly why we are using a GP both for the outcome variables directly and for the residuals. Thank you very much again for the literature pointer - we are planning to fix our evaluation to take this into account.

From: ~Konstantin_Klemmer1
comment: Thanks for your response, however you didn't address my concern regarding spatial autocorrelation in your model. Did you test for spatial autocorrelation in (1) your outcome variables (because I would be surprised if there weren't any) and (2) in your model residuals? 
title: Some comments and questions

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper20/Authors
title:  
comment: We thank the reviewer for your thoughtful comments and feedback. Below, we provided detailed responses below for the questions raised by the reviewer:

1) The main principle in using spatial information for analyzing spatial data is to exploit the smooth transitioning of measurements across space. Interpolation with GPs (also referred to as Kriging in geo-statistics) effectively formulates a transition model and interpolates in between observed measurements across space, as shown here, and time, for instance when analyzing temporal data. Similarly, neural networks can also be used for this purpose. The role of GPC and NN-C experiments were to demonstrate the performance of this approach and provide a baseline. Our experiments showed that including images improved upon this baseline, suggesting complementary information provided by GSV. Furthermore, our experiments with models that combine spatial and imagery information using CNN outputs (only using imagery) and GPs improved upon another baseline that used only the image information using CNN (CNN-I).

Integrating spatial information with GPs does not exactly correspond to memorizing lat/lon information but exploiting spatial smoothness of measurements, which has also been investigated for predictions with satellite images by Jean, Xie, and Ermon (2018), and You, Li, Low, Lobell, and Ermon (2017). That said, due to the way we do train/test splits, it might be too easy for the models to learn the underlying spatial structure. As we pointed out in the paper, we are currently extending the work to make stricter evaluations of proposed methods where larger blocks of neighborhoods are excluded from training. 

2) As the reviewer pointed out, the focus of this paper is incorporating spatial information and not on the use of GSV images for predicting income. We agree that the idea of making measurements on SES using street imagery has been introduced previously, and different methods were proposed. While not the focus of this article, we note that there are methodological differences between papers that are cited (including the article the reviewer pointed out) and our approach. Specifically, we use CNNs to directly predict the outcomes automatically extracting relevant features from the entire image, while previous works mostly relied on user-defined features (e.g. extracting car types, roof types, walls, streets etc.). 

Due to space restrictions, we are not able to present additional figures.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper20/Authors
title:  
comment: Thank you for this thoughtful comment. 

The ground truth data for London is available at the LSOA level, hence prediction performances are also tested at this level. Images come from postcode locations that are contained in LSOAs. Postcode level SES information is not available; hence we were not able to evaluate performances at that level. LSOA level predictions are made using averages of postcode level predictions as explained in the text. We have also experimented with using weighted averages based on headcounts data for each postcode; yet this did not change prediction performances. 

One hypothesis we wanted to test was that street imagery contained information (which can be extracted using deep learning methods) on SES measures in addition to spatial information. Our findings showed that using a combination of images and spatial information achieves higher prediction performances compared to approaches where only images or only spatial information is used. This suggests the complementary nature of images and spatial information in making predictions on SES variables we used. 

The Matern kernel is often preferred in spatial statistics as the RBF is rather too smooth. We also empirically tested different kernels in cross-validation experiments; Matern kernel produced higher accuracy.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper20/Authors
title:  
comment: Thank you very much for the constructive feedback. Below, we provide detailed responses below for the questions raised by the reviewer:

** 1) In the CNN-IC approach, why is the CNN only allowed to produce a 1-dimensional feature to be concatenated with the lat/lon spatial information? ** 

We have empirically tested a few architectures using cross validation in the training set where the CNN was designed to produce multi-dimensional features (5D and 10D) that were concatenated with the lat/lon spatial information, as suggested by the reviewer. The performance did not change for dimensions larger than one, hence we used the 1D version in the final paper. 

** 2) For the GP-based approaches, it looks like the outputs of the CNN are mostly treated as target variables — except for in the CNN-I-GPPK approach, where one component of the product kernel takes the CNN-I prediction as input. What if instead of composing kernels, you concatenated the CNN output with the lat/lon similarly to in the CNN-IC approach? Again, it seems possible that using more image features could be helpful for the GPs.**

We thank the reviewer for the suggestion on concatenating the CNN outputs with the lat/lon pairs and making use of more image features. For the first part, we have empirically tested the approach suggested by the reviewer, and performances were much lower; hence we kept the version we had in the initial paper. We believe the cause is that the numerical ranges of lat/lon spatial information and CNN outputs differ substantially. Therefore, even when they are concatenated, different spatial lengths would be required in the same kernel for good performance. Instead, we used product of two different kernels, which is very similar to using different spatial lengths for a variety of kernel choices. We will also update the paper to incorporate this additional discussion.

For the second part of the question, it is an option to make CNN output multiple dimensions that are later fed into the GP kernel. Based on our empirical observations with CNN-IC, mentioned above, we decided to use 1D CNN outputs and feed the GP kernel with this information. We wanted to keep the network architectures across different experiments similar to facilitate direct comparison. That said, we agree that for an extended comparison, reviewer’s suggestions would be an ideal alternative to explore.

From: ~Ransalu_Senanayake1
comment: Dear authors,

You should now be able to see the reviews. We encourage you to engage in the discussion by commenting on the reviews. 

Regards,
Ransalu, Neal, Fabio, and Girish
title: Rebuttal

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper20/AnonReviewer2
evaluation: Thank you for the submission, and I hope the following comments can help improve current and future projects.

Quick summary: The study uses Google Street View (GSV) imagery and spatial coordinates as input to pre-trained VGG networks and GPs to predict income, crowding, and education levels for areas in London. The authors find high correlation between outputs predicted by models using only imagery, only coordinates, or both and actual socioeconomic values.

The application area is meaningful, and machine learning methods for improving the spatial resolution and timeliness of socioeconomic indicators should continue to be pursued.

That said, this work has some significant weaknesses: 

(1) The difference between the previous work and this paper is the use of spatial information. However, this premise is problematic. Spatial information is fixed across time and alone won't allow for updated predictions without new survey data, undermining the original motivation for the work (cheap, frequently updated socioeconomic info). Furthermore, high prediction performance occurs when train/test sets are split at random, and a model memorizes the latitudes and longitudes corresponding to various socioeconomic values in a city. The paper hints at this on page 4 but does not elaborate on its full implications. Since there is nothing in lat, lon information inherently correlated with socio-economic status (on the scale of city neighborhoods, at least), such a model won't generalize across space or time.

(2) Putting aside spatial information leaves the models with GSV imagery. While it is still interesting to see what prediction performance can be obtained using GSV data in London, it is no longer a novel methodological concept due to previous work cited by the authors as well as work done by Glaeser, Kominers, Luca, and Naik (2017) on predicting household income level using GSV imagery.

Some comments on presentation: the paper is mostly clear in its writing. Readers would benefit from a figure showing examples of GSV imagery used as well as an improved figure of CNN architecture (not a table format).

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper20/AnonReviewer2
relevance: 4: high
impact: 2: low
novelty: 2: low

From: ~Konstantin_Klemmer1
comment: Thank you for your submission, I think the topic of your paper is quite appealing. That being said, I am a bit skeptical about the use of deep learning for analysing social processes. Sure, it works well with image data and scales fine, but to actually tackle the problems that you want to tackle, more inference is needed on the processes that drive income, crowding and educational attainment. Just predicting poverty does not help as much as finding driving, causing factors or at least getting reliable uncertainty measures. Your GP approach of course offers more inference, but overall, this study for me falls a bit into the pit of purely predicting social outcomes rather than explaining them. I fully acknowledge though that some of this bias might be personal preference.
Now, I have some more direct comments and questions:
-  A UK postcode usually covers more than one address, so one 360 degree view (in four images) doesn't necessarily cover all addresses in a postcode, hence making the images not always representative. Did you use these images as lined out or am I missing something here?
- Did you test for spatial autocorrelation in your outcome variable? Working in a highly spatially autocorrelated regime, k-fold CV MIGHT still be vulnerable to overfitting, even though GPs are a good way to account for that. Check out spatial CV (https://ieeexplore.ieee.org/document/6352393/ or https://onlinelibrary.wiley.com/doi/full/10.1111/ecog.02881).
- It is not quite clear to me from the paper whether you predict on the level of each image or on LSOA level?
- Why did you use a Matern kernel? Not opposed to it, just wondering. I thought RBF would be the standard kernel for geospatial applications.

FUTURE WORK:
- It would be interesting to get some information about computing times, if applicable, since that is one of the main reasons for using deep learning techniques over something like GPs.
- It might be interesting to do some object recognition with the images, as to identify specific objects that can be associated with higher or lower socioeconomic status. This relates to my comment on lacking inference / acknowledgment of driving social processes.

title: Some comments and questions

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper20/AnonReviewer1
relevance: 5: very high
impact: 4: high
novelty: 2: low

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper20/AnonReviewer1
evaluation: Large-scale data sources such as remote sensing and Google Street View (GSV) imagery hold significant promise for measuring socioeconomic status (SES) at high spatial and temporal resolution. However, traditional ML methods such as CNNs, although good at extracting features and making predictions from imagery, fail to take into account spatial information that is potentially very informative. This paper empirically investigates and compares different approaches for combining spatial and image (GSV) information for predicting income, crowding, and education levels in London, UK. They concluded that there is significant value in incorporating spatial information, and evaluate several CNN and GP based approaches.

Although previous studies have considered combining neural networks and Gaussian processes in the context of satellite imagery, this paper seems to be the first to apply this combination to street view imagery. As far as I know, their dataset is also unique (UK 2011 Census and Greater London Authority household survey 2015) — previous work using satellite imagery mainly have focused on rural environments. Predicting socioeconomic indicators from satellite imagery is much more difficult in urban environments due to the greater heterogeneity, so methods based on street-level images could provide much more granular information.

I have a few questions about choices made in Section 3: In the CNN-IC approach, why is the CNN only allowed to produce a 1-dimensional feature to be concatenated with the lat/lon spatial information? For the GP-based approaches, it looks like the outputs of the CNN are mostly treated as target variables — except for in the CNN-I-GPPK approach, where one component of the product kernel takes the CNN-I prediction as input? What if instead of composing kernels, you concatenated the CNN output with the lat/lon similarly to in the CNN-IC approach? Again, it seems possible that using more image features could be helpful for the GPs.

The authors give a thoughtful analysis of their quantitative results, noting that the GPC model is able to achieve good performance without using any image information at all — for these tasks, it seems like spatial information is actually more useful than street-view images, indicating that CNN-only approaches would be missing the point in some ways. On the other hand, they also make a note that a stricter evaluation with larger geographical train/test neighborhood splits could swing the results a bit more in favor of the CNNs since the GPs would have less spatial context to work with.