Combining street imagery and spatial information for measuring socioeconomic status
Abstract
Emerging sources of large-scale data, such as remote sensing, street imagery, GPS trajectories, coupled with advances in deep learning methods have the potential for signiﬁcantly advancing how fast, how frequently, and how locally we can measure urban features and population characteristics to inform and evaluate policies.
One such example that attracted increasing attention from the research community is utilizing street level imagery for various measurement tasks in this broader context.
We believe incorporating spatial information with Gaussian Processes (GPs) can give us better performance when using street images.
To test this hypothesis, we empirically investigated multiple approaches for combining spatial and street image information using neural networks and GPs for predicting income, crowding, and education levels in London, UK.
Results demonstrated using GPs only with spatial information (without any inputs from images) gives us a good baseline.
Complementary value of street images were demonstrated for the socioeconomic status measures we investigated.
Further, our results showed superior performance of GP regression of residuals compared to other methods including feeding spatial information as input directly to neural networks.
Introduction and Related Work
Urbanization and social inequalities are two of the major policy themes of our time, intersecting in large cities where rich and poor live side by side.
Reducing inequalities is at the forefront of the global sustainable development agenda as well as a policy objective in many cities [GLA, 2017, 2018].
However, datasets for informing these policies and measuring their actual impacts are currently from disjointed and inefﬁcient surveillance systems.
Measuring socioeconomic status (SES) at high spatial and temporal resolution, for instance, is crucial yet poses a signiﬁcant challenge even in developed parts of the world.
Emerging sources of large-scale data, such as remote sensing, imagery and GPS trajectories, have the potential for signiﬁcantly advancing how fast, how frequently and how locally we can measure urban features and population characteristics.
Researchers both from domain sciences and computer science are increasingly interested in tackling problems associated with applying advanced learning techniques focusing on automatic feature extraction to measurement and data collection tasks.
Relevant applications of machine learning with imagery include: poverty detection [Jean et al., 2016, Steele et al., 2017, Xie et al., 2015] and harvest size and crop yield [Lobell, 2013, You et al., 2017] from satellite data, and income [Gebru et al., 2017], perceived safety [Naik et al., 2014, 2017], and greenness and openness
[Seiferling et al., 2017,
Richards and Edwards, 2017] from Google Street View (GSV) images.
Here, building on our previous work, we use London to evaluate the feasibility of using street level images for measuring multiple indicators of socioeconomic status including income, crowding, and education.
We believe incorporating spatial information with Gaussian Processes (GPs) can give us better performance; to test this hypothesis we compare measurement performances of multiple methods for combining street images and spatial information using neural networks (NN) and GPs.
Two recent studies by You et al.
[2017] and Jean et al.
[2018] have proposed using GPs in the context of making predictions from satellite imagery.
In our work, we also investigate the use of GPs combined with NN, but on a problem with different types of images (i.e. street level images) and multiple SES indicators as outputs.
Our focus here is to investigate the performance of multiple approaches, some with NN only and some with a combination of NN and GPs.
Data and Methods
We ﬁrst obtained the Ofﬁce for National Statistics (ONS)
Postcode Directory for the UK 1 and selected the 181,150 postcodes assigned to the 33 local authority districts of the Greater London administrative area.
Unique images were available from GSV for 119,681 postcode locations in London.
Four images for each location were extracted by specifying the camera direction (i.e., 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ) relative to the GSV vehicle to cover a 360 ◦ view.
Hence, we used a total of 478,724 images corresponding to 119,681 locations in London.
Output data (labels) was obtained from the UK Census 2011 2 and Greater London Authority household income estimates 2015 3 .
In the UK, Census results and other neighborhood statistics are reported using output areas designed speciﬁcally for statistical purposes.
The output labels used here are detailed below and were available for lower super output areas (LSOA; average population of 1, with a total of 4,833 LSOAs in London).
For income, we used the mean annual household income estimates by Greater London Authority, reported for each LSOA.
For the regression tasks, the income values that were used are represented in 10000 GBPs taking values between 3 and 19.
For education, we used the percentage of population with low educational attainment levels (i.e. people who do not have at least a Level 2 education where the ﬁve categories for highest attained qualiﬁcation were: no qualiﬁcation, Level 1, Level 2, Level 3, and Level 4 and above).
For crowding, we used the percentage of households classiﬁed as being overcrowded (i.e., having at least one fewer room than required) by ONS.
ONS derives the number of rooms required using a formula based on ages of the household members and their relationships to each other.
For education and crowding, percentages of deprived populations for each LSOA are used, hence the values are between 0 and 1.
Images correspond to photos taken by GSV vehicles at speciﬁc locations represented by coordinates and postcodes.
Each postcode location is assigned to a single LSOA where ground truth data is available; individuals GSV images were matched with output labels using this information.
We took a transfer learning approach where we used VGG16 network [Simonyan and Zisserman, 2014] pre-trained with ImageNet [Russakovsky et al., 2015] as a ﬁxed feature extractor to convert
RGB images to 4096 dimensional codes.
On top of the convolutional neural network (CNN) layers with pre-trained weights, we trained a fully connected NN to perform regression from four GSV images extracted for each location.
In some of our experiments longitude and latitude coordinates were also fed as inputs to the trained part of the network with a slightly different network architecture explained further in the next section.
We compute LSOA level predictions as the average of postcode level predictions assigned to that LSOA.
To evaluate performance, we used 60% of LSOAs for training and made predictions for the held-out 40% test set; 5-fold cross validation was carried out for hyper parameter tuning and NN architecture selection using the training set only.
In a set of our experiments, GP regression was used as an additional separate step for incorporating spatial information (i.e longitude and latitude coordinates of LSOA centroids) on top of CNN outputs from images, as detailed in the next section.
Experiments and Results
We compared seven different approaches in terms of prediction performance.
In the ﬁrst approach,
CNN-I, four images for each location were fed into the VGG16 network and 4x4096 codes were extracted from layer
h 5 .
A fully connected NN was trained using the architecture presented in Table 1.
The main principle for the fully connected part is that the network used all four images from each location jointly in the four channels shown; the information coming from different channels are then aggregated and fed into the ﬁnal layers to yield a single continuous output.
No location information is used in this experiment.
The second approach, CNN-IC, used spatial information (i.e. latitude/longitude) in addition to GSV images, using the architecture shown in Table 1.
The same architecture from CNN-I was used for the initial layers, the 1D ﬁnal layer output was then concatenated with longitude and latitude coordinates associated with each image location yielding a 3D input to the following layers of the network.
NN-C, the third approach was used as baseline comparison where only coordinate information is used.
The same architecture as in CNN-IC was used where the 1D input from CNN-I architecture was replaced with random noise as in Table 1, allowing the use of exactly the same architecture.
[Table]
[Table]
[Table] [Table] different camera directions 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ capturing a 360 ◦ view.
For the next set of approaches, several alternatives for using GPs to incorporate spatial information were investigated.
The main idea is to get LSOA level predictions using CNN-I, and then incorporate spatial information using a GP based on centroid coordinates of LSOAs.
In all the approaches, we used Matern-3/2 kernel for the GPs based on initial ﬁve-fold cross-validation experiments on the training set.
Throughout the experiments, we used the GPy [GPy, 2012] package to ﬁt the GP models, where all the parameters of the Matern-3/2 kernel and the noise parameters were determined using maximum-likelihood (II) estimation, optimizing the log-marginal likelihood.
The ﬁrst GP-based approach, GP on Coordinates - GPC, does not use the image information at all and simply ﬁts a GP regressor on the coordinates to predict labels from the training set.
The second approach, GP for Residuals on coordinates - CNN-I-GPR, ﬁts a GP regressor to the residuals of the
CNN-I predictions.
The third approach, GP on coordinates and Predictions with Product KernelCNN-I-GPPK, feeds predictions of CNN-I as additional input to the GP regressor that uses a product kernel with two terms, the ﬁrst using spatial coordinates and second CNN-I predictions.
We note that using a linear model for the second kernel yields a similar method as CNN-I-GPR hence we explore Matern-3/2 kernel instead.
We also empirically evaluated an alternative approach, where the
CNN-I predictions were concatenated with the spatial coordinates instead of using a product kernel, the performances were lower.
We believe the cause is the differences in numerical ranges; different spatial lengths would be required when concatenated in the same kernel for good performance.
In the last approach, we refer to as CNN-I-GPC, we use GP based on coordinates as a spatial smoothing model and ﬁt it to ground truth labels on the training set and CNN-I predictions on the test set.
Observation noise is assumed minimal on the ground truth set while a higher value is assigned to the predictions on the test set.
This latter higher value of noise used on predictions was estimated based on ﬁve-fold cross validation experiments on the training set.
[Table]
Results are shown in Table 2.
In line with previous work, prediction performances for all output labels were high, demonstrating the potential for using GSV images and corresponding location information for improving measurements in cities.
There are differences in performance across different labels, that potentially relate to differences in available information contained in imagery and space.
We note the difference in how accuracy metrics relate to interpretable units.
For income, MAPE will relate to errors in absolute values of income in GBPs.
For crowding and education, it relates to errors in the percentage of the population that live in overcrowded households and with lower educational attainment.
Hence we expect the former to have lower values for MAPE, and higher values for
NRMSE.
In this study [Table] our focus was on performance comparison of different approaches for combining imagery and spatial information; we observed several interesting points.
First, all GP-based approaches for combining imagery and spatial information (i.e. CNN-I-GPR, CNN-I-GPPK, and CNN-I-GPC) outperformed the NN based approach i.e. CNN-IC.
Strikingly, GPC that uses only the coordinates achieved very similar performance, even higher in some cases, compared to CNN-I and CNN-IC.
It gives us a good baseline to which image only predictions should always be compared.
Using NN only on the coordinates i.e. NN-C did not show similar success.
These results demonstrated the richness of spatial information and success of GPs in leveraging it to improve performances of learning based measurement tasks.
It also suggests that a stricter evaluation, leaving out larger neighborhoods, is necessary, as the current train/test split might favor GP regression with test locations located among train locations.
Second, combined use of imagery and coordinates improved performance compared to baseline where only images or coordinates were used.
Speciﬁcally, CNN-IC outperformed CNN-I and CNN-C for all indicators.
Similarly, CNN-I-GPR and CNN-I-GPPK outperform GPC.
Such difference suggests complementary information is available from GSV images for predicting these measures.
Third, the way in which one integrates CNN based approaches with GP also made a difference.
CNN-I-GPR and CNN-I-GPPK yielded higher performance increase compared to CNN-I-GPC.
Both of these models aim to learn the residual error structure with different means.
Their performance were similar across the different measures.
Conclusions
In experiments with multiple SES indicators from London, i.e. income, crowding, and education, we ﬁnd that GP-based methods for explicitly incorporating spatial information with GSV imagery can substantially improve prediction and measurement performance.
The focus here was on making predictions for the city where we also do the training.
Transferability of learned features to different target cities using CNN-I only was investigated in previous work.
Investigation of transferability of learned spatial correlation structures to different target cities will be very valuable especially in places where measurement data on SES indicators is not available or outdated - we leave this to future work.
Additionally, GPs conveniently give Bayesian posteriors that are useful for quantifying uncertainty.
In future work, we also plan to investigate predictions from GPs not just for RMSE but also in terms of how well-calibrated the uncertainty intervals are.