From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper22/Authors
title: Revision online
comment: We have now added a revised version including changes motivated by the reviewer suggestions.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper22/Authors
title: Response to comments from Reviewer 1
comment: Dear Reviewer,
Thank you for your valuable thoughts and comments on our submission. We have added our response to each of the points raised by you below

“- There is a good chance that the demographics of these big cities would have changed a lot over the span of 5 years(2012 to 2017). Since the delay in crime reporting also largely depends on the individuals themselves, how does this change in the demographics influence the predicting power of the model? Would it be better if we analyzed smaller time frames individually?” 

Demographic changes are something we want to look into going forward, however due to the scope of this first, preliminary paper and after all not too substantially long time span we opted for static predictors. We mention a more dynamic model as a future research direction in the conclusion. We did also experiment with smaller time frames, that is i.e. just predicting reporting delays for crime reported in 1 or 2 years. These experiments did not show substantially different predictive performance. We will add a sentence mentioning this into the paper. 

"- Also the reporting delays (or even the victim not reporting the crime) might depend on how the accused was related to the victim and the location (workplace, outdoor spaces etc). If the accused was for instance, the manager of the victim in a company, the victim might refrain from complaining if the manager threatened to remove them from employment. Is there a possibility to account for such factors?"

Yes, after talking to the NYPD special victims unit about this issue, we decided to drop all rape crime from our data that was committed in the outdoors or at other places where we are very uncertain about victim – perpetrator relationship. Our final dataset consists only out of rapes that happened in the (1) domestic space (residential buildings) and the (2) professional space (schools, office buildings,…). This way we try to ensure that the socio-demographic predictors are actually representative of the circumstances. We also assume that victim and perpetrator do know each other which makes the reporting delay so important to examine, as it represents exactly the social stigmatization and repercussions that you mention.  We will make this more clear in our data section.

"- An R^2 0.243(NY) or 0.176 probably hints that there needs to be a lot more factors that could be incorporated. There is a chance that these factors might be purely random as well. How could this be handled? Overall it is great topic and interesting work."

We are currently looking into more potential predictor variables that are more representative of cultural backgrounds, such as the language spoke at home. Yet, we assume that a lot of the true process of reporting delays will remain unknown, without getting more specific victim and perpetrator information. We also don’t see this as the purpose of our study, as we try to look into the effect of temporal, spatial and socio-demographic characteristics. Studying these effects might enable the development of targeted, subpopulation-level policies, which could not be inferred from victim and / or perpetrator information.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper22/Authors
title: Response to comments from Reviewer 2
comment: Dear Reviewer,
Thank you for your valuable comments to our submission. We agree very much with your ideas for further research directions and fully acknowledge that this paper is a first step into a novel application field for spatio-temporal machine learning. Please see below for more detailed comments:

"One thing I didn't know before reading this paper was that *disaggregated* data on rape statistics are publicly available. This immediately led me to ask questions about the potential identifiability of individual rape victims in public datasets (for example, by attackers conditioning on prior knowledge of particular persons having been raped), since age, gender, race, occurrence date and spatial location are all variables available in the disaggregated dataset. The paper is not about privacy or security as such, but the question seems more relevant in light of the paper's key finding for the disaggregated dataset that only victim-specific information seems to lead to meaningful improvements in model predictions. I might therefore have expected the authors to at least briefly highlight identifiability and data security questions, for example in the Conclusion section when talking about potential for training future models on even more expansive (and personal) datasets." 

This is indeed a very interesting question. As you mentioned already, we did not focus on that particular aspect of the data, however we plan on addressing the issue more thoroughly in future work. We will add a note regarding this in our conclusion section! Besides, also non victim-specific data like date of crime or (in the case of LA) block level geography might be potential threats to victim identification.

"The particular machine learning models deployed seem fairly standard. The paper addresses decisions being made about spatiotemporal processes, although the models being deployed (e.g. Gaussian process / random forest classification) don't seem to give especially privileged status to space or time. "

While it is true that temporal structures are integrated into the model only as discrete features (season, holiday, weekend), we do address the spatial relationship more in-depth, first by testing for spatial autocorrelation in the outcome variable and second by running explicit, spatial only models for both the aggregated and disaggregated case. This allows us to assess the presence of spatial processes more thoroughly and for instance leads to the finding that LA exhibits less spatial variation then NY. We do acknowledge though that the methodology can be improved upon and mention this as future research directions in our conclusion.

"The authors use different techniques (regression vs classification) to analyze aggregated vs disaggregated data, and hence use different performance metrics, which makes it difficult to address the impact of aggregation on model performance (e.g. bias-variance trade-off). A Bayesian approach might have been more suitable for this application, since then aggregation simply corresponds to marginalizing over the aggregated variables, and other aspects of the model can be more directly compared between the aggregated and disaggregated cases."

This is a very interesting suggestions and we will explore it further. We will add a sentence mentioning the potential benefit from Bayesian modelling in the conclusion section.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper22/AnonReviewer1
evaluation: A few points that could be considered :
- There is a good chance that the demographics of these big cities would have changed a lot over the span of 5 years(2012 to 2017). Since the delay in crime reporting also largely depends on the individuals themselves, how does this change in the demographics influence the predicting power of the model? Would it be better if we analyzed smaller time frames individually?
- Also the reporting delays (or even the victim not reporting the crime) might depend on how the accused was related to the victim and the location (workplace, outdoor spaces etc). If the accused was for instance, the manager of the victim in a company, the victim might refrain from complaining if the manager threatened to remove them from employment. Is there a possibility to account for such factors?
- An R^2 0.243(NY) or 0.176 probably hints that there needs to be a lot more factors that could be incorporated. There is a chance that these factors might be purely random as well. How could this be handled?

Overall it is great topic and interesting work.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper22/AnonReviewer2
evaluation: This paper focuses on building predictive models for delays in reporting rapes to police, a factor implicated in whether they are reported at all.  Such models could be used by law enforcement to provide targeted support for victims and communities based on their vulnerability.  The issue is of social importance in addressing structural challenges to investigation and prosecution of rape, and the paper is topical and timely.

One thing I didn't know before reading this paper was that *disaggregated* data on rape statistics are publicly available.  This immediately led me to ask questions about the potential identifiability of individual rape victims in public datasets (for example, by attackers conditioning on prior knowledge of particular persons having been raped), since age, gender, race, occurrence date and spatial location are all variables available in the disaggregated dataset.  The paper is not about privacy or security as such, but the question seems more relevant in light of the paper's key finding for the disaggregated dataset that only victim-specific information seems to lead to meaningful improvements in model predictions.  I might therefore have expected the authors to at least briefly highlight identifiability and data security questions, for example in the Conclusion section when talking about potential for training future models on even more expansive (and personal) datasets.

The particular machine learning models deployed seem fairly standard.  The paper addresses decisions being made about spatiotemporal processes, although the models being deployed (e.g. Gaussian process / random forest classification) don't seem to give especially privileged status to space or time.  The authors use different techniques (regression vs classification) to analyze aggregated vs disaggregated data, and hence use different performance metrics, which makes it difficult to address the impact of aggregation on model performance (e.g. bias-variance trade-off).  A Bayesian approach might have been more suitable for this application, since then aggregation simply corresponds to marginalizing over the aggregated variables, and other aspects of the model can be more directly compared between the aggregated and disaggregated cases.

I'm not a social scientist so I'm not entirely sure how this paper would stack up in the social science literature.  However, it seems like an important example of an application of machine learning to promote social justice.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper22/AnonReviewer2
relevance: 4: high
impact: 4: high
novelty: 4: high

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper22/AnonReviewer1
relevance: 4: high
impact: 3: borderline
novelty: 4: high