Paper: Where have you been? A Study of Privacy Risk for Point-of-Interest Recommendation (ICLR-2024)

Reviewer comment (Experiments):
The study only evaluates the privacy risks of POI recommendation models in a controlled setting. They didn't provide some insights into how these models might perform in real-world scenarios, where more complex factors are at play.

Author response:
__W1 Only Evalutes the Privacy Risk in A Controlled Setting But Not Real-World Scenarios__  
Our attack is performed on real-world data sets collected by leading social media companies. Since the real-world POI recommendation models are also trained on a similar data source, we believe that our attack on the current dataset should be able to simulate realistic scenarios where the attack might happen.

Also, as mentioned in *Section 2.2*, our attacks do have real-world effects, majorly in two cases where the adversary should have the abilities mentioned in our paper: 

(1) In a scenario mentioned by service providers [1,2,3], user data is collected and used to train models, which are then provided as an API to third parties for application development. In this context, a malicious third party might gain access to the confidence scores from the model's output

(2) In a second scenario, a situation arises on the service provider side during the retention period of the training data expires. Still, the model owner retains the model, and an adversary (e.g., a malicious insider of location service providers) can aim to use the model to perform our attack aiming to infer information about the data that has been deleted.

As discussed in the *“Limitation and Future Work”, Section 6*, we also plan to adapt our attack method to some public real-world services with more limitations (i.e. Map APIs which might only provide the predicted label) using methods like label-only attacks.

Reviewer comment (Evaluation):
The author did not provide insights on how these risks compare to privacy risks associated with ride-sharing or food-delivery apps.

Author response:
__W2 Comparison to Privacy Risks Associated With Ride-sharing or Food-delivery Apps__  
The prior ride-sharing and food-delivery problems mostly lie in solving the different problems of different privacy stakeholders compared to the POI Recommendation Models. 

Firstly, prior ride-sharing and food-delivery research targets the following problems: 

(1) The privacy of company-owned data. For example, Lyft and Uber can use the spatial distribution information of their instrumented fleets for better placement and routing [4]. 

(2) Privacy attacks targeting passengers or companies' service providers, such as drivers. For instance, attackers could steal continuous trajectories and analyze behaviors. However, our paper focuses on attacking a different data format: the user's POIs in location formats, which can be combined into sparse trip trajectories closely related to the user's daily life.

Secondly, the defense mechanisms are different. The studies of Ride-sharing or Food-delivery primarily focus on the data aggregation process, with most relying on secure system design utilizing methods like cryptographic approaches or differential privacy. Our attacks, however, target the privacy leakage of deep learning models, representing a new attack surface. The defense mechanisms for ride-sharing or food delivery are not suitable to defend against our attack.

Reviewer comment (Theory):
According to the paper, no definitive defense mechanism can protect against all the proposed attacks simultaneously. Can you please explain why this is the case? Additionally, could you provide insights into the challenges that must be addressed to develop more effective defenses against such attacks?

Author response:
__Q1 Explanation for Defense Mechanism Not Working and Insights Into More Effective Defenses__  
Thanks for asking this question. This is a great question. We have listed the results of commonly used defense mechanisms in *Appendix E*. These defense mechanisms(e.g. differential privacy), often result in significant drops in utility, primarily due to the nature of the task. The reason is that the utility of existing POI recommendation models heavily depends on memorizing user-specific trajectory patterns from limited user-level data, which lacks sufficient semantic information as in CV/NLP. However, defense mechanisms like differential privacy can diminish the level of memorization.

We believe that to solve this problem, we need a better-designed scheme that can help with generalizing the POI recommendation model. For example, more information in the dataset, such as POI categories or trip purposes, should be included, and this information should be better utilized. Also, we believe that using a pre-trained model on a public dataset and fine-tuning it on a smaller private dataset might help solve this problem. This approach has been successfully used in other tasks in CV [6,7] and NLP [8,9]. This can be a potential future direction, and we plan to work on this in the future.

Reviewer comment (Novelty):
The motivation of this work is not convincing enough.

Author response:
__Q3/W1  GDPR and Attack Motivation__  
We would like to emphasize that the threat model of the paper is well-motivated, and the privacy study of the work is urgently needed, as required by GDPR. 

First of all, GDPR does not mention requirements for using specific protection mechanism mechanisms like differential privacy in collecting user data like location data. GDPR only requires “the collected content is needed, minimized, and stored safely” in articles *#5 and #25* [3,4]. Since the potential threats in POI recommendation models have not been well studied, companies currently have not adopted adequate protection mechanisms when receiving the data [8,9]. Even if there are protection mechanisms like differential privacy on training data, existing works lacks measurement study to quantify whether those data protection mechanisms are properly applied to the data. Our work could bridge this gap by using our proposed attack suite to perform such kind of measurement.

Moreover, no prior work has studied how POI recommendation models leak sensitive information about users’ data, and our work is the first one to study this problem. This is an important problem since POI recommendation models could unintentionally leak user’s sensitive information.

Reviewer comment (Writing):
There are some typos in this paper. For example, at line 16 in algorithm 4, $f_out$ should be $f_\theta$.

Author response:
__W4  Typos In The Paper__  
Thank you for pointing out the typos. We will fix them in the next version.

Reviewer comment (Experiments):
There have been quite a few studies on protecting spatial/location/trajectory privacy. However, most of them were not reviewed or evaluated by this paper. Thus, it is uncertain whether existing privacy preservation mechanisms could help with the mentioned limitation of POI recommendation.

Author response:
__Q2  Need Defense-Related Literature Survey__  
 We have listed the attack-related works in *section 5* and defense-related works in *appendix F*.

As mentioned in *Section 5*, prior research primarily focused on exploiting side channels such as social relationships and historical trajectories to extract sensitive information during the data aggregation and release processes. However, these studies have not yet concentrated on potential leakages through deep neural networks. Given that more recent LBS relies on deep neural networks, which present a new attack surface for adversaries seeking to access private user information, these earlier attack studies are insufficient. We are the first to investigate privacy leakages in POI recommendation models based on deep neural networks. Our work highlights the need for a privacy-preserving ML-based POI recommendation system in both industry and academia.

As for defense, prior works primarily focused on data aggregation and release processes tailored to the attack mentioned above. Since our paper focuses on evaluating the privacy leakage associated with the model, we only focus on the defense mechanisms during the training process, which is widely ignored in prior works. We also will discuss more defenses, e.g., LDP) in the related work section.

We also appreciate it if you could identify some missing related work, and we will cite and compare them accordingly.

Reviewer comment (Experiments):
Taking private spatial data publishing as an example, based on GDPR, an LBS platform can only collect user’s check-in data that has been well protected (e.g., by differential privacy). Under this practical setting, deriving the platform’s data will not leak the sensitive information of users, which makes the attacker model proposed in this work less meaningful.

Author response:
__Q3/W1  GDPR and Attack Motivation__  
We would like to emphasize that the threat model of the paper is well-motivated, and the privacy study of the work is urgently needed, as required by GDPR. 

First of all, GDPR does not mention requirements for using specific protection mechanism mechanisms like differential privacy in collecting user data like location data. GDPR only requires “the collected content is needed, minimized, and stored safely” in articles *#5 and #25* [3,4]. Since the potential threats in POI recommendation models have not been well studied, companies currently have not adopted adequate protection mechanisms when receiving the data [8,9]. Even if there are protection mechanisms like differential privacy on training data, existing works lacks measurement study to quantify whether those data protection mechanisms are properly applied to the data. Our work could bridge this gap by using our proposed attack suite to perform such kind of measurement.

Moreover, no prior work has studied how POI recommendation models leak sensitive information about users’ data, and our work is the first one to study this problem. This is an important problem since POI recommendation models could unintentionally leak user’s sensitive information.

Reviewer comment (Evaluation):
If the input data has been well protected by existing privacy mechanisms and then trained by a POI recommendation model, is there any sensitive information leakage?

Author response:
__Q4  Effect of Using Differential Privacy On Training Data__  
This is a good question. Adding noises with differential privacy guarantees to the training data before model training can help mitigate privacy leakage. However, it comes with a significant utility drop for POI recommendation models. In our study, we also apply a variant of differential privacy called geo-indistinguishability [5] in *section E*. By only protecting part of the training data, we already observed a significant utility drop, as shown in Figure 18. 

Besides, existing works [6,7] that utilize Local Differential Privacy (LDP) to protect training data are insufficient in protecting POI recommendation models. In particular, [6] has to aggregate all the POIs in large regions to form some new (region-level) POIs to achieve reasonable performance, which alters the definition of the original task. While [7] provides POI recommendations for original POIs using an LDP-protected dataset, they show that adding LDP to the training data results in poor utility with a precision of only 0.08. To this end, we highlight that purely using differential privacy on training data might not be suitable for direct application in our task and require better design.

Reviewer comment (Novelty):
Since privacy-preserving learning has been well studied in recent years, it is sometimes possible to extend existing POI recommendation models with privacy guarantees (e.g., by adding differential privacy noise in the gradients). Does this fact significantly change the main insight?

Author response:
__Q5  Effect of Using DP-SGD During Model Training__  
 We conduct the corresponding experiments by adding differential privacy noise in the gradients (e.g., DP-SGD, Selective-DP)  and demonstrate their effects in section *Appendix E*. From the result, we can see that although using these defense mechanisms can mitigate privacy loss, they also cause significant utility loss. That’s why we highlighted that the current defense mechanism might not be enough, and we have listed stronger defense mechanisms tailored to this problem in the section of future work. One promising direction to improve the defense is to use the pre-trained model on a public dataset and fine-turned on a smaller private dataset, which might help with solving this problem.

Reviewer comment (Experiments):
In Appendix E, the epsilon setting of DP-SGD is a little large. Please provide more justifications.

Author response:
__Q6  Epsilon Selection of DP-SGD__  
For DP-SGD, the top-10 accuracy is dropped to 4.97% when the mechanism satisfies (5, 0.001)-DP, which already makes the task not practical. We do not show the result of an epsilon of 0.5 in the paper. This is because when epsilon is set to 0.5, there is a significant drop in utility; specifically, the top-10 accuracy decreases from 71% (when not protected) to below 1%. which makes the protection less practical. We also show the result of epsilon of 1 and 5 for the selective DP in *Appendix E*. and do not further show smaller epsilons which makes the protection less practical. 

The significant utility drop happens because the POI recommendation requires the model to memorize user-specific patterns from very limited user-level training data. Thus, the training is quite sensitive to the noises introduced by DP-SGD. Adding semantics information to the task or finetuning on pre-trained models might mitigate this problem.

Reviewer comment (Experiments):
Both datasets are a little outdated and relatively smaller-scale than the current LBS platform. However, the major insights are strongly related to the data sparsity. Maybe it would be better to conduct experiments on large-scale datasets.

Author response:
__Q7  Need Larger-scale Dataset__  
Thanks for the suggestion!  We currently use standard datasets and SOTA models for POI recommendations in academia. The FourSquare NYC dataset we used is also used in the latest works used in the literature [10,11]. We believe the dataset size does not affect our key insights, as our attacks are derived primarily from the task's inherent characteristics. For example, POIs frequently visited by a user are typically more unique to that user (e.g., home address). These locations are always more susceptible to our attack since they are well memorized by the models as shown in Figure 4.

To confirm this, we conducted additional experiments using a larger FourSquare dataset from Tokyo, containing about *4x* more check-ins than the NYC dataset. Our LocExtract and LocMIA attacks on models trained on this dataset are still effective, showing that our attack is agnostic to the dataset scale. Due to the time limitation in the rebuttal period, we have not finished the experiment on the trajectory-level attacks but we will add them to our paper when finished.

*LocExtract Result:*
|          | ASR-1 | ASR-3 | ASR-5 |
|----------|-------|-------|-------|
| **4SQ(TKY)** | 40.4% | 66.0% | 72.9% |
| **4SQ(NYC)**| 33.0% | 57.3% | 64.9% |

*LocMIA Result:*
|          | TPR@%10FPR | AUC  | ACC  |
|:--------:|:----------:|:----:|:----:|
| **4SQ(TKY)** |     0.72     | 0.88 | 0.81 |
| **4SQ(NYC)** |     0.74     | 0.80 | 0.87 |

Reviewer comment (Evaluation):
Why does the curve in Figure 5(b) first rise and then drop when the number of POI increases?

Author response:
__Q8  Explanation of Fluctuation in Figure 5(b)__  
We believe that the fluctuation in Figure 5(b) can be attributed to two main reasons.

(1) We hypothesize that when the trajectory is very short, it may be more easily influenced by other factors, such as increased location sharing. Additionally, as mentioned in our paper, many other characteristics can impact the performance of the attack. So they can also contribute to the fluctuations.

(2) When the trajectory is longer, it is more difficult to be memorized by the model. Since our attack considers all location points in the trajectory, it is more likely some locations are less well memorized than others and cause a drop in attack success rate. This also explains why the overall trend is that longer trajectories are less vulnerable to our MIA, and a similar result can also be observed in Figure 17, where we conduct a study from the attack parameter selections.

Reviewer comment (Experiments):
It is mentioned that k is usually 1, 5, and 10 when using top-k to measure accuracy in page 3. However, only 1, 3, and 5 were tested in the experiment as shown in Figure 1. What is the rationale of this experimental setting?

Author response:
__Q9  Different Selection of “topk-accuracy” and ”topk-asr”__  
Thanks for pointing out this notation confusion. The *topk-accuracy* of (1,5,10) on page 3 is in evaluating the performance of the POI recommendation task itself, whereas in the later part of the paper, we also use the same setting (and mostly k=10) to evaluate the utility of the model with/without protection.

However, the result in Figure 1 is actually *topk-ASR* for the data extraction attacks, where a smaller k makes more sense as it represents the reliability of the attack. That’s why we use  k=1/3/5 to evaluate the attack results. We will state this clearly in the next version of the paper.

Reviewer comment (Experiments):
The appropriate baselines are missing. Can the existing data extraction or membership inference attacks be applied to the POI recommendation models, e.g., [1]?

Author response:
__W1  Baselines Missing__  
Thanks for the suggestions! We believe that the existing inference attacks can work, but we have not included them in the paper for the following reasons:

(1) Since our paper primarily focuses on evaluating the privacy risks in POI recommendation models. Thus, we select the SOTA attack – LiRA [2], as it has been shown to outperform other MIAs, including [1].  

(2) Our main contribution is in adapting the existing methods to the POI context. Our proposed mechanism should also work across different attack approaches like [1] as well, as they both utilize similar techniques like shadow model training.

Reviewer comment (Experiments):
The threat models assume that the adversaries are capable of accessing the confidence scores, which makes them impractical. In practice, the model owner only releases the final result to the users. In this case, whether the proposed attacks are still effective is unknown.

Author response:
__W3  Threat Model and Confidence Scores__  
 Thanks for asking. We define the corresponding threat model and realistic cases in *section 2.2 (Adversary Knowledge)*. We highlight the scenarios below:

(1) Instead of focusing on publicly available models like Google Maps, which might only provide the final predicted label, we are targeting the scenarios of service providers [3,4,5], who have mentioned in their service that collected user data and trained models, then providing these trained models as services to 3rd party to build applications. Under this setting, a malicious 3rd party might have the confidence scores from the model output.

(2) Also, we have mentioned another scenario where the retention period of the training data expires. Still, the model owner keeps the model, and an adversary (e.g., a malicious insider of location service providers) can aim to use the model to perform our attack to infer about the deleted data.