Original: Early Detection of Sexual Predators with Federated Learning
Revised: Early Detection of Sexual Predators with Federated Learning

Original: Author

Original: AffiliationAddress

Original: Abstract
Revised: Abstract

Original: The rise in screen time and the isolation brought by the different containmentmeasures implemented during the COVID-19 pandemic have led to an alarmingincrease in cases of online grooming.
Revised: The rise in screen time and the isolation brought by the different containment measures implemented during the COVID-19 pandemic have led to an alarming increase in cases of online grooming.

Original: Online grooming is defined as all thestrategies used by predators to lure children into sexual exploitation.
Revised: Online grooming is defined as all the strategies used by predators to lure children into sexual exploitation.

Original: Previousattempts made in industry and academia on the detection of grooming rely onaccessing and monitoring users’ private conversations through the training ofa model centrally or by sending personal conversations to a global server.
Revised: Previous attempts made in industry and academia on the detection of grooming rely on accessing and monitoring users’ private conversations through the training of a model centrally or by sending personal conversations to a global server.

Original: Weintroduce a first, privacy-preserving, cross-device, federated learning frameworkfor the early detection of sexual predators, which aims to ensure a safe onlineenvironment for children while respecting their privacy.
Revised: We introduce a first, privacy-preserving, cross-device, federated learning framework for the early detection of sexual predators, which aims to ensure a safe online environment for children while respecting their privacy.

Original:  Introduction
Revised:  Introduction

Original: The unprecedented rise in screen time and isolation brought about by the COVID-19 pandemic hasleft children more vulnerable than ever to online sexual exploitation.
Revised: The unprecedented rise in screen time and isolation brought about by the COVID-19 pandemic has left children more vulnerable than ever to online sexual exploitation.

Original: In 2021 alone, 85 millionpictures and videos of child sexual abuse have been reported worldwide [12].
Revised: In 2021 alone, 85 million pictures and videos of child sexual abuse have been reported worldwide [13].

Original: In May 2022, to fightagainst these growing numbers, the European Commission proposed a new regulation to compelchat apps to scan private user messages for child abuse and exploitation [12].
Revised: In May 2022, to fight against these growing numbers, the European Commission proposed a new regulation to compel chat apps to scan private user messages for child abuse and exploitation [13].

Original: This new regulationwas strongly condemned by privacy experts, who believed that implementing such mechanisms andbreaking end-to-end encryption of users’ messages could lead to mass surveillance [30].
Revised: This new regulation was strongly condemned by privacy experts, who believed that implementing such mechanisms and breaking end-to-end encryption of users’ messages could lead to mass surveillance [31].

Original: Previous work on the identification of sexual predators has shown that the sexual predators’ discoursecontains specific indicators that can be leveraged for the detection of online grooming [25, 19, 24].
Revised: Previous work on the identification of sexual predators has shown that the sexual predators’ discourse contains specific indicators that can be leveraged for the detection of online grooming [26, 20, 25].

Original: Some researchers focused on finding these linguistic cues by extracting lexical, syntactical, andbehavioral features from chat messages [15, 21].
Revised: Some researchers focused on finding these linguistic cues by extracting lexical, syntactical and behavioral features from chat messages [16, 22].

Original: Others have used deep learning techniques to learnuseful representations from text [32, 23].
Revised: Others have used deep learning techniques to learn useful representations from text [33, 24].

Original: Only  few treated the grooming detection problem as anearly risk detection task [18, 31], i.e. recognizing grooming while it is happening and interventionis possible, as opposed to detection afterwards.
Revised: Only a few treated the grooming detection problem as an early risk detection task [19, 32], i.e. recognizing grooming while it is happening and intervention is possible, as opposed to detection afterward.

Original: Furthermore, none of the proposed solutions wereconcerned with ensuring the privacy of the training examples.
Revised: Furthermore, none of the proposed solutions were concerned with ensuring the privacy of the training examples.

Original: This represents a major limitation forthe applicability of these models in a real-life setting, which is the main focus of this paper.
Revised: This represents a major limitation for the applicability of these models in a real-life setting, which is the main focus of this paper.

Original: We present a novel privacy-preserving decentralized approach to train a context-aware languagemodel [7] for the early detection of sexual predators in ongoing conversations.
Revised: We present a novel privacy-preserving decentralized approach to train a context-aware language model [8] for the early detection of sexual predators in ongoing conversations.

Original: To do this, we leveragefederated learning (FL) [22], an alternative to centralized machine learning (ML) that relies on aglobal server orchestrating the training of different entities without sharing any raw data, enhancedwith differential privacy (DP) [8] to provide formal privacy guarantees.
Revised: To do this, we leverage federated learning (FL) [23], an alternative to centralized machine learning (ML) that relies on a global server orchestrating the training of different entities without sharing any raw data, enhanced with differential privacy (DP) [9] to provide formal privacy guarantees.

Original: Our key contributions are: (1)a practical, cross-device, privacy-preserving FL framework for the early detection of sexual predatorsin ongoing conversations; (2) an end-to-end implementation of our framework with an extensiveevaluation on a real-world dataset.
Revised: Our key contributions are: (1) a practical, cross-device, privacy-preserving FL framework for the early detection of sexual predators in ongoing conversations; (2) an end-to-end implementation of our framework with an extensive evaluation on a real-world dataset.

Revised: The remainder of this paper is organized as follows: in Section 2 we present the existing related work.

Revised: Section 3 introduces the preliminaries of our framework, which is then introduced in Section 4.

Revised: In

Revised: Section 5 we discuss its implementation details and evaluate it on a real-world dataset.

Revised: Finally, we conclude with the ethical considerations surrounding such an application in Section 6 and discuss possible future works in Section 7.

Original:  Related Work
Revised:  Related Work

Revised: In this section, we review the most relevant works to our proposed approach in three main categories.

Revised: First, we look at what has been done in the literature for the detection of sexual predators, then we introduce related work on the early detection of sexual predators before presenting existing work on decentralized text classification.

Original: Detection of sexual predators.
Revised: Detection of sexual predators.

Original: A competition organized at PAN-12 attracted attention to the taskof identifying sexual predators with the creation of a new annotated dataset for the detection ofgrooming in messages [15].
Revised: A competition organized at PAN-12 attracted attention to the task of identifying sexual predators with the creation of a new annotated dataset for the detection of grooming in messages [16].

Original: Two problems were to be solved: (1) identify the predators among allthe users and (2) identify the grooming messages.
Revised: Two problems were to be solved: (1) identify the predators among all the users and (2) identify the grooming messages.

Original: The winners of the first problem [29] used neuralnetworks and SVMs to identify suspicious conversations on a pre-filtered version of the PAN-12dataset, whereas the winners of the second problem [26] treated texts as sequences of symbols andused kernel-based learning methods to classify the grooming messages.
Revised: The winners of the first problem [30] used neural networks and SVMs to identify suspicious conversations on a pre-filtered version of the PANdataset, whereas the winners of the second problem [27] treated texts as sequences of symbols and used kernel-based learning methods to classify the grooming messages.

Original: Recent work mainly adopteddeep learning techniques to solve the task [32, 23].
Revised: Recent work mainly adopted deep learning techniques to solve the task [33, 24].

Original: But all these approaches treated the problem froma forensic perspective rather than for prevention.
Revised: But all these approaches treated the problem from a forensic perspective rather than for prevention.

Original: Early risk detection.
Revised: Early risk detection.

Original: To block harm from occurring, grooming should be detected before a victim islured.
Revised: To block harm from occurring, grooming should be detected before a victim is lured.

Original: Escalante et al.
Revised: Escalante et al.

Original: [10] made the first attempt at the early detection of sexual predators by adaptinga naive Bayes classifier for grooming prediction with partial information.
Revised: [11] made the first attempt at the early detection of sexual predators by adapting a naive Bayes classifier for grooming prediction with partial information.

Original: The authors evaluated theperformance of their model with different percentages of words from the test set in a chunk-by-chunkevaluation framework that was later extended using profile-based representation [11, 18].
Revised: The authors evaluated the performance of their model with different percentages of words from the test set in a chunk-by-chunk evaluation framework that was later extended using profile-based representation [12, 19].

Original: Morerecently, Vogt et al.
Revised: More recently, Vogt et al.

Original: [31] formally defined the task of early detection of sexual predators (eSPD),moving away from existing work to propose a sliding window evaluation, and creating a new datasetthat is better suited for the task.
Revised: [32] formally defined the task of early detection of sexual predators (eSPD), moving away from existing work to propose a sliding window evaluation, and creating a new dataset that is better suited for the task.

Original: We build on top of this work and use their proposed evaluationframework and dataset.
Revised: We build on top of this work and use their proposed evaluation framework and dataset.

Original: Federated learning for text classification.
Revised: Federated learning for text classification.

Original: The approaches above assume training and deploymentof models for grooming detection without concerns for privacy, i.e. while fully disclosing the users’personal messages to a central server for model training.
Revised: The approaches above assume training and deployment of models for grooming detection without concerns for privacy, i.e. while fully disclosing the users’ personal messages to a central server for model training.

Original: FL, a method for training models in adecentralized fashion at the clients’ end, and intermittently aggregating them via a central server,has been proposed as an alternative for natural language processing and text classification tasks (seee.g.
Revised: FL, a method for training models in a decentralized fashion at the clients’ end, and intermittently aggregating them via a central server, has been proposed as an alternative for natural language processing and text classification tasks (see e.g. [14, 15]).

Original: While privacy is preserved to some extent in FL because no raw data is disclosed,information about the clients’ training data may leak from the gradients or model parameters sentto the central server [5, 6].
Revised: While privacy is preserved to some extent in FL because no raw data is disclosed, information about the clients’ training data may leak from the gradients or model parameters sent to the central server [5, 6].

Original: This information leakage can be mitigated by combining FL withanother privacy-enhancing technology such as differential privacy (DP), e.g. by training models withdifferentially private gradient descent (DP-SGD)
Revised: This information leakage can be mitigated by combining FL with another privacy-enhancing technology such as differential privacy (DP), e.g. by training models with differentially private gradient descent (DP-SGD)

Original: Basu et al.
Revised: Basu et al.

Original: [3] have for instance recently applied
Revised: [3] have for instance recently applied

Original: FL and DP-SGD for financial text classification.
Revised: FL and DP-SGD for financial text classification.

Original: To the best of our knowledge, privacy-preservingearly detection of abusive content in a decentralized manner by leveraging both FL and DP-SGD, aswe propose in this paper, has not been investigated in the literature.
Revised: To the best of our knowledge, privacy-preserving early detection of abusive content in a decentralized manner by leveraging both FL and DP-SGD, as we propose in this paper, has not been investigated in the literature.

Original:  Preliminaries
Revised:  Preliminaries

Revised: In this section, we review several key topics upon which our proposed privacy-preserving early detection of sexual predator framework relies.

Revised: In our work, we leverage federated learning and differential privacy to protect the privacy of users, hence we first introduce DP and then provide a brief overview of the DP-SGD algorithm.

Original: Whilst FL protects the privacy of the clients by not requiring any raw data to be disclosed, FL initself does not offer formal privacy guarantees, and the resulting model can leak information aboutthe training data [6].
Revised: Whilst FL protects the privacy of the clients by not requiring any raw data to be disclosed, FL in itself does not offer formal privacy guarantees, and the resulting model can leak information about the training data [6].

Original: To mitigate such information leakage, FL can be combined with DP [8] toprovide plausible deniability regarding an instance being in a dataset, i.e. offering protection againstmembership inference attacks.
Revised: To mitigate such information leakage, FL can be combined with DP [9] to provide plausible deniability regarding an instance being in a dataset, i.e. offering protection against membership inference attacks.

Original: Formally, DP revolves around the idea of a randomized algorithm – such as an algorithm to train
Revised: Formally, DP revolves around the idea of a randomized algorithm – such as an algorithm to train

Original: ML models – producing very similar outputs for adjacent inputs.
Revised: ML models – producing very similar outputs for adjacent inputs.

Original: In the context of this paper, twodatasets d and d ′ are considered adjacent if they differ in one record (one labeled instance).
Revised: In the context of this paper, two datasets d and d ′ are considered adjacent if they differ in one record (one labeled instance).

Original: Arandomized algorithm M : D (cid:55)→ R with domain D and range R is said to be ( ϵ , δ )-differentiallyprivate if for any adjacent datasets d and d ′ and for all subsets of outputs S ⊆ R we have Pr [ M ( d ) ∈
Revised: A randomized algorithm M : D (cid:55)→ R with domain D and range R is said to be ( ϵ , δ )-differentially private if for any adjacent datasets d and d ′ and for all subsets of outputs S ⊆ R we have Pr [ M ( d ) ∈

Original: S ] ≤ e ϵ Pr [ M ( d ′ ) ∈ S ] + δ , where ϵ is the metric of privacy loss (privacy budget) whereas δ isthe probability of data being accidentally leaked.
Revised: S ] ≤ e ϵ Pr [ M ( d ′ ) ∈ S ] + δ , where ϵ is the metric of privacy loss (privacy budget) whereas δ is the probability of data being accidentally leaked.

Original: The smaller these values, the stronger the privacyguarantees.
Revised: The smaller these values, the stronger the privacy guarantees.

Original: An ( ϵ, δ ) -DP randomized algorithm M is commonly created out of an algorithm M ∗ by addingnoise that is proportional to the sensitivity of M ∗ , in which the sensitivity measures the maximumimpact a change in the underlying dataset can have on the output of M ∗ .
Revised: An ( ϵ, δ ) -DP randomized algorithm M is commonly created out of an algorithm M ∗ by adding noise that is proportional to the sensitivity of M ∗ , in which the sensitivity measures the maximum impact a change in the underlying dataset can have on the output of M ∗ .

Original: This technique is used inthe differentially private stochastic gradient descent (DP-SGD) algorithm which aims at controllingthe influence the training data has on the final model by making the minibatch stochastic optimizationprocess differentially private through clipping and adding noise to the gradients [1].
Revised: This technique is used in the differentially private stochastic gradient descent (DP-SGD) algorithm which aims at controlling the influence the training data has on the final model by making the minibatch stochastic optimization process differentially private through clipping and adding noise to the gradients [1].

Original: At the end ofthe training, the overall privacy cost of the mechanism ( ϵ , δ ) can be computed from the accumulated costs across all training iterations.
Revised: At the end of the training, the overall privacy cost of the mechanism ( ϵ , δ ) can be computed from the accumulated costs across all training iterations.

Original: Often, a target ϵ is defined in advance whereas δ should be smallerthan the inverse of the size of the training data.
Revised: Often, a target ϵ is defined in advance whereas δ should be smaller than the inverse of the size of the training data.

Original: We refer to Abadi et al.
Revised: We refer to Abadi et al.

Original: [1] for details.
Revised: [1] for details.

Original:  Methodology
Revised:  Methodology

Original: While protecting children from cybercrime is important, the main challenge is the balance betweensafety and users’ privacy.
Revised: While protecting children from cybercrime is important, the main challenge is the balance between safety and users’ privacy.

Original: We introduce a privacy-preserving framework for the identificationofsexual predators which aims at taking advantage of the growing use of mobile devices by children andteenagers.
Revised: In this section, we introduce the two steps of our privacy-preserving framework for the identification of sexual predators.

Original: Our proposed framework consists of, first, training a classifier on the training set (trainingphase) before evaluating its performance for the early detection task on the test set (inference phase).
Revised: Our proposed framework consists of, first, training a model in a federated manner on the local personal conversation of users with local DP (training phase), and then evaluating its performance for the early detection task on the test set (inference phase).

Original: 4.1 Training Phase: eSPD via Federated Learning
Revised: 4.1 Training Phase: eSPD via Federated Learning

Original: We introduce a cross-device federated architecture for the early detection of online grooming: ourmodel is intended to be deployed on each user’s cellular device, and trained locally on their local datawithout the need for monitoring them as shown in Figure 1 in Appendix A.
Revised: We introduce a cross-device federated architecture for the early detection of online grooming: our model is intended to be deployed on each user’s cellular device and trained locally on their local data without the need for monitoring them.

Original: Our framework addresses multiple task-specific challenges: (1) training with imbalanced data, (2)training with non-independent and identically distributed (non-IID) data and (3) ensuring that users’personal data are protected during training.(1)
Revised: Our framework addresses multiple task-specific challenges: (1) training with imbalanced data, (2) training with non-independent and identically distributed (non-IID) data and (3) ensuring that users’ personal data are protected during training.

Original:  Dealing with imbalanced data.
Revised: (1) Dealing with imbalanced data.

Original: To deal with the problem of imbalanced data – namely very fewpositive instances – that often comes with early risk detection problems, we implement Errecaldeet al.
Revised: To deal with the problem of imbalanced data – namely very few positive instances – that often comes with early risk detection problems, we implement Errecalde et al.

Original: [9]’s oversampling technique.
Revised: [10]’s oversampling technique.

Original: They considered that the minority class is formed not only by thecomplete conversation but also by portions of the full conversation at different time steps.
Revised: They considered that the minority class is formed not only by the complete conversation but also by portions of the full conversation at different time steps.

Original: Therefore,to account for the sequential nature of the eSPD problemand mitigate the imbalanced nature of thedata, we enrich our dataset with chunks of conversations from the minority class, in our case, theconversations with a predator.
Revised: Therefore, to account for the sequential nature of the eSPD problem we enrich our dataset with chunks of conversations from the minority class, in our case, the conversations with a predator.

Original: By giving our system more training examples of the beginning of aconversation with a predator, we are able to gain detection speed.(2) Training with non-IID data.
Revised: By giving our system more training examples of the beginning of a conversation with a predator, we are able to gain detection speed. Furthermore, in a federated setting, having more data gives more weight to a client during aggregation when we use the FedAvg algorithm, which helps alleviate the imbalance problem. (2) Training with non-IID data.

Original: One of the major challenges of FL is dealing with non-IID datasince each client’s local data distribution is not representative of the population [34].
Revised: One of the major challenges of FL is dealing with non-IID data since each client’s local data distribution is not representative of the population [35].

Original: This statisticalchallenge is even more prevalent in the context of online grooming since most users are less likelyto interact with sexual predators.
Revised: This statistical challenge is even more prevalent in the context of online grooming since most users are less likely to interact with sexual predators.

Original: Thus, the detection of online grooming in a federated setting canbe viewed as an extreme case of non-IID data where most users will only have access to one labelfor training.
Revised: Thus, the detection of online grooming in a federated setting can be viewed as an extreme case of non-IID data where most users will only have access to one label for training.

Original: Indeed, only the victims of online grooming will have access to both grooming andnon-grooming conversations.
Revised: Indeed, only the victims of online grooming will have access to both grooming and non-grooming conversations.

Original: We use Zhao et al.
Revised: We use Zhao et al.

Original: [33]’s data-sharing strategy during training in which a small portion of warm-updata is distributed to each device in addition to the initial model.
Revised: [34]’s data-sharing strategy during training in which a small portion of warm-up data is distributed to each device in addition to the initial model.

Original: The warm-up data , which containspublic examples from both classes and is balanced, can be seen as a starting point for training, andhelps alleviate the statistical challenge.(3)
Revised: The warm-up data , which contains public examples from both classes and is balanced, can be seen as a starting point for training, and helps alleviate the statistical challenge.

Revised: In their paper, Zhao et al.

Revised: [34] also suggested sharing a warm-up model with each client: a model trained centrally on the warm-up data.

Revised: We experimented with this strategy but realized that each client did not have enough data and with this strategy the warm-up model may not learn from users’ raw data.

Revised: Instead, we decided to only share a small fraction of the warm-up data with each user during the training phase.

Original:  Protecting users’ privacy.
Revised: (3) Protecting users’ privacy.

Original: Although each client’s local data does not leave their device duringfederated training, it has been shown that it is possible to reconstruct a client’s private data using itsshared updates [16], hence a federated architecture by itself does not guarantee privacy.
Revised: Although each client’s local data does not leave their device during federated training, it has been shown that it is possible to reconstruct a client’s private data using its shared updates [17], hence a federated architecture by itself does not guarantee privacy.

Original: We thereforetrain each client’s model using DP-SGD (see Section 3), to mitigate leakage of personal informationto the server.
Revised: We therefore train each client’s model using DP-SGD (see Section 3), to mitigate leakage of personal information to the server.

Original: By clipping the gradient norm of outliers and randomly adding noise during training,we ensure that our model does not memorize any particular information about a single training datapoint.
Revised: By clipping the gradient norm of outliers and randomly adding noise during training, we ensure that our model does not memorize any particular information about a single training data point.

Revised: Figure 1 illustrates the training phase of our framework.

Revised: A global server selects clients to participate and distributes a model to them; the clients will then further train the model in a privacy-preserving manner on their mobile devices using their own personal data as well as a portion of warm-up data, as we can see in Alice’s cellular device.

Original: 4.2 Inference Phase: Early Detection of Sexual Predators
Revised: 4.2 Inference Phase: Early Detection of Sexual Predators

Original: Our work is an extension of the framework proposed by Vogt et al.
Revised: Our work is an extension of the framework proposed by Vogt et al.

Original: [31] for eSPD, i.e. the earlyrisk detection problem [20] of sequentially classifying a conversation and detecting early signs ofonline grooming as soon as possible.
Revised: [32] for eSPD, i.e. the early risk detection problem [21] of sequentially classifying a conversation and detecting early signs of online grooming as soon as possible.

Original: Vogt et al.
Revised: Vogt et al.

Original: [31]’s approach for the inference phase of an eSPDsystem relies on the use of a sliding window for  sequential classification of a conversation.
Revised: [32]’s approach for the inference phase of an eSPD system relies on the use of a sliding window for the sequential classification of a conversation.

Original: Here, aconversation consists of a sequence of messages t 1 , t 2 , . . .
Revised: Here, a conversation consists of a sequence of messages t 1 , t 2 , . . .

Original: For a window of length l , at step s the classifier labels the sequence t s , t s +1 , . . .
Revised: For a window of length l , at step s the classifier labels the sequence t s , t s +1 , . . .

Original: , t l − 1 , at steps + 1 the classifier labels the sequence t s +1 , t s +2 , . . .
Revised: , t l − 1 , at step s +the classifier labels the sequence t s +1 , t s +2 , . . .

Original: , t l etc.
Revised: , t l etc.

Original: After every window prediction, the system decides whether to raise a warning or not based on the inferred labels of the last 10 windowpredictions.
Revised: After every window prediction, the system decides whether to raise a warning or not based on the inferred labels of the last 10 window predictions.

Original: If a pre-defined threshold – called skepticism level – is reached, a warning is raised andthe whole conversation is classified as a grooming conversation.
Revised: If a pre-defined threshold – called skepticism level – is reached, a warning is raised and the whole conversation is classified as a grooming conversation.

Original: A conversation is only classified as anon-grooming conversation if it is finite and no warning has been raised.
Revised: A conversation is only classified as a non-grooming conversation if it is finite and no warning has been raised.

Original: Indeed, an eSPD systemnever classifies a conversation as non-grooming if there are messages left, or if it is still ongoing.
Revised: Indeed, an eSPD system never classifies a conversation as non-grooming if there are messages left, or if it is still ongoing.

Original:  Figure 2 in Appendix A illustrates the inference phase of our framework.
Revised: In Figure 2, we can see how the different messages received by Alice are analyzed by first being turned into word embeddings and then passed to a classifier given a sliding window for classification. Note that the final prediction is determined based on the previous sequence of predictions and that a warning notification is triggered only when multiple messages are sequentially classified as being grooming messages. We can envision a system where users will be able to report their own suspicious conversations to the messaging platforms, and will receive a notification if a warning is raised (see Figure 3).

Original:  Evaluation5.1 Data
Revised:  Evaluation

Original: [Figure]
Revised: In this section, we show the effectiveness of our proposed approach for the early detection task by performing an empirical evaluation. All our experiments were performed on the PANC dataset . 5.1 Data

Original: The PANC dataset was introduced by Vogt et al.
Revised: The PANC dataset was introduced by Vogt et al.

Original: [31] as a better alternative for the eSPD task.
Revised: [32] as a better alternative for the eSPD task.

Original: It wascreated by merging the “negative” (non-grooming) conversations from the PAN 12 competition [15],sampled from IRC logs and the Omegle forum, 1 and the “positive” (grooming) conversations fromthe ChatCoder2 dataset [21]: 497 complete conversations extracted from the Perverted Justice [-(PJ)website. 
-]
Revised: It was created by merging the “negative” (non-grooming) conversations from the PAN 12 competition [16], sampled from IRC logs and the Omegle forum, 1 and the “positive” (grooming) conversations from the ChatCoder2 dataset [22]: 497 complete conversations extracted from the Perverted Justice [-(PJ)website. 
-](PJ) website 2 .

Original: They filtered the full grooming conversations and split them into segments to make themcomparable to the non-predatory examples and create a corpus better suited for the task of earlydetection.
Revised: They filtered the full grooming conversations and split them into segments to make them comparable to the non-predatory examples and create a corpus better suited for the task of early detection.

Original: Despite its numerous limitations, such as the lack of full negative conversations andsmall differences in formatting between the two classes, we found that the PANC dataset is the mostappropriate available data for our task.
Revised: Despite its numerous limitations, such as the lack of full negative conversations and small differences in formatting between the two classes, we found that the PANC dataset is the most appropriate available data for our task.

Original: See Appendix B for more statistics about the  dataset. 
Revised: The PANC dataset (see Table 1) was split into a training set (60%) and a test set (40%). The training set consists of 1,753 positive segments (representing in total 298 full-length positive conversations and 9.06% of the training examples) and 17,598 negative segments, whereas the test set contains 10.84% examples of grooming. In Figure 3, we present a visualization of a synthetic setup based on our proposed framework using a predatory conversation from the PANC dataset. It can take weeks or even months before a warning notification is triggered when a child is being lured by an abuser.

Revised: Our goal is to minimize the harm by detecting the abuse early and sending a notification to the user.

Revised: It is up to the user to decide whether to continue the conversation or report the predator.

Revised: Note that in our framework, both training and

Revised: [Table]

Revised: inference phases are happening locally and users’ personal conversations are never shared with a third party.

Revised: Moreover, the global aggregated model from the server can further be tuned and personalized based on users’ local data.

Original: 5.2 Evaluation Metrics
Revised: 5.2 Evaluation Metrics

Original: In addition to the established metrics of precision, recall, and F1 score, we use the latency-weighted
Revised: In addition to the established metrics of precision, recall, and F1 score, we use the latency-weighted

Original: F1 score introduced by Sadeque et al.
Revised: F1 score introduced by Sadeque et al.

Original: [27] for the early risk detection task.
Revised: [28] for the early risk detection task.

Original: The F-latency metricmeasures the trade-off between the speed of detection (i.e. how early in a converation grooming isdetected) and the accuracy of the warning by applying a penalty that increases with the warninglatency.
Revised: The F-latency metric measures the trade-off between the speed of detection (i.e. how early in a converation grooming is detected) and the accuracy of the warning by applying a penalty that increases with the warning latency.

Original: A higher F-latency score means a better-performing eSPD system.
Revised: A higher F-latency score means a better-performing eSPD system.

Original: The warning latency isdefined as the number of messages exchanged before a warning is raised [31].
Revised: The warning latency is defined as the number of messages exchanged before a warning is raised [32].

Original: The penalty can becomputed for each warning latency l ≥ 1 as follows:
Revised: The penalty can be computed for each warning latency l ≥ 1 as follows:

Original: [Equation]
Revised: [Equation]

Original: where p defines how quickly the penalty should increase.
Revised: where p defines how quickly the penalty should increase.

Original: As suggested by Sadeque et al.
Revised: As suggested by Sadeque et al.

Original: [27], pshould be set such that the latency penalty is 50% at the median number of messages of a user.
Revised: [28], p should be set such that the latency penalty is 50% at the median number of messages of a user.

Original: The “speed” of an eSPD system over a test set of grooming conversations is defined as speed =1 − median { penalty ( l ) | l ∈ latencies } where “latencies” corresponds to the list of warning latenciesproduced by the system for all grooming conversations for which a warning is raised.
Revised: The “speed” of an eSPD system over a test set of grooming conversations is defined as speed = 1 − median { penalty ( l ) | l ∈ latencies } where “latencies” corresponds to the list of warning latencies produced by the system for all grooming conversations for which a warning is raised.

Original: We can then formally define F-latency as: F-latency = F1 · speed.
Revised: We can then formally define F-latency as: F-latency = F1 · speed.

Original: While F1 is computed across theentire test set of positive and negative messages, penalty and speed are computed for the positiveconversations only.
Revised: While F1 is computed across the entire test set of positive and negative messages, penalty and speed are computed for the positive conversations only.

Original: This is common practice in the literature as the delay needed to detect truepositives is a key component of the early risk detection task [20, 27].
Revised: This is common practice in the literature as the delay needed to detect true positives is a key component of the early risk detection task [21, 28].

Original: 5.3 Experimental Set-Up
Revised: 5.3 Experimental Set-Up

Original: Data manipulation.
Revised: Data manipulation.

Original: As explained in Section 4, we leverage the oversampling technique proposedby Errecalde et al.
Revised: As explained in Section 4, we leverage the oversampling technique proposed by Errecalde et al.

Original: [9] to our training data to improve the speed of our system’s detection.
Revised: [10] to our training data to improve the speed of our system’s detection.

Original: As such,we add four additional segments to each grooming conversation in our training set: the first 10%characters of the full conversation, then 20%, 30%, and 40% of the full conversation.
Revised: As such, we add four additional segments to each grooming conversation in our training set: the first 10% characters of the full conversation, then 20%, 30%, and 40% of the full conversation.

Original: We selectedthe number of augmented data portions with the help of hyperparameter tuning.
Revised: We selected the number of augmented data portions with the help of hyperparameter tuning.

Original: Furthermore, toimplement the data-sharing strategy, we split the augmented PANC training set into three: 10% of thedataset is randomly selected to create the warm-up data, and the rest is split between a training set(81%) and a validation set (9%).
Revised: Furthermore, to implement the data-sharing strategy, we split the augmented PANC training set into three: 10% of the dataset is randomly selected to create the warm-up data, and the rest is split between a training set (81%) and a validation set (9%).

Original: Finally, since neither the test set nor real-life data will be augmented,we remove the additional chunks of data from the validation set.[Table]
Revised: Finally, since neither the test set nor real-life data will be augmented, we remove the additional chunks of data from the validation set.

Original: To ensure that no bias came from the warm-up split, we repeated the process three times and testedour model with every split.
Revised: To ensure that no bias came from the warm-up split, we repeated the process three times and tested our model with every split.

Original: We have also experimented with different sizes of warm-up data (1% and5%) and concluded that a 10% split was better suited for the task.
Revised: We have also experimented with different sizes of warm-up data (1% and 5%) and concluded that a 10% split was better suited for the task.

Original: Federated set-up.
Revised: Federated set-up.

Original: In our cross-device federated framework, we create each client by randomlyselecting one user from the training set.
Revised: In our cross-device federated framework, we create each client by randomly selecting one user from the training set.

Original: In our dataset, each user corresponds to a unique conversation,either predatory or non-predatory.
Revised: In our dataset, each user corresponds to a unique conversation, either predatory or non-predatory.

Original: And as seen in Subsection 5.1, whereas each "positive" userhas multiple segments of data, each "negative" user only has one segment of data.
Revised: And as seen in Subsection 5.1, whereas each "positive" user has multiple segments of data, each "negative" user only has one segment of data.

Original: Therefore, tocompensate for the lack of non-predatory examples, if the user selected is a "negative" user, we thenselect 10 additional "negative" users and combine their data.
Revised: Therefore, to compensate for the lack of non-predatory examples, if the user selected is a "negative" user, we then select 10 additional "negative" users and combine their data.

Original: Furthermore, at initialization, eachclient receives a random, balanced portion of the warm-up data: 10 segments with a “negative” labeland 10 segments with a “positive” one to complement their own data.
Revised: Furthermore, at initialization, each client receives a random, balanced portion of the warm-up data: 10 segments with a “negative” label and 10 segments with a “positive” one to complement their own data.

Original: Choice of the classifier.
Revised: Choice of the classifier.

Original: Although fine-tuning BERT has been shown to give better results for theearly detection task [31], we use the pre-trained feature-based approach with logistic regression (LR)since it is far less computationally expensive and better suited for scaling federated training to a largenumber of clients.
Revised: Although fine-tuning BERT has been shown to give better results for the early detection task [32], we use the pre-trained feature-based approach with logistic regression (LR) since it is far less computationally expensive and better suited for scaling federated training to a large number of clients.

Original: In our framework, each user uses the BERTBASE model to create a context-awarerepresentation of their personal conversation by extracting fixed features from the pre-trained model.
Revised: In our framework, each user uses the BERTBASE model to create a context-aware representation of their personal conversation by extracting fixed features from the pre-trained model.

Original: The [CLS] representation of the last layer is then used as an input for LR with a binary cross entropyloss function.
Revised: The [CLS] representation of the last layer is then used as an input for LR with a binary cross entropy loss function.

Original: For each user’s segment, we, therefore, obtain a 768 length vector.
Revised: For each user’s segment, we, therefore, obtain a 768 length vector.

Original: Implementation.
Revised: Implementation.

Original: We use Flower [4], an FL framework that facilitates large-scale experimentsthrough its simulation tools, to implement our setup and collaboratively train a logistic regressionmodel with 10,000 clients for 100 rounds.
Revised: We use Flower [4], an FL framework that facilitates large-scale experiments through its simulation tools, to implement our setup and collaboratively train a logistic regression model with 10,000 clients for 100 rounds.

Original: At each round of training, we select 10% of the clientsrandomly to participate in the training, and the parameters are aggregated with the FedAvg algorithm[22].
Revised: At each round of training, we select 10% of the clients randomly to participate in the training, and the parameters are aggregated with the FedAvg algorithm [23].

Original: The optimal number of rounds was determined by following the evolution of the validation lossof different models during training.
Revised: The optimal number of rounds was determined by following the evolution of the validation loss of different models during training.

Original: Training with DP-SGD.
Revised: Training with DP-SGD.

Original: Every client selected for the training process will train its data with logisticregression with differentially private stochastic gradient descent.
Revised: Every client selected for the training process will train its data with logistic regression with differentially private stochastic gradient descent.

Original: A random grid search was conductedto test for different hyperparameters: notably, the selected range for the gradient clipping level is(0 .  and we tried (0 . 01 , 0 . 05 , 0 . 001 , 0 . 0001) for the learning rate, (8 , 16 , 32 , 100) for thebatch size, and (5 , 10 , 15 , 20 , 100) for the number of local epochs of training.
Revised: A random grid search was conducted to test for different hyperparameters: notably, the selected range for the gradient clipping level is (0 . 5 , 1 , 2 , 5 , 7) , and we tried (0 . 01 , 0 . 05 , 0 . 001 , 0 . 0001) for the learning rate, (8 , 16 , 32 , 100) for the batch size, and (5 , 10 , 15 , 20 , 100) for the number of local epochs of training.

Original: All the models were evaluated using a 50 -message sliding window and a skepticism level of 5 , i.e.5 of the last 10 predictions had to be positive before a warning was raised.
Revised: All the models were evaluated using a 50 -message sliding window and a skepticism level of 5 , i.e. 5 of the last 10 predictions had to be positive before a warning was raised.

Original: Finally, Appendix Epresents the resources used for training our models.
Revised: Finally, Appendix A presents the resources used for training our models.

Original: Our eSPD implementation will be made publiclyavailable upon acceptance.
Revised: Our eSPD implementation can be found at https://github.com/khaoulachehbouni/fl-espd.

Original: 5.4 Empirical Results
Revised: 5.4 Empirical Results

Original: We investigate three research questions in our experiments:
Revised: We investigate three research questions in our experiments:

Original:  RQ1: How is the utility of the eSPD system affected by the FL framework?
Revised: [Table] RQ1: How is the utility of the eSPD system affected by the FL framework?

Original: To address the first research question, we compare the utility of our cross-device approach with twobaselines: (1) Baseline (warm-up data) :
Revised: To address the first research question, we compare the utility of our cross-device approach with two baselines: (1) Baseline (warm-up data) :

Original: A logistic regression model trained centrally on the warm-updata only, to ensure that our framework is not too biased by the warm-up data distributed to eachclient; and (2) Centralized LR : A logistic regression model trained centrally on the training data andthe warm-
Revised: A logistic regression model trained centrally on the warm-up data only, to ensure that our framework is not too biased by the warm-up data distributed to each client; and (2) Centralized LR : A logistic regression model trained centrally on the training data and the warm-up data.

Revised: Both centralized models used five-fold cross-validation for hyperparameter tuning whereas the best hyperparameters for the federated models have been chosen using a random search.

Original: In Table 1 we can see that the federated frameworks show competitive results for the early detectiontask.
Revised: In Table 2 we can see that the federated frameworks show competitive results for the early detection task.

Original: Indeed, the cross-device FL model has the higher F-latency score, and the loss of utilitythat comes with making our model differentially private is moderate.
Revised: Indeed, the cross-device FL model has the higher F-latency score, and the loss of utility that comes with making our model differentially private is moderate.

Original: We believe that this good performance can be attributed to the fact that cross-device training gives more importance to theminority class than centralized training.
Revised: We believe that this good performance can be attributed to the fact that cross-device training gives more importance to the minority class than centralized training.

Original: Indeed, the FedAvg algorithm takes into consideration theamount of data held by each client to aggregate the models, and in our case, the "positive" users trainwith more examples.
Revised: Indeed, the FedAvg algorithm takes into consideration the amount of data held by each client to aggregate the models, and in our case, the "positive" users train with more examples.

Original: The warm-up data also alleviates the imbalance problem by giving each userenough examples of both class.
Revised: The warm-up data also alleviates the imbalance problem by giving each user enough examples of both class.

Original: In addition to showing slightly better results for the early detectiontask (with a 64% F-latency score), the cross-device framework also has the lowest false positive rate(3%).
Revised: In addition to showing slightly better results for the early detection task (with a 64% F-latency score), the cross-device framework also has the lowest false positive rate (3%).

Original: Furthermore, we see that the speed of the baseline model is very high but it also comes withthe higher false positive rate (FPR):as illustrated in Appendix C, detection speed always comes atthe cost of precision.
Revised: Furthermore, we see that the speed of the baseline model is very high but it also comes with a higher false positive rate (FPR): detection speed always comes at the cost of precision.

Original: The baseline model also has a very low F-latency: our model is therefore notbiased by the data sharing strategy and it is indeed learning from each client’s personal data.
Revised: The baseline model also has a very low F-latency: our model is therefore not biased by the data sharing strategy and it is indeed learning from each client’s personal data.

Original: RQ2: How to reduce the harm of false positives in eSPD?
Revised: RQ2: How to reduce the harm of false positives in eSPD?

Original: In eSPD, the emphasis is often put on the detection of predators since missing one could cause a lotof harm.
Revised: In eSPD, the emphasis is often put on the detection of predators since missing one could cause a lot of harm.

Original: However, we should also consider the cost of falsely accusing someone.
Revised: Indeed, the F-Latency metric depends on both the F1-score and the speed. And while the F1-score takes into consideration both recall and precision, the speed does not penalize for precision: a model that predicts every conversation as being predatory will have a very high speed. Therefore, we propose an approach to adjust our model to consider the cost of falsely accusing someone as predator.

Original: For this purpose,for each of our models, we identify the classification threshold that is needed to achieve a 1% falsepositive rate (FPR) when evaluated on the test set.
Revised: For this purpose, for each of our models, we identify the classification threshold that is needed to achieve a 1% false positive rate (FPR) when evaluated on the test set.

Original: Using this new threshold, we re-evaluate ourmodels.
Revised: Using this new threshold, we re-evaluate our models.

Original: Table 2 shows that varying the threshold comes with a loss in speed, which is to be expectedsince higher prediction scores are now needed to classify a window as a grooming conversation.
Revised: Table 3 shows that varying the threshold comes with a loss in speed, which is to be expected since higher prediction scores are now needed to classify a window as a grooming conversation.

Original: Furthermore, the results for the baseline model are not presented because the smaller FPR attainedfor this model with a 0.99 classification threshold is 9%, showing that it was falsely classifyingnon-predatory conversations as predatory.
Revised: Furthermore, the results for the baseline model are not presented because the smaller FPR attained for this model with a 0.99 classification threshold is 9%, showing that it was falsely classifying non-predatory conversations as predatory.

Original: Finally, we notice a decrease in F-latency for all themodels, a necessary trade-off to achieve better precision.
Revised: Finally, we notice a decrease in F-latency for all the models, a necessary trade-off to achieve better precision.

Original: [Table]
Revised: Figure 4 shows the distribution of the warning latencies after we change the classification thresholds of each model to attain a target low false positive rate (FPR=1%). We can see that a larger number of messages is needed in average to attain better precision. In early risk detection, a trade-off is always necessary between the speed of detection and the precision of a warning.

Original: RQ3: How does differential privacy impact the eSPD system?
Revised: RQ3: How does differential privacy impact the eSPD system?

Original: To evaluate the cost of privacy on eSPD systems, we experiment with adding various amounts of noiseϵ to the training process.
Revised: To evaluate the cost of privacy on eSPD systems, we experiment with adding various amounts of noise ϵ to the training process.

Original: It is not surprising to observe that the less performing model is the one withthe highest privacy constraints: with an ϵ of 0.50, we notice a drop of 8% of the F-latency score forthe most private model as seen in Figure 7 in Appendix D.
Revised: [Table] We observe that the less performing model is the one with the highest privacy constraints: with an ϵ of 0.50, we notice a drop of 8% of the F-latency score for the most private model as seen in Figure 6.

Original: However, we notice that there is no loss inutility when ϵ is greater than 10.
Revised: However, we notice that there is no loss in utility when ϵ is greater than 10.

Original: Furthermore, as we can see in Figure 6 in Appendix D, the precisiongraph has a steeper slope and therefore seems to be more impacted by the differentially-privatetraining.
Revised: Furthermore, as we can see in Figure 5, the precision graph has a steeper slope and therefore seems to be more impacted by the differentially-private training.

Original: Indeed, it has been shown that DP-SGD does not affect the performance of a model equallyand that minority classes may be more affected by the training process [2].
Revised: Indeed, it has been shown that DP-SGD does not affect the performance of a model equally and that minority classes may be more affected by the training process [2].

Original: In our case, making ourmodel more private may result in a decrease in its ability to detect predators adequately.
Revised: In our case, making our model more private may result in a decrease in its ability to detect predators adequately.

Original:  Conclusion and Future Directions
Revised:  Limitations and Ethical Considerations

Original: In this paper, we presented a first-of-its-kind federated learning framework for the early detectionof sexual predators and we showed that the utility of our system is comparable to the utility of amodel trained in a centralized manner while fully protecting users’ personal data rights.

Original: We believethat protecting children from sexual exploitation should not come at the cost of privacy or additionalabuse: Appendix F presents the limitations of our approach.

Original: Finally, it is also essential to consider thepossible biases such a model could have and the high cost of falsely accusing someone as a predatorsince large pre-trained models come with racial and gender biases inherited during training [17].

Original: Addressing these challenges remain as future direction of this work.

Original: Finally, we believe that ourframework can be extended to any early risk detection problem: future work could explore the use ofour framework for the detection of cyberbullying or depression.

Original: Checklist

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope? [Yes] Section 5 details the implementation and evaluation of our privacy-preserving framework. (b) Did you describe the limitations of your work?
Revised: In this section, we explore the limitations of our proposed approach and ethical considerations relating to the implementation of such a tool in a real-life setting.

Original: [Yes] Appendix F present the main

Original: limitations of our approach.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] Section 6and Appendix F discuss the potential negative impacts our model could have.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex-

Original: perimental results (either in the supplemental material or as a URL)?

Original: [No] But theimplementation is open-source and a GitHub repository will be provided upon

Original: accep-

Original: tance.

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] Section 5 explains the details of our implementation.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Appendix E presents the

Original: computational resources used during training and inference.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Section 4 cites the

Original: papers we used to build our framework

Original: (b) Did you mention the license of the assets?

Original: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]

Original: The link to the code will be provided at publication.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [No] How to obtain the data is explained in the paper we build our

Original: work upon.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [No] But Appendix F discuss the limitations of the

Original: dataset

Original: A Visualizations of the Framework

Original: Figure 1 illustrates the training phase of our framework.

Original: A global server selects clients to participateand distributes a model to them; the clients will then further train the model in a privacy-preservingmanner on their mobile devices using their own personal data as well as a portion of warm-up data,as we can see in Alice’s cellular device.

Original: In Figure 2, we can see how the different messages received by Alice are analyzed by first beingturned into word embeddings and then passed to a classifier given a sliding window for classification.

Original: Note that the final prediction is determined based on the previous sequence of predictions and that awarning notification is triggered only when multiple messages are sequentially classified as beinggrooming messages.

Original: In Figure 3, we present a visualization of a synthetic setup based on our proposed framework using apredatory conversation from the PANC dataset.

Original: It can take weeks or even months before a warningnotification is triggered when a child is being lured by an abuser.

Original: Our goal is to minimize the harmby detecting the abuse early and sending a notification to the user.

Original: It is up to the user to decidewhether to continue the conversation or report the predator.

Original: Note that in our framework, both trainingand inference phases are happening locally and users’ personal conversations are never shared witha third-party.

Original: Moreover, the global aggregated model from the server can further be tuned andpersonalized based on users’ local data.

Original: B Data

Original: The PANC dataset (see Table 3) was split into a training set (60%) and a test set (40%).

Original: The trainingset consists of 1,753 positive segments (representing in total 298 full-length positive conversationsand 9.06% of the training examples) and 17,598 negative segments, whereas the test set contains10.84% examples of grooming.

Original: [Table]

Original: C Detection Speed

Original: Figure 4 shows the distribution of the warning latencies during the early detection evaluation ofour different models with the default classification threshold.

Original: Figure 5 shows the distribution of thewarning latencies after we change the classification thresholds of each model to attain a target lowfalse positive rate (FPR=1%).

Original: We can see that a larger number of messages is needed in average toattain better precision.

Original: In early risk detection, a trade-off is always necessary between the speed ofdetection and the precision of a warning.

Original: E Computational ressources

Original: E.1 Training phase

Original: Table 4 presents the resources used to train the different models.

Original: Note that the embeddings extractionwith BERT was done separately and with an NVIDIA Tesla T4 GPU (it takes around 20 minutes).

Original: [Table]

Original: [Table]

Original: h

Original: E.2 Inference phase

Original: The early detection evaluation takes around 2 hours with an NVidia A100 GPU (with 40 GB ofmemory).

Original: F Limitations and Ethical Considerations

Original: Beyond the privacy issues, a main challenge in addressing the sexual predators’ identification taskthrough machine learning comes from the lack of publicly available labeled and realistic datasets.
Revised: Beyond the privacy issues, a main challenge in addressing the sexual predators’ identification task through machine learning comes from the lack of publicly available labeled and realistic datasets.

Original: The different datasets used in the literature  take their grooming examples from the PJ website, whichare examples of conversations between predators and adults posing as children to catch them.
Revised: The different datasets used in the literature all take their grooming examples from the PJ website, which are examples of conversations between predators and adults posing as children to catch them.

Original: Suchchats have been shown to differ from real-life conversations and lack certain aspects of groominglike overt persuasion and sexual extortion [28].
Revised: Such chats have been shown to differ from real-life conversations and lack certain aspects of grooming like overt persuasion and sexual extortion [29].

Original: Indeed, volunteers are often actively trying to getthe offenders to be sexually explicit and to arrange an encounter, which is not the case in real-lifesettings.
Revised: Indeed, volunteers are often actively trying to get the offenders to be sexually explicit and to arrange an encounter, which is not the case in real-life settings.

Original: Furthermore, the non-grooming examples often come from forums and chatrooms wherestrangers can interact or engage in cyber-sex.
Revised: Furthermore, the non-grooming examples often come from forums and chatrooms where strangers can interact or engage in cyber-sex.

Original: Lack of negative examples of trusting and intimaterelationships between family members, friends, or partners is an issue of the current datasets whichare essential components for a realistic eSPD task.
Revised: Lack of negative examples of trusting and intimate relationships between family members, friends, or partners is an issue of the current datasets which are essential components for a realistic eSPD task.

Original: We hope that the federated architecture we propose in this paper, will give access to a larger rangeof training examples.
Revised: We hope that the federated architecture we propose in this paper, will give access to a larger range of training examples.

Original: Indeed, since each user will be given the option to report abusive content, theconversations flagged as alleged grooming will then be added to the pool of training examples, thusalleviating the lack of realistic and available labeled datasets.
Revised: Indeed, since each user will be given the option to report abusive content, the conversations flagged as alleged grooming will then be added to the pool of training examples, thus alleviating the lack of realistic and available labeled datasets.

Original: Such a system will allow the trainingexamples to be updated regularly, and will consider the growing speed at which language, especiallyinternet slang, evolves.
Revised: Such a system will allow the training examples to be updated regularly and will consider the growing speed at which language, especially internet slang, evolves.

Original: However, we can imagine that even with such a framework, the labeling will still be an issue since itwill rely on users self-reporting cases of grooming.
Revised: However, we can imagine that even with such a framework, the labeling will still be an issue since it will rely on users self-reporting cases of grooming.

Original: We could think of a preliminary training phasewith real data of convicted predators before deploying a pre-trained model to evaluate each user’spersonal conversation and send a notification where a warning is raised by the eSPD system.
Revised: We could think of a preliminary training phase with real data of convicted predators before deploying a pre-trained model to evaluate each user’s personal conversation and send a notification where a warning is raised by the eSPD system.

Original: Sucha model will also alleviate the privacy cost since the first training phase will happen on publiclyavailable data.
Revised: Such a model will also alleviate the privacy cost since the first training phase will happen on publicly available data.

Original: In this setting, the user will be able to give feedback on the model’s prediction.
Revised: In this setting, the user will be able to give feedback on the model’s prediction.

Original: Butsuch a set-up is certainly not ideal, since actual victims of online grooming often trust their abuserand may not realize that they are being manipulated.
Revised: But such a set-up is certainly not ideal, since actual victims of online grooming often trust their abuser and may not realize that they are being manipulated.

Original: Notifying a third party, such as a legal guardianor a social worker tasked with monitoring the flagged content, may increase the chances of a case ofgrooming being reported but will undoubtedly infringe on the privacy of the victim.
Revised: Notifying a third party, such as a legal guardian or a social worker tasked with monitoring the flagged content, may increase the chances of a case of grooming being reported but will undoubtedly infringe on the privacy of the victim.

Original: Involving law enforcement could also have disastrous consequences.
Revised: Involving law enforcement could also have disastrous consequences.

Original: As we have mentioned insubsection 5.4, the resulting model could be biased towards certain populations like sex workers,people from the LGBTQI+ community, or people prone to online dating.
Revised: As we have mentioned in subsection 5.4, the resulting model could be biased towards certain populations like sex workers, people from the LGBTQI+ community, or people prone to online dating.

Original: Evaluating and selecting thebest model based on a classification threshold that guarantees a 1% false positive rate can be a first step towards ensuring that the eSPD system does not falsely incriminate.
Revised: Evaluating and selecting the best model based on a classification threshold that guarantees a 1% false positive rate can be a first step towards ensuring that the eSPD system does not falsely incriminate.

Original: Furthermore, pre-trainedlanguage models used to extract a context-aware representation of personal conversations, like BERT,have been shown to reproduce racial and gender biases [17].
Revised: Furthermore, pre-trained language models used to extract a context-aware representation of personal conversations, like BERT, have been shown to reproduce racial and gender biases [18].

Original: Using such models as a basis foridentifying potential suspects to be prosecuted could lead to unanticipated outcomes.
Revised: Using such models as a basis for identifying potential suspects to be prosecuted could lead to unanticipated outcomes.

Original: Such a systemshould therefore never be used directly by law enforcement agencies at the risk of exacerbatingexisting social inequalities and persecuting innocents.
Revised: Such a system should therefore never be used directly by law enforcement agencies at the risk of exacerbating existing social inequalities and persecuting innocents.

Original: Finally, the literature and datasets used for our experiments concern male predators, both heterosexualand homosexual, that do not know their victims.
Revised: Finally, the literature and datasets used for our experiments concern male predators, both heterosexual and homosexual, that do not know their victims.

Original: The lack of data available about female abusers doesnot allow us to assume that our model is applicable to the detection of female predators.
Revised: The lack of data available about female abusers does not allow us to assume that our model is applicable to the detection of female predators.

Revised:  Conclusion and Future Directions

Revised: In the wake of the new European Commission’s regulation [13], social media companies will be expected to take action to ensure that their underage users are safe from sexual exploitation when using their platforms.

Revised: Doing so would entail breaking end-to-end encryption and monitoring users’ content, which can easily lead to human rights infringements as we have seen recently with the case of the teenager charged for abortion in Nebraska after Meta turned over her personal chat messages to the police [7].

Revised: Alternatives to existing privacy-invasive monitoring systems are therefore more pressing since the COVID-19 pandemic has increased the need for children’s safety.

Revised: In this paper, we presented a first-of-its-kind federated learning framework for the early detection of sexual predators and we showed that the utility of our system is comparable to the utility of a model trained in a centralized manner while fully protecting users’ personal data rights.

Revised: We believe that protecting children from sexual exploitation should not come at the cost of privacy or additional abuse.

Revised: Finally, it is also essential to consider the possible biases such a model could have and the high cost of falsely accusing someone as a predator since large pre-trained models come with racial and gender biases inherited during training [18].

Revised: Addressing these challenges remain as future direction of this work.

Revised: Finally, we believe that our framework can be extended to any early risk detection problem: future work could explore the use of our framework for the detection of cyberbullying or depression.

Revised: Acknowledgments

Revised: This project was partially funded by Mila AI for Humanity and Canada CIFAR AI chair.

Revised: This research was enabled in part by support provided by Calcul Québec and Compute Canada.

Revised: We would also like to thank Dr. April Edwards for sharing with us the ChatCoder 2 dataset and Mathias Vogt for taking the time to answer our questions regarding their early detection framework.

Revised: A Appendix: Computational ressources A.1 Training phase

Revised: [Table]

Revised: [Equation]

Revised: 3 h

Revised: [Equation]

Revised: 32 GB 3 h

Revised: Table 4 presents the resources used to train the different models.

Revised: Note that the embedding extraction with BERT was done separately and with an NVIDIA Tesla T4 GPU (it takes around 20 minutes).

Revised: Many of the experiments performed in this work were limited by scalability issues.

Revised: For example, we could not use a GPU while running an experiment with more than 10,000 clients since, in the Flower simulation setting, resources must be shared between all participating devices, which is not the case in a real-life setting where each device brings its own resources.

Revised: A.2 Inference phase

Revised: The early detection evaluation takes around 2 hours with an NVidia A100 GPU (with 40 GB of memory).