Early Detection of Sexual Predators in Chats (2021)

Abstract: An important risk that children face today is online grooming, where a so-called sexual predator establishes an emotional connection with a minor online with the objective of sexual abuse. Prior work has sought to automatically identify grooming chats, but only after an incidence has already happened in the context of legal prosecution. In this work, we instead investigate this problem from the point of view of prevention. We define and study the task of early sexual predator detection (eSPD) in chats, where the goal is to analyze a running chat from its beginning and predict grooming attempts as early and as accurately as possible. We survey existing datasets and their limitations regarding eSPD, and create a new dataset called PANC for more realistic evaluations. We present strong baselines built on BERT that also reach state-of-the-art results for conventional SPD. Finally, we consider coping with limited computational resources, as real-life applications require eSPD on mobile devices.

## introduction online grooming denotes the process where a socalled sexual predator establishes an emotional connection with a minor online to systematically solicit and exploit them for sexual purposes @xcite . online grooming is a major concern of public safety that, sadly, is rapidly growing. for instance, in england and wales in the year to mid-2020, police recorded 5,083 offenses of sexual communication with a child @xcite , an average of 14 offenses per day. in germany, there were 2,632 recorded cases in 2020 where a child was sexually abused through internet communication technologies @xcite , an increase of 50 % to the previous year. as such crimes often go unreported or undetected, police-recorded incidents certainly do the problem of detecting whether or not a child is being groomed by a predator is called sexual predator detection (spd). most previous approaches to spd have cast this as the problem of identifying predatory authors in a corpus of segments of chats @xcite . other approaches interpreted it as a binary classification problem over segments of a chat @xcite , or the entire chat @xcite . approaches were evaluated mostly using data from the pan shared task on sexual predator detection @xcite . however, most prior work has viewed spd from the point of view of forensics: they focused on identifying completed grooming chats in preparation for legal prosecution. we believe that it is also important to study approaches that may prevent online grooming -as early as possible, i.e., during an ongoing chat. ideally, the grooming process should be disrupted before it succeeds to protect children from harm. this task is non-trivial as the content of grooming chats changes over time: chats often start with the exchange of personal information and building of trust, a phase in which they are difficult to detect. in a second stage, predators further develop trust with their victims in a cycle of entrapment. they try to desensitize their victims to sexual topics, isolate them from others, and arrange meetings @xcite . even in this second stage, it is difficult to distinguish between grooming and consensual conversations between minors or adults. for this, a model needs to be able to detect discriminative features like a user talking about age difference, checking on the victim's relationship with their parents, isolating them from their support network, reframing sexual actions as appropriate and more (see @xcite , pp. 234ff). an example of arranging a meeting is shown in figure 1 . here, an alert is triggered only late in the grooming process, when an in-person meeting is already explicitly being discussed. ideally, such chats should be detected far sooner. however, the real-world consequences of a triggered espd alert can be considerable and may involve police actions. this means that false alerts should be avoided as much as possible. at the same time, false negatives must be avoided by all means as these could lead to a sexual assault. it is therefore as important as ethically difficult to find the best balance between the earliness of an alert and the certainty that an alert is justified. ## analysis of available datasets due to privacy and legal reasons, grooming chats are extremely difficult to obtain. we introduce the (few) known corpora of this kind and discuss their limitations, motivating the assembly of the panc dataset we discuss in section 3. ## early sexual predator detection in this section, we propose an evaluation setup for espd. we give a formal definition of the task followed by suitable evaluation metrics. finally, we discuss how we use and combine existing spd datasets to create panc for the evaluation of espd. ## baseline approach: two-tier espd we present a straightforward espd approach to demonstrate the validity of our task setup and to establish baselines for future works. it consists of two tiers of classification: (1) a local tier (tier 1) that moves a sliding window over the messages of a chat and classifies them, and (2) a global tier (tier 2) that decides after each window prediction whether to raise a warning or not based on the sequence of recent window predictions. the purpose of this architecture is to balance earliness and accuracy and especially to prevent single suspicious windows from triggering warnings. ## evaluation we evaluate our baseline approach in our espd task setup using the proposed metrics for warning earliness, accuracy, and f latency . we compare three different espd systems: s bert-large , s bert-base , and s mobilebert , which use the respective transformer models as described above as the tier-1 classifier. we use a window size of 50 and a skepticism of 5; an evaluation of the impact of the skepticism parameter can also be found below. we fine-tune each of our bert models on panc and vtpan. as the results of fine-tuning bert models often vary heavily based on the random seed used @xcite , we repeat this process three times. in the evaluation, we always report the mean of the resulting measures together with standard deviation. we fine-tune b ert base and mobile b ert using the tensorflow lite model maker @xcite library and b ert large using flair @xcite . ## discussion we discuss several issues that must be considered before planning to apply an algorithm like the ones presented in this work in practice. ## related work online grooming is a real and pressing problem faced by any chat system open to children. accordingly, social media sites and games often use automated grooming detection systems @xcite . for example, youtube applies nlp to detect predatory messages in video comments and livestream chats followed by human verification @xcite . microsoft uses a similar approach for xbox live and skype chat @xcite and also licenses their software to other service providers free of charge @xcite . their obvious advantage over academic research is the access to much larger datasets. however, these solutions are server-based and cannot be applied for end-to-end encrypted chats. many parents also resort to using parental control apps, some of which send children's chats to external servers for analysis, which is a privacy concern. because of these reasons, there is a need for espd systems even on mobile devices. in academia, espd so far has seen comparably little research despite its high societal importance, probably due to the difficulties of obtaining appropriate datasets. @xcite was the winning team of the first problem of the pan12 competition, which was the identification of the predatory authors of the pan12 segments. they approached the problem by first predicting segments as grooming or not and then distinguishing victim from predator. this two-step method was refined by @xcite who additionally used behavioral features, such as the number of questions asked, achieving an f 0.5 of 0.934 for segment classification on a subset of pan12 test . @xcite studied the same problem and included an analysis of early segment classification, i.e., an attempt to find predators early within a segment. they explored their method also by applying it to 10 full-length pj chats, which could be seen as the first instance of espd we are aware of. ## conclusion we defined the problem of early sexual predator detection (espd) in online chats and proposed an evaluation setup for this task. to this end, we assembled the panc dataset, which, albeit having clear limitations, in our mind is the currently best effort possible with the data available. we also showed that a baseline built on current bert-based language models achieves strong results on this dataset, and beats previous methods in related settings. notably, results are only modestly impacted for models that can run on mobile devices. we discussed open issues in our data and evaluation setup that must be studied carefully in future work before espd systems could go live (and expand on this discussion in appendix d). we hope that making our task setup accessible to the research community will encourage more research into the highly important topic of early sexual predator detection.