Deep Learning Approaches to Detecting Safeguarding Concerns in Schoolchildren’s Online Conversations (2023)

Abstract: For school teachers and Designated Safeguarding Leads (DSLs), computers and other school-owned communication devices are both indispensable and deeply worrisome. For their education, children require access to the Internet, as well as a standard institutional ICT infrastructure, including e-mail and other forms of online communication technology. Given the sheer volume of data being generated and shared on a daily basis within schools, most teachers and DSLs can no longer monitor the safety and wellbeing of their students without the use of specialist safeguarding software. In this paper, we experiment with the use of state-of-the-art neural network models on the modelling of a dataset of almost 9,000 anonymised child-generated chat messages on the Microsoft Teams platform. The data was manually classified into eight fine-grained classes of safeguarding concerns (or false alarms) that a monitoring program would be interested in, and these were further split into two binary classes: true positives (real safeguarding concerns) and false positives (false alarms). For the fine grained classification, our models achieved a macro F1 score of 73.56, while for the binary classification, we achieved a macro F1 score of 87.32. This first experiment into the use of Deep Learning for detecting safeguarding concerns represents an important step towards achieving high-accuracy and reliable monitoring information for busy teachers and safeguarding leads.

## introduction as our lives become ever more digital, traditionally "offline" activities are steadily moving online, and child safeguarding is no exception. in simpler times, it might have been enough for a schoolteacher to walk up and down a classroom to warning: this paper contains offensive examples. *the two authors contributed equally to this work. so necessary is this technology that the uk's statutory guidance for schools and colleges on safeguarding children, keeping children safe in education (kcsie) foot_0 , heavily emphasises the dangers posed by the internet in schools and outlines the obligations of staff to ensure that appropriate web filtering and monitoring systems are in place. as a result, such systems are commonplace and are used in schools and colleges across the uk as well as abroad. kcsie points to a range of online risks to which schools must be vigilant, ranging from harmful web content (e.g. pornography, fake news, extremism) to problematic forms of contact (e.g. online grooming, child exploitation), bad behaviour (e.g. cyberbullying, sharing of explicit images), and financial traps (e.g. online gambling, inappropriate advertising, phishing). given that no digital monitoring system can be perfect, and given the seriousness of child safety, human discernment is still required even for the most sophisticated risk-detecting algorithms. the output of a school's online monitoring system is typically reviewed by a designated safeguarding lead (dsl) or other trusted member of staff before incidents can be triaged and acted upon. as such, it is a priority that such systems capture as many true positive cases as possible while minimising the number of false positives (i.e. noise). for a sensitively tuned safeguarding system that's geared more towards recall than precision, false positives are unavoidable, but they also represent a burden on the dsl in that they require time and energy to review and discard before real safeguarding concerns can be acted upon. while much progress is being made in online safeguarding technology, most products are still bedevilled by the same nlp challenges faced in every other sector that utilises computational linguistics: word-sense disambiguation, parsing, coreference resolution, and sentiment analysis, just to name a few. meanwhile, we have witnessed huge strides in nlp applications with the assistance of neural networks and other advanced machine learning techniques, the likes of which are only very recently becoming visible in the educational technology and child safeguarding sectors. in this paper, we describe some initial experiments into applying deep learning (dl) techniques to the problem of online safeguarding for schoolchildren. we carry out these experiments in the hope of developing more useful and accurate safeguarding technology that will save schools time and effort and ultimately help to protect children better. in this particular case, we focus on messages sent between children on school-owned devices, specifically on the chat platform of microsoft teams, as captured by a keylogging cloud-based safeguarding tool, senso.cloud. a safeguarding concern in such chat messages might be anything from bullying and discriminatory language to disclosures of self-harm and other indications of mental health risks. the remainder of the paper is structured as follows. in section 2, we explore some of the related work that has already been carried out, as well as the gap that we aim to address with our ongoing work. section 3 describes the process of data collection and data annotation, followed by section 4, which explains the use of machine learning models in our experiments. section 5 reports on the results of our experiments, and in section 6, we conclude the paper with a brief discussion and some comments on future work. ## related work while there is not, to our knowledge, a safeguarding study that is directly comparable with this one, we discuss in this section some examples of ma-chine learning and deep learning in nlp generally, as well as the use of nlp for various safeguarding applications. ## data in this section we outline our data collection and annotation as well as ethical considerations. ## methodology our methodology mainly consists of two steps: data preprocessing and machine learning, which we describe in the following subsections. ## results we show our results in two levels: binary classification in section 5.1 and fine-grained classification in section 5.2. for each level, we experiment with the machine learning models described in section 4 to see how they perform. all the models were trained on the training set and then evaluated by predicting the labels for the held-out test set. as the label distribution is highly imbalanced, we evaluate and compare the performance of the different models using macro-averaged f1 score. we further report per-class precision (p), recall (r), and f1 score (f1) for the binary classification. we also experimented with several resampling methods to balance the classes, such as upsampling and down-sampling. however, we did not see a significant improvement in the results. therefore, we continued the experiments with the original training set distribution. ## conclusions we have presented the first study using deep learning to detect generalised safeguarding concerns in schoolchildren's online conversations. we have developed and employed a new and highly relevant dataset consisting of more than 8,850 instances annotated on binary labels as well as fine-grained labels. we employed ten machine learning models, including state-of-the-art transformer models, on the two tasks. we showed that deep learning architectures provided the best results, and among them, the roberta transformer model provided the best result. with this study, we show that machine learning and, particularly, deep-learning-based models can be employed to detect safeguarding concerns in schoolchildren's online conversations. as for limitations, we acknowledge that the dataset is imperfect on a few fronts. for one, it is limited to the english language, and it is captured from just one app, which is microsoft teams chat. as a result of the data collection method via the senso.cloud software, which nonetheless gains us access to a high volume of primary data generated by our target demographic, the data we receive is pre-filtered. that is to say, we only have access to messages that have been captured according to senso.cloud's a priori safeguarding keyword libraries (an important limitation for personal data protection purposes), and as such we cannot comment on recall. it also means that there is an imbalance of data and this imbalance is reflected in the distribution across the classes. the classes themselves emerged in response to the nature of the data, and as such, they are fitted to our specific software and set of keywords. finally, we acknowledge that the dataset is necessarily opaque, for sensitivity and proprietary reasons, as are the models developed during this industry research. at this very early stage in the work, we are not yet able to make these resources public or provide the level of detail that one would find with open-access resources. in future endeavours, we hope to find a safe and satisfactory way of doing so. this initial study opens many exciting avenues in detecting safeguarding concerns in online conversation. in this research, we focused on english, and given that the dataset is anonymised, we cannot safely attribute each instance to a specific variety of english (e.g. british english, american english). however, the machine learning models that we explored are language-independent. in the future, we hope to evaluate these machine learning approaches in multilingual conversations. while the transformer models provided the best results, these models are large in size and computationally expensive. therefore, it can be difficult to use them in real time. recent work has shown that knowledge distillation can transfer knowledge from large models to computationally light models such as svcs. in future work, we hope to build more practical models to detect safeguarding concerns in online conversations in real time. in terms of direct, practical applications, the present research demonstrates the usefulness of pretrained deep learning architectures in reliably identifying a concerning online message from a child, even without the wider context of the conversation. for teachers and dsls, this translates to an intelligent system that can support them in processing the safeguarding alerts they receive daily via their school's safeguarding software. with more data and experiments, this vein of research promises to produce real-world benefits for those faced with high volumes of student safeguarding data in their day-to-day work.