B ull S top: A Mobile App for Cyberbullying Prevention (2020)

Abstract: Social media has become the new playground for bullies. Young people are now regularly exposed to a wide range of abuse online. In response to the increasing prevalence of cyberbullying, online social networks have increased efforts to clamp down on online abuse but unfortunately, the nature, complexity and sheer volume of cyberbullying means that many cyberbullying incidents go undetected. BullStop is a mobile app for detecting and preventing cyberbullying and online abuse on social media platforms. It uses deep learning models to identify instances of cyberbullying and can automatically initiate actions such as deleting offensive messages and blocking bullies on behalf of the user. Our system not only achieves impressive prediction results but also demonstrates excellent potential for use in real-world scenarios and is freely available on the Google Play Store.

## introduction cyberbullying is defined as 'wilful and repeated harm inflicted through the use of computers, cell phones, and other electronic devices @xcite . it is estimated that as many as 59% of us teenagers would have experienced some form of cyberbullying by the time they become young adults @xcite . nowhere is cyberbullying more prevalent than on social media, where, as much as 69% of reported incidents of cyberbullying took place @xcite . in response to the increased proliferation of online abuse, social media platforms like twitter, facebook, and instagram have, in recent years, introduced policies and features to combat and mitigant cyberbullying and its effects. these include preventing the creation of multiple accounts using similar details and suspending abusive users. human moderators are also employed by online social networks to review thousands of posts daily. unfortunately, such is the prevalence of cyberbullying, and online harassment that, despite these efforts, it remains a significant online risk for many young people. furthermore, as cyberbullying is highly subjective, what is deemed offensive differs amongst people, human moderators can only apply generalised rules in making a judgement. existing mobile tools to combat cyberbullying mostly either use wordlists or lack the flexibility to cope with the evolving nature of social media. @xcite developed a "sentence checker" mobile app that allows users to check for offensive content in messages before sending. however, as the app is incapable of sending messages or integrating with other messaging applications, users will have to type or copy the message to other messaging applications to send the message. @xcite developed mcdefender, a mobile app that serves as a cyberbullying detection companion to the facebook app. the app detects words typed in the facebook app and analyses each word to determine if the user is engaging in bullying activities. the app's tight integration to the facebook app, however, exposes some limitations. for example, the app cannot detect messages sent if facebook is accessed via a web browser or if an unofficial facebook messaging app is used. a better implementation would be to integrate via the social media platform's api, which is the approach adopted by our system. this work is licensed under a creative commons attribution 4.0 international licence. licence details: http://creativecommons.org/licenses/by/4.0/ . there are two critical challenges in developing viable tools to combat cyberbullying on social media. these are; the evolution of the mainly colloquial language used in social media and the need for the tool to react in a timely fashion to cope with real-time communication. to overcome these challenges, our system uses a microservices-based architecture that allows the introduction of classifiers in a "plug and play" manner and leverages containerisation and cloud-based technologies to exponentially scale to meet the demands of a modern social media platform. in this paper, we present bullstop; a mobile app developed to help young people combat cyberbullying and online abuse. while twitter is currently supported, it is designed to work with multiple social media platforms. ## architecture the app utilises a novel approach comprised of two key strategies. firstly, the detection of cyberbullying and online abuse is modelled on how email applications treat spam. it uses a generalised deep learning model to identify the majority of cyberbullying instances and then improves the base model via online training utilising the user's input as ground truth. in this way, the app becomes a personalised cyberbullying detector for each user with a deep understanding of how the user communicates. secondly, the use of a loosely coupled microservices architecture disassociates the deep learning model from the rest of the system, thus allowing models to be "plug and play". thus, any model can be easily incorporated into the system; all that is required is a model capable of multi-label classification that accepts textual data and provides its output in the required json format. this dramatically improves the system's flexibility, allowing newer and more advanced classifiers to be introduced into the system as required. the system architecture is illustrated in figure 1 . messages and contact information from the online social network are introduced into the system via the synchroniser or webhook. the synchroniser is a mobile component that performs regular data harmonisation with online social networks. it connects to the social media platform via an api and extracts new messages, posts and contacts from the user's account. the webhook provides a similar function, but unlike the synchroniser which retrieves information on a schedule via a pull mechanism, the webhook receives new data via a push from the online social network. the webhook is an optional component only used when the social media platform provides a mechanism to send such notifications. it is an event-triggered feature that allows the social network to send new data to an external interface when monitored events occur, for example, a new friend request or message received. data received from the online social network is placed on the message queue where it is processed by the abuse detection module (adm). the adm is comprised of one or more machine learning models that predict the labels for the messages retrieved from the message queue. the models have been trained to predict the labels, cyberbullying, insult, profanity, sarcasm, threat, exclusion, spa, porn for each message. bert, distilbert, roberta and xlnet are some of the models available in the adm, but other models can be easily added. the labels predicted for a message can be corrected by the user, and these are stored and used to retrain the model. the output from the adm determines actions taken by the marshaller. the marshaller communicates with the social networks via the api and initiates appropriate actions on behalf of the user. 3 use case ui design is a critical component for mobile apps, more so for the young target audience of our system. therefore, from the onset, the app was designed to appeal to this audience, and many of the design decisions taken were in furtherance of this goal. feedback from a cross-section of stakeholders, including young people, parents, educators, law enforcement and child mental health professionals was actively sought and influenced the user interface of the application. for the adult stakeholders, we conducted six focus group sessions to gain insight into their views on cyberbullying and its prevention. individual interviews were then conducted with twenty-five adolescents representing the target audience which was followed by a participatory design phase where six young people worked with us to collaboratively design the mobile application. the functionalities and ui of the mobile app is based on the outcome of these sessions. after installing the app, users are prompted to create a profile which is used to store their settings and preferences. industry-standard cryptographic and encryption technologies are used to create and manage accounts securely. users can then associate their profiles with social network accounts (see figure 2 ) and configure personal settings (see figure 3 ). the app will continuously monitor the associated social media accounts for new messages/posts to analyse. relevant labels are assigned to the message based on the model's prediction (see figure 4 ). each label is associated with a score, and the cumulative score for the message is computed and compared against the user's preferences which determine if the message should be deleted and the sender blocked. users can review received messages (including deleted ones) and update the assigned labels. any such message is used by the app as data for online training. we acknowledged that providing users with the ability to read deleted offensive messages in order to reclassify them may seem counterproductive. the stakeholders were however in favour of this feature, and as a mitigating feature, the app permanently removes the most offensive messages automatically after a configurable period. 4 experiments dataset. the models used in the adm were trained on 62,587 tweets extracted from twitter using query terms designed to return offensive tweets. query terms used included the 15 most frequently used profane words on twitter @xcite as well as hashtags such as #sarcasm, #stayinyourlane, #maga, #blacklivesmatter to capture different forms of online abuse and bullying. our collection strategy was aimed at creating a dataset with high concentration of bullying content. this approach differs from standard practice, which seeks to emulate natural distribution and typically results in datasets with minimal cyberbullying content thus requiring oversampling techniques to improve cyberbullying distribution within the dataset. we used a pool of 17 annotators to label the dataset, and each annotator was provided with 6,000 -10,000 tweets. each tweet was assigned to 3 different annotators and majority agreement required for each label. krippendorff's alpha (rater agreement) was calculated to be 0.67. standard preprocessing steps including removal of punctuation, symbols, non-ascii characters, user mentions, url and lower casing were performed. the number of tweets associated with each class is as shown in ## conclusion we have presented our system for fine-grained detection and prevention of cyberbullying and online abuse on social media. besides its flexible architecture which allows the use of any classifier, it incorporates online training using ground truth provided by the user for retraining. the app is available