Multiple Instance Learning for Offensive Language Detection (2022)

Abstract: Automatic offensive language detection has become a crucial issue in recent years. Existing researches on this topic are usually based on a large amount of data annotated at sentence level to train a robust model. However, sentence-level annotations are expensive in practice as the scenario expands, while there exist a large amount of natural labels from historical information on online platforms such as reports and punishments. Notably, these natural labels are usually in bag-level corresponding to the whole documents (articles, user profiles, conversations, etc.). Therefore, we target at proposing an approach capable of utilizing the bag-level labeled data for offensive language detection in this study. For this purpose, we formalize this task into a multiple instance learning (MIL) problem. We break down the design of existing MIL methods and propose a hybrid fusion MIL model with mutual-attention mechanism. In order to verify the validity of the proposed method, we present two new bag-level labeled datasets for offensive language detection: OLID-bags and MINOR. Experimental results based on the proposed datasets demonstrate the effectiveness of the mutual-attention method at both sentence level and bag level.

## introduction detection of offensive content online has attracted widespread attention in recent years. offensive content could not only be human-produced on social media platforms such as twitter and facebook, but also could be system-generated due to the pervasive usage of pre-trained language models @xcite . to tackle the problem in practice, a general solution is to train models capable of identifying messages containing offensive language. then the identified messages are checked and processed by human moderators. there have been a great many studies on dealing with offensive language based on deep learning approaches @xcite . previous researches mainly work on detection task based on sentence-level annotated corpus from a specific resource @xcite , which requires massive manual effort. such approaches are effective but resourceconsuming, especially when transferring to a new platform or language. in social media, there are many existing information sources that could substitute for manual labeling. historical records of online platforms like user feedback (report or dislike) and punishments made by moderators could act as supervision. unfortunately, in many cases those "natural labels" are associated with a larger object (i.e. a user, an article or a dialogue) rather than one certain sentence, which are not suitable for fully supervised learning. however, by regarding each sentence as an instance and the article/dialogue as a bag of instances, we can formalize this scenario into a multiple instance learning task. in this way, we are able to train neural models only using the natural bag-level labels from platforms. as illustrated in figure 1 , the model could not only predict bag-level tags, but also locate offensive sentences to provide more explainable results for moderators. mil is a typical weakly-supervised task where each label is associated with a bag of instances, following the rule that a bag is labeled positive if the bag contains at least one positive instance. in offensive language detection task, an offensive sentence will be regarded as a positive instance. most works of mil concentrate on the original main target of mil, which is to predict test bag labels. in offensive language detection tasks, models are supposed to predict not only the bag label but also the sentence labels in order to locate the offensive contents. as benchmarks are necessary for studying mil in offensive language detection, we first reconstruct an existing supervised corpus-olid @xcite into bag-form. regarding that bags pieced together randomly with independent instances may lack internal relevance, we first cluster the sentences and then sample bags inside each cluster. we also collect a new corpus named multi-instance offensive response (minor) dataset. bags in minor are constructed with tweets and replies, which accord better with the practical application. in order to study mil methods for natural language processing tasks systemically, we break down the design of mil methods into four categories according to where instance-level information is fused into bag level: text fusion, embedding fusion, score fusion and hybrid fusion. we notice that embedding fusion methods perform well in bag-level prediction, while score fusion ones have advantages in instance-level. so we propose a hybrid fusion method with mutual-attention mechanism, which enhances both instance and bag level representation at the same time. experimental results demonstrate that our mutual-att method outperforms other models at both levels. ablation studies further illustrate the effectiveness of each component in mutual-att. to summarize, our contributions are as follows: offensive language detection offensive language detection has always been a concerned topic for researchers. great effort has been made to collect corpus from social media (i.e. twitter) @xcite and establish benchmarks (i.e. olid, trac) @xcite . as offensive language online is a world-wide problem, researchers also constructed many non-english @xcite and multi-language (kumar et al., 2018) datasets. semi-supervised dataset solid @xcite has also been proposed to provide large-scale training data without heavy annotation efforts. offensive language detection is a typical text classification task. classic machine learning classifiers including naive bayes and support vector machine have been widely employed to detect offensive language. besides universal features such as bag-of-words @xcite and n-grams @xcite , @xcite also developed task-specific feature extraction method. neural models like lstm and cnn have been applied in numerous recent studies, while pre-trained language models like bert have achieved sota performances in numbers of challenges @xcite . multiple instance learning multiple instance learning was originally proposed by @xcite for drug activity prediction. as the framework @xcite of mil could be extended into various scenarios, it attracts attention from communities of many areas. computer vision (cv) is the major application field of mil. numerous studies have applied mil methods to cv tasks including image classification @xcite , object tracking @xcite and medical prediction @xcite . several natural language processing (nlp) tasks like document modeling @xcite and sentiment analysis @xcite @xcite also meet the definition of mil. various models have been adopted as the base model in mil tasks. the base model of mil varies from classic machine learning methods @xcite to deep models @xcite over time, and from convolutional networks @xcite to language models @xcite ) over application fields. besides the choice of base model, we find in this study that mil fusion methods are essential to mil frameworks. in section 3, we discuss when and how instance-level information is fused into bag level in detail. ## mil fusion the input of multi-instance offensive language detection task is the text of several instances from a bag, while the supervision is the bag-level label. so the calculation process of a mil method could be represented as a path from instance-level text to bag-level score, which is shown in figure 2 . according to when instance-level information is fused into bag level, we organize mil methods for nlp tasks into four categories: text-level fusion, embedding-level fusion, score-level fusion and hybrid fusion. we also discuss different fusion operations in section 3.2 including pooling methods and attention mechanism. ## method we propose a mutual-attention mechanism composed of instance to bag attention (i2b-att) and bag to instance attention (b2i-att). as figure 4 shows, representation and score of instances are both fused into bag-level via i2b-att, while bag score enhances instance scores by b2i-att. instance to bag attention instance embedding and score are fused via the same attention-i2b-att. following @xcite , we directly calculate the instance weight α i according to the output of instance prediction layer z i . in this way, instance weight is ensured to be consistent with the prediction probability, that is to say, the instance with a higher possibility to be positive has a larger attention weight. what's more, no extra parameters need to be introduced, so the model can remain high-efficient. bag to instance attention in order to avoid disagreement between instance and bag predictions, b2i-att is applied to constrain instance prediction with bag score. when the bag label is offensive, an instance label can be offensive or non-offensive, thus low constraint from bag level is supposed to be added to the instance score. by contraries, an instance label should be constrained to non-offensive if the bag is not offensive. the trainable weight β in equation 5 controls how much instance predictions are supposed to be influenced by the bag prediction. fusions of mutual-attention model having i2b and b2i weight α and β, we then calculate the final prediction p f inal following equation 6, where λ is a hyper-parameter. ## conclusion and discussion in this paper, we propose that mil can utilize the historical information from social media platforms as natural labels for offensive language detection. we present two multi-instance datasets for offensive language detection: olid-bags and minor. olid-bags is reconstructed from olid while bags of minor are composed with real tweets and responses. we systematically categorize mil methods for textual tasks into four classes namely text fusion, embedding fusion, score fusion and hybrid fusion. we observe on olid-bags and minor that embedding fusion has higher bag-level performance, while score fusion methods are good at instancelevel prediction. hybrid fusion methods could integrate the advantages of them, among which our proposed mutual-attention achieves state-of-the-art performance on both datasets at both levels. we also carry out a detailed ablation study to investigate the effectiveness of the proposed i2b-att and b2i-att and how they address the prediction disagreement problem. discussion in section 6.3 also verifies that our semi-supervised strategy is efficient and effective. the proposed datasets, olid-bags and minor, could support future studies on multi-instance offensive language detection. also, the presented mil formalization could expand to other online risky content identification tasks. we hope our work could inspire researchers from social media platforms and be instantiated in real scenarios.