eevvgg at S em E val-2023 Task 11: Offensive Language Classification with Rater-based Information (2023)

Abstract: A standard majority-based approach to text classification is challenged with an individualised approach in the Semeval-2023 Task 11. Here, disagreements are treated as a useful source of information that could be utilised in the training pipeline. The team proposal makes use of partially disaggregated data and additional information about annotators provided by the organisers to train a BERT-based model for offensive text classification. The approach extends previous studies examining the impact of using raters’ demographic features on classification performance (Hovy, 2015) or training machine learning models on disaggregated data (Davani et al., 2022). The proposed approach was ranked 11 across all 4 datasets, scoring best for cases with a large pool of annotators (6th place in the MD-Agreement dataset) utilising features based on raters’ annotation behaviour.

## introduction semeval-2023 task on learning with disagreements (le-wi-di) challenges the standard approach in natural language processing (nlp) that there is a single interpretation of language @xcite . at the same time, it follows a recent trend in computational linguistics -a perspectivist approach @xcite . different opinions and views are taken into account here, when studying language phenomena. this change from a majority-based to individualised approach is particularly important in the case of studying subjective phenomena such as abusive/offensive/hateful language. studies show that experts and amateur raters have different strategies for hate speech annotation @xcite . @xcite in turn found that some individuals are more likely to mark african american english dialect as toxic language. the approach proposed by the eevvgg team follows previous studies in this area @xcite . in the case of all four datasets (sub-tasks) a model is trained on partially disaggregated data and additional features created with rater-based information. all four detection tasks fall into a broad definition of offensive language detection, however the hs-brexit dataset @xcite is annotated with hate speech, a task for the convabuse dataset @xcite regards abusiveness detection, the armis dataset @xcite focuses on misogyny and sexism detection, and the annotation task in the md-agreement dataset @xcite involves offensiveness detection. the armis dataset comprises of arabic texts, and the other three contain english data. in regard to the md-agreement dataset, a pool of annotators is larger than in other datasets -the corpus was annotated by over 800 individuals compared with 3 to 8 raters in the case of other datasets. therefore, the task could be regarded as more challenging in this case. the proposed system utilises additional information provided in the datasets (besides text) that are employed to model disagreements between annotators. therefore, information about raters and their annotation behaviour is utilised to train a bertbased model. regarding state-of-the-art performance on offensive language detection, bert-based models systematically achieved best results in the previous semeval editions @xcite . regard-ing additional data, the use of hate speech lexicons was a popular option. pre-trained models are fine-tuned separately on each dataset. specifically, bert encodings from the cls token are fed to a first fully-concatenated network, followed by a dropout layer. then, additional features are concatenated and fed to a second fully-concatenated network. lastly, there is a classification layer for a binary prediction with softmax activation. architecture of a model employed for the task is presented in figure 1 . train and dev data splits were combined and used to train a bert model. individual annotations were retrieved from data files instead of majority voted labels. then, instances of text that raters did not unanimously agree on the label were used in the train data twice -first with the label "1", and second with the label "0". otherwise one instance of text with the agreed label was used in the training set. thus, data fed to a model is partially disaggregated, i.e. if not all annotators agreed on the label, a text sample with both labels "0" and "1" is used for training purposes. in regard to data pre-processing, minor changes were applied to the text. in convabuse "prev_agent", "prev_user", "agent", and "user" tokens were removed, so that only text content was extracted and fed to a bert model. in hs-brexit and md-agreement multiple occurrences of "<@xmath0. ## experimental setup besides text, rater-based features were engineered by making use of the additional information ("other info") provided by the organisers in the data files. thus, in regard to each of the four dataset, different features were fed to the model besides text encodings from bert. hyperparameters set for bert-based models are presented in tab. 1. model architectures are based on similar systems developed for classification tasks @xcite as well as recommendations provided by the authors of the transformers library used for model training @xcite . in the transformers library an additional layer added for the classification purposes comprises 768 nodes, and similar works propose to lower the size for the consecutive layers of a classification model. pre-experimental sessions were conducted on train and dev splits (a train split used for training and a dev split used for evaluation purposes) in order to validate the settings. different hyperparameters were established for the layers of the model in the case of the armis dataset because of the underperformance of the system on the initial settings compared to the other three datasets (other systems approached or reached .80 micro-f 1 ). initial number of nodes were lowered by half, and then gradually added to the first and the second fully-connected layers until the system approach .80 micro-f 1 . in addition, all systems were trained for 2 or 3 epochs and the better performing alternative was used in the final settings. the final bert-based models were developed on a combined train and dev splits. in the hs-brexit dataset additional data comprises annotations for aggressive and offensive language, and raters group identity (target vs. control group). the former is utilised in the proposed system -for each text sample a number of positive labels for aggressive and offensive language annotation is calculated. in addition, specific keywords for the hate speech class are extracted. here, words that appear exclusively in the positive ("1") class with a minimum document occurrence of 3 in a combined train and dev splits are considered as keywords. then, this feature is binarised, i.e. marked as "1" in cases with at least 1 keyword in a text sample, and "0" otherwise. in order to extract those keywords, text was lemmatised with the use of spacy library 3 . in the armis dataset there is no variability in the case of annotators pool and every sample was rated by the same group of individuals. therefore, a different approach for feature engineering was employed. in regard to the armis dataset, keywords for the misogyny and sexism class were extracted as an additional feature to text encodings from bert. in particular, words with a high precision score for the positive class were retrieved. here, a precision score is calculated as a number of occurrences of a word in the positive class divided by the overall number of occurrences of this word in data samples. then, words above a selected threshold are extracted. here, 70% threshold was chosen based on performance in experimental trials. as a result, 1429 of such words were retrieved. similar approach is employed in the case of the convabuse dataset, where such keywords are extracted for the abusive class, as well as a selected set of additionally annotated labels (homophobic, intellectual, racist, sexist, sex harassment, target.generalised, target.individual, target.system, explicit, implicit; other categories were annotated very infrequently). regarding the abusive class 55% threshold and minimum document occurrence of 2 was chosen (55 keywords were extracted here); threshold of 51% was chosen for the additional categories 4 . then, keywords extracted for each text 3 https://spacy.io ; version 3.4.1 4 for example, "moron" and "imbecile" were extracted as instance were transformed into a binary feature separately for each category (marked as "1" if at least 1 keyword was extracted and "0" otherwise). as a result, 11 binary features were obtained. ## results the team is officially ranked 11 across 4 datasets, scoring best for the md-agreement dataset (6th place) in terms of cross entropy. official results from the test sets are presented in in addition to the official metrics, macroaveraged f 1 is reported on the test split as it weights equally both labels and is especially informative about the system performance in the case of imbalanced distribution of categories. results indicate that the proposed system achieves good performance for both classes, particularly in the case of convabuse and md-agreement datasets (macro-f 1 approaching 0.9 and 0.8, respectively). in figure 2 confusion matrices for each of 4 datasets are presented. regarding incorrect classification, all 4 models underperform for the positive category (predict "0" when the true label is "1"). regarding soft evaluation, 75th percentile falls between 0.11 and 0.36 in the convabuse and armis datasets, respectively, measured as the mere difference between the true distribution and predicted probability of two categories. regarding correct classification by the proposed system, examples from the test data split include the following cases of (almost) perfect classification in terms of distribution of labels: keywords for intellectual category, "histeric" and "b*tch" for sexist category, and "faggit" and "homo" for homophobic category ## conclusion although the proposed system is simple in its architecture and employed features, it achieves good performance in terms of both f 1 metrics as well as soft evaluation in the official ranking (see in future experiments some form of pre-training on larger datasets could be utilised to further adapt the system to the domain of offensive/abusive/hate speech. although in pre-experimental sessions conducted with the use of hatebert instead of bertweet, it did not show improvement in the system performance. the proposed approach could be further tested in the prediction of labels for individual raters which could be a step towards personalised hate speech detection and filtering systems.