AIR - JPMC @ SMM 4 H ’22: Classifying Self-Reported Intimate Partner Violence in Tweets with Multiple BERT -based Models (2022)

Abstract: This paper presents our submission for the SMM4H 2022-Shared Task on the classification of self-reported intimate partner violence on Twitter (in English). The goal of this task was to accurately determine if the contents of a given tweet demonstrated someone reporting their own experience with intimate partner violence. The submitted system is an ensemble of five RoBERTa models each weighted by their respective F1-scores on the validation data-set. This system performed 13% better than the baseline and was the best performing system overall for this shared task.

## introduction intimate partner violence (ipv) refers to any form of physical or emotional abuse or aggression within a romantic relationship. many people use social media, particularly twitter, as a means of selfreporting their experiences with ipv. evidently, being able to parse through social media for these instances of self-reported ipv domestic violence is vital for finding victims most in need of resources and support. while scraping tweets for ipv-related content is relatively simple by keyword search, properly labeling and characterizing whether these tweets refer to self-reported ipv proves to be more challenging. it is already known that using bert models or transformer-based models already produces stateof-the-art results for the field of natural language understanding @xcite . in fact, shared tasks from previous years have already demonstrated successful results incorporating this technology @xcite . this paper elaborates on our submission for the task of classifying selfreported intimate partner violence on twitter (in english). the existing research surrounding the provided data-set also demonstrates the effectiveness of roberta models @xcite final submission for this task is composed of a weighted ensemble of five roberta-large models. ## dataset there were three different data-sets provided: training, validation, and test data-sets. the training and validation data-sets were labeled while the test dataset was not. all data-sets are composed entirely of tweets that are related to the topic of domestic violence. among these tweets, 11% are identified as self-reported ipv @xcite . ## method this submission utilizes an ensembling of multiple trained roberta models which each used their best epoch by means of f1-score to outperform the previous results by almost a tenth of a point. the majority ensemble takes the mode model prediction of all five roberta guesses for a given tweet. the weighted ensemble takes a precise linear combination of all five roberta guesses based on each model's initial performance with the validation data-set. we have two defined classes: nonself-reported (0) and self-reported (1). let us define p e (@xmath0, x. we can represent this explicitly as: the roberta-large models performed significantly better than any other model we considered. results from a majority voting ensemble and a weighted ensemble of five roberta-large models were also calculated. they performed identically. figure 1 shows the corresponding confusion matrix. ## results the performances of the designed ensemble compared to the median codalab results and the best performing model from the original research are shown on ## conclusions this study implements ensembling with transformer-based language models to drastically improve precision and recall for this task. this overall system outperforms previous metrics by nearly 13%. this study also included initial preprocessing into part-of-speech frequency and the significance of narrative voice for self-reporting in tweets. initial results suggests that tweets written in second or third person can help fine-tune against false positives. however, the final submission did not utilize any systems built off of this due to time limitations. thus, these results are omitted from this paper. @xcite have already examined systems for narrative diegesis and point of view analysis. ensembling transformer-based models with models trained off of narrative diegesis and point of view data may further improve results.