UA ntwerp at S em E val-2021 Task 5: Spans are Spans, stacking a binary word level approach to toxic span detection (2021)

Abstract: This paper describes the system developed by the Antwerp Centre for Digital humanities and literary Criticism [UAntwerp] for toxic span detection. We used a stacked generalisation ensemble of five component models, with two distinct interpretations of the task. Two models attempted to predict binary word toxicity based on ngram sequences, whilst 3 categorical span based models were trained to predict toxic token labels based on complete sequence tokens. The five models’ predictions were ensembled within an LSTM model. As well as describing the system, we perform error analysis to explore model performance in relation to textual features. The system described in this paper scored 0.6755 and ranked 26th.

## introduction semeval 2021 task 5: toxic spans detection was organised by john pavlopoulos and colleagues, and described in detail in their task description paper @xcite . competing teams were asked to develop systems capable of detecting spans of toxic text. predictions were evaluated using a pairwise f1-score of toxic character offset predictions, described in section 5.1. initial analysis of the development data revealed that toxic spans were varied in content and not limited to single words. though most examples contained single toxic words or phrases, others contained longer spans and complete sentences. figure 1 illustrates this phenomena. with this in mind, we sort a strategy that combined longer span based detection with binary word classification. ## background toxic span detection is a development of binary toxicity detection which has garnered recent attention, in the form of shared-tasks and datasets @xcite . features teams were supplied with development data consisting of 7939 text samples in varying lengths up to 1000 characters, and tested on 2000 text samples. target span detection asks systems to detect which specific series of characters are toxic, irrespective of the text's overall toxicity. figure 2 illustrates the target value for semeval 2021 task 5. unlike named entity recognition, systems were not scored on their performance at negative, beginning, middle, or end token detection. this target definition led to a focus on positive optimisation, where false positives were of more importance than true negatives. in section 5.3 on error analysis we compare model scores using a binary word level representation of toxicity, that scores both positive and negative prediction. ## conclusion our initial hypothesis, that combining word based and span based approaches would yield a significant performance boost, did not stand up. we measured a 5% increase in f1-score on development data, but this was not transferred to test data. in future work, we would look to a strategy that incorporated model transferability in component model selection, with the intention of better handling fluctuations in annotation rationale. drawing on recent work @xcite .