A D utch Dataset for Cross-lingual Multilabel Toxicity Detection (2021)

Abstract: Multi-label toxicity detection is highly prominent, with many research groups, companies, and individuals engaging with it through shared tasks and dedicated venues. This paper describes a cross-lingual approach to annotating multi-label text classification on a newly developed Dutch language dataset, using a model trained on English data. We present an ensemble model of one Transformer model and an LSTM using Multilingual embeddings. The combination of multilingual embeddings and the Transformer model improves performance in a cross-lingual setting.

## introduction toxic comment detection is becoming an integral part of online discussion, and most major social media platforms use it. however, that success is not shared equally across languages. low resource languages still lack the accurate pre-trained models that are readily available in more resourced languages, such as english. this is mostly due to a lack of annotated corpora. inconsistent task definitions of task compound the problem. where quality data does exist, it often uses alternative task definitions. this paper aims to overcome that challenge by annotating a new dataset and evaluating it within a cross-lingual experiment. we perform multi-label text classification, using an ensemble approach of transformer and lstm models with multilingual embeddings @xcite @xcite . the system is trained on english data by @xcite annotated dutch text from the amica corpus @xcite . we selected multi-label toxicity over other label definitions based on its adaptability and feedback from annotators. toxicity draws its origins from chemistry, referring to how a substance can damage an organism. from experience in annotator training and feedback, this is a straightforward term to communicate to annotators who relate quickly to the concept of harmful language that degrades a conversation or debate, much like a poison. ## related research the conversation ai group defined multi-label toxicity, and wulczyn et al. @xcite . the term goes beyond its counterparts by adding fine-grained sub-labels. the original motivation of wulczyn et al. was for multi-label toxicity to serve as a compatible annotation model for tasks beyond the original wikipedia dataset. unlike other similar initiatives, their work focused on the risk that communities break down or turn silent, "leading many communities to limit or completely shut down user comments" @xcite . for a detailed overview of multi-label toxicity, look to van aken et al., or gunasekera et al. @xcite . a current challenge within the sub-field of toxicity detection is the definition and operationalisation as a concrete task. though there is research within the area, many projects take up alternative interpretations and definitions. this has led to grey areas between terms like offensive language and profanity, cyberbullying, and online harassment. in practice, many projects are classifying the same data and phenomena under alternative definitions. this problem is explored in greater detail by emmery and colleagues @xcite . cross-lingual classification uses training material in one language and test material in another. in this paper, we use english language training data to improve performance on dutch language test data. this resourceful combination relies on recent advancements in multilingual models and benefits underrepresented languages greatly. data @xcite . amica was a collaborative project between dutch-speaking nlp research groups into cyberbullying. van hee et al. facilitated the detailed annotation of many data sets for a range of bullying labels, using real and simulated conversations between children. ## data we use a newly annotated version of the amica dataset, initially developed by van hee et al., for cyberbullying tasks. in addition, we performed further annotation for multi-label toxicity, following the label guidelines of wulczyn et al.. ## method we performed cross-lingual classification using an ensemble approach of two component models, a fine-tuned multilingual bert-base and an lstm model using multilingual unsupervised and supervised embeddings (muse) @xcite . we also used two baseline models for comparison, an lstm without multilingual embeddings and a support vector machine. ## analysis we performed error analysis to interpret model performance in relation to labels and the language of comments. cross-lingual performance we explored the models' cross-lingual performance by comparing all en nl ensemble 0.6401 0.7587 0.7323 bert 0.7112 0.7213 0.7017 muse 0.4812 0.4512 0.6512 ## summary we have demonstrated that by using multilingual pre-trained language models within an ensemble approach, we can classify multi-label toxicity in an alternate language. furthermore, we have demonstrated that the bert model's underlying training affects target language performance by analysing the performance of baseline, component and ensemble models in cross-lingual features. furthermore,