A Metric Learning Approach to Misogyny Categorization (2020)

Abstract: The task of automatic misogyny identification and categorization has not received as much attention as other natural language tasks have, even though it is crucial for identifying hate speech in social Internet interactions. In this work, we address this sentence classification task from a representation learning perspective, using both a bidirectional LSTM and BERT optimized with the following metric learning loss functions: contrastive loss, triplet loss, center loss, congenerous cosine loss and additive angular margin loss. We set new state-of-the-art for the task with our fine-tuned BERT, whose sentence embeddings can be compared with a simple cosine distance, and we release all our code as open source for easy reproducibility. Moreover, we find that almost every loss function performs equally well in this setting, matching the regular cross entropy loss.

## introduction whether it is at the word or at the sentence level, learning robust representations allows neural networks to consolidate knowledge that can later be transferred to other tasks and domains. many approaches have dealt with this problem in different ways, for instance with cbow or skip-gram from word2vec @xcite for contextindependent word embeddings, or more recently with bert's @xcite sentence embeddings and contextual word embeddings. in order to learn sentence representations, a neural encoder enc needs to learn a mapping from an initial representation x i to a target vector space. in a metric learning approach, the distances between each pair of sentence embeddings (enc(x i ), enc(x j )) should be low if classes y @xmath0. to achieve this objective, the angle θ ij separating a pair of embeddings (as depicted in figure 1 ) can be used to redefine the model's loss function. in the domain of face recognition, many loss functions @xcite @xcite @xcite have been proposed to learn better face representations, motivated by high intra-class variability due to lighting, position or background. other studies have experimented with these methods in different domains with similar characteristics, like speaker verification @xcite @xcite , and even as an enhancement of bert's sentence representations (reimers and gurevych, 2019) for semantic textual similarity. a recent study @xcite has also focused on comparing these methods on face verification, showing that angular margin losses achieve superior performance. on the other hand, the automatic misogyny identification (ami) evaluation campaign @xcite was proposed to address misogyny on tweets. included tasks were identification (i.e. misogynous or not), categorization over five different misogyny types, and target identification (to an individual or a group). however, no participant has proposed a metric learning model. the best system @xcite uses a bidirectional lstm with word embeddings of size 100 for the identification task, and ensemble methods with feature engineering for category and target classification. they achieve a macro f1 score of 36.1 on the misogyny categorization part of sub-task b, which is the one we address as well. a different architecture @xcite uses a multi-layer character bidirectional lstm for categorization, obtaining a macro f1 score of 14.1. in this paper, we focus on five metric learning losses for the task of misogyny categorization, using the ami @xcite dataset. our hypothesis was that metric learning might reduce the natural intra-class variability within misogyny categories, making representations robust to writing styles, irony, insults, etc. the loss functions we experiment with are contrastive loss @xcite , triplet loss @xcite , center loss @xcite , congenerous cosine loss @xcite and additive angular margin loss @xcite , as well as cross entropy loss. we optimize these loss functions with two different architectures: a bidirectional lstm @xcite and bert @xcite , and we evaluate their performance using a simple k-nearest neighbors (knn) classifier to better measure representation quality. our main contributions consist of new state-ofthe-art performance for the misogyny categorization task, as well as empirical evidence that these methods do not perform better than cross entropy loss on closed-set sentence classification. moreover, our code is released as open source for easy reproducibility. ## loss functions in this section, we present the loss functions chosen for our study, which can be separated into contrastbased and classification-based, according to how they are computed. ## task the term misogyny is defined as hatred towards women. hate speech of this nature is unfortunately common in social internet interactions, and current language models are generally unable to accurately detect and classify it. the ami task and corpus were proposed in the context of the ibereval 2018 @xcite and evalita 2018 @xcite evaluation campaigns, allowing researchers to train models focused specifically on misogyny. the corpus consists of an ensemble of tweets with three different types of annotations: misogyny (binary), misogyny category and target (active or passive). we use the same dataset as in @xcite and we focus exclusively on misogyny categorization, using an additional class for non misogynous tweets. our results are thus compared to the categorization part of sub-task b. an explanation of misogyny categories according to the definitions given in @xcite as the corpus does not provide a development set, one was constructed from the training set following the same class distribution. the final train set is composed of 3200 tweets, and the dev and test sets of 800 and 1000 tweets respectively. class distribution is described in detail in ## results the results are summarized in figure 2 . with a fixed architecture, it is clear that all loss functions perform equally, with the exception of lstm with contrastive and triplet loss. as the lstm encoder is rather shallow (4.4m parameters) in comparison to bert (110m parameters), it is possible that contrast-based losses need bigger models to perform competitively. the fact that almost all losses perform equally well shows that, contrary to what we thought, metric learning models perform no better than cross entropy, in contrast to other findings @xcite on face verification. one possible explanation is that the ami dataset may not contain enough examples or classes for these models to exploit. however, another factor might be responsible for this behavior. one of the key differences of ami with respect to face verification is the closedset nature of the problem. an open-set task is evaluated with unseen classes, while a closed-set task is evaluated with unseen instances of the train as a final note, results in ## conclusion in this work we have addressed the problem of misogyny categorization from a metric learning perspective, comparing the performance of several loss functions. we hypothesized that reducing intra-class variability in this way would be beneficial. however, we have shown that none of the considered losses can outperform the regular cross entropy on the task. our results suggest that metric learning approaches might not be suited to closedset sentence classification tasks. finally, our fine-tuned bert sets new state-ofthe-art performance, with a macro f1 score of 40.5.