Enhancing Crisis-Related Tweet Classification with Entity-Masked Language Modeling and Multi-Task Learning (2022)

Abstract: Social media has become an important information source for crisis management and provides quick access to ongoing developments and critical information. However, classification models suffer from event-related biases and highly imbalanced label distributions which still poses a challenging task. To address these challenges, we propose a combination of entity-masked language modeling and hierarchical multi-label classification as a multi-task learning problem. We evaluate our method on tweets from the TREC-IS dataset and show an absolute performance gain w.r.t. F1-score of up to 10% for actionable information types. Moreover, we found that entity-masking reduces the effect of overfitting to in-domain events and enables improvements in cross-event generalization.

## introduction messages on social media during disaster events have become an important information source in crisis management @xcite . in contrast to traditional sources (e.g., official news), social media posts immediately provide details about developments, first-party observations, and affected people in an ongoing emergency situation @xcite . having access to this information is crucial for developing situational awareness and supporting relief providers, government agencies, and other official institutions @xcite . one key challenge poses the information refinement of high-volume social media streams which requires automatic methods for reliable detection of relevant content @xcite . most recent work has focused on binary, multi-class, and multi-label text classification techniques to classify posts into coarse (e.g., relevant, irrelevant) or fine-grained (e.g., infrastructuredamage, missingpeople) categories composed of flattened or hierarchical structures @xcite @xcite . another challenge in natural language processing (nlp) is the nature of data prevalent in social media and microblogging platforms. for example, most works in the crisis-related domain focus on twitter data @xcite which inherits properties such as short texts (280 characters limitation per tweet), less contextual information, hashtags, and noise (e.g., misspellings, emojis) @xcite . according to @xcite , different types of disasters (e.g., flood, wildfire) can be identified by only a few text-based features. however, eventrelated biases and entities as shown in figure 1 prevent models from generalizing to unseen disaster events and therefore degrade w.r.t. detection performance. to circumvent this problem, approaches such as adversarial training @xcite , domain adaptation @xcite , and hierarchical label embeddings @xcite have been proposed but suffer from mixed event types, assume unlabeled data or require semantic label descriptions. contrary to this work, we aim to enhance the detection of rare actionable information for unseen events by masking out entities, applying adaptive pre-training, and incorporating the hierarchical structure of labels. contributions our main contributions are as follows: (1) we introduce an adaptive pre-training strategy based on entity-masking. (2) we incorporate the hierarchical structure of labels as multi-task learning (mtl) problem. (3) we empirically show that our approach improves generalization to new events and increases detection performance for actionable information types. ## related work crisis tweet classification besides conventional detection approaches such as filtering @xcite or crowdsourcing @xcite , machine learning has received much attention in this area. researchers experimented with several methods such as naive bayes, support vector machines, and decision trees either with termfrequency features @xcite or static embeddings @xcite . more recently, the combination of word2vec @xcite with convolutional and recurrent neural networks achieved remarkable improvement in this field @xcite . due to the success of transformers @xcite and the follow-up language models @xcite , most works have been built upon this and outperformed previous approaches @xcite . adaptive pre-training transfer learning with language models essentially contributes to stateof-the-art results in a variety of nlp tasks @xcite @xcite . typically, such language models follow the three training steps @xcite : (1) pre-training on massive corpora; (2) optional pre-training on task-specific data; hierarchical multi-label classification hierarchical multi-label classification (hmc) covers local and global approaches and the combination of both worlds @xcite . a popular categorization of local methods is the subdivision into local classifier per parent node (lcpn) (dumais and chen, 2000), local classifier per node (lcn) @xcite , and local classifier per level (lcl) @xcite . hybrid approaches integrate the global part as a particular constraint such as hierarchical softmax @xcite or combine multiple local and global prediction heads @xcite . recent work in information type classification introduced label embeddings which utilize the hierarchical structure @xcite . finally, the classification can also be viewed as mtl by combining certain loss functions @xcite . ## trec-is in this work, we mainly focus on the dataset of the shared-task trec-is, which represents a collection of annotated crisis-related tweets @xcite . each tweet belongs to a disaster event and is annotated with high-level information types which are derived from an ontology composed of hierarchical stages. however, information type labels are only shipped as a two-level hierarchy with four upper classes l t and 25 lower classes l b . thus, both hierarchy levels represent a multi-label classification task. following the trec-is track design, we split the dataset into train and test events which corresponds to the trec-is 2020b task. this split poses a challenging setup due to the requirement of cross-event generalization @xcite . transformer encoder manufacturer [num] people killed in explosion at [loc] manufacturer 2 people killed in explosion at houston ner [mask] [cls] t 1 t 5 [mask] t n ... e mask e cls e 1 e 5 e mask e n ... fc mlm fc mlm [loc] [num] (a) entity-masked language modeling e cls pooler pooler pooler pooler l t l b l t l b-1 l b-4 ... lcl lcpn hmcn local l b l t e cls e cls + pooler pooler hmcn global l b l t e cls + + l g + (b) multi-task classification heads ## method as depicted in figure 2 our approach combines the two concepts entity-masked language modeling (e-mlm) and mtl. in the following, we briefly describe our method as a combination of those two. ## conclusion and future work in this work, we identified shortcomings in the field of crisis tweet classification for unseen events. for the trec-is data, we found contrasting effects in terms of pre-training and observed an absolute improvement of up to 3% w.r.t. f1-score for actionable information types by incorporating the hierarchical structure. furthermore, we confirmed the effectiveness of our method based on the sharedtask trec-is. future work includes pre-training on a larger corpus, the mitigation of the trade-off between major and minor classes performances, and to analyse the influence of label semantics.