Event-Related Bias Removal for Real-time Disaster Events (2020)

Abstract: Social media has become an important tool to share information about crisis events such as natural disasters and mass attacks. Detecting actionable posts that contain useful information requires rapid analysis of huge volumes of data in real-time. This poses a complex problem due to the large amount of posts that do not contain any actionable information. Furthermore, the classification of information in real-time systems requires training on out-of-domain data, as we do not have any data from a new emerging crisis. Prior work focuses on models pre-trained on similar event types. However, those models capture unnecessary event-specific biases, like the location of the event, which affect the generalizability and performance of the classifiers on new unseen data from an emerging new event. In our work, we train an adversarial neural model to remove latent event-specific biases and improve the performance on tweet importance classification.

## introduction effective management of crisis situations like natural disasters (e.g. earthquakes, floods) or attacks (e.g. bombings, shootings) is an extremely sensitive and complex phenomenon that requires efficient coordination of people from multiple disciplines along with proper allocation of time and resources @xcite . given that we live in the era of information and social media, filtering important nuggets of information from real-time data and using them into decision-making constitutes a crucial research direction @xcite . critical information from social media is found only in small amounts. hence it is difficult to ex-* equal contribution. the main reason why common approaches fail is the lack of in-domain data @xcite . most emerging crisis are unexpected and data analysis must be done real-time, within a small time-frame @xcite . even if we might have high quality annotated data from previous similar crisis situations, we will not have data from the emerging event that we want to classify. for example, let us assume an earthquake in seattle happens right now. although we may have annotated data from a previous earthquake in los angeles, most of the parameters would be entirely different (e.g. location names, damages, times, etc) since the cities and populations differ. furthermore, because some of those parameters might indeed play an important role in the classification of a tweet from the specific event (e.g. location, if monroe is the epicenter of the seattle earthquake), a traditional model would learn them as important features. this creates a highly-biased model that does not generalize on future events, since we cannot fine-tune properly on-the-fly. on the other hand, some other features are actually important in the general setting (e.g. severity of the earthquake, casualties etc.). the problem we tackle in this work is how to construct an event-based zeroshot learning model that can learn unbiased representations, instead of relying on a highly-biased set of features from seen data. in this paper we explore a technique that helps a neural model to distinguish and discard informa-tion that is related only to specific events, resulting in a more generalizable model with improved performance on unseen events without any fine-tuning. since the main task is to classify the importance of the information contained in a tweet (criticality), we use an adversarial classifier that intends to learn which specific event the tweet refers to, hence remove the event specific bias through a reversal gradient. our experiments represent a reallife crisis management scenario, where the model is evaluated on a new incoming event through a leaveone-out experimental setup, and show substantial improvement over baseline classification methods. finally, we share our code for reproducibility and ease of use 1 . ## related work recent work on crisis informatics focuses on developing nlp solutions to classify and extract information from twitter streams and other social media data related to an emergency event (e.g. attacks, natural disasters). as discussed by @xcite , there are several problems under the umbrella of crisis informatics, such as determining if a snippet of text is related to a specific event, if it is reliable and trustworthy, the type of information it contains, whether the information is actionable, etc. most previous work focuses on the relevance problem: given a set of tweets or other source of information and a specific event, classify which data refer to that event. @xcite uses a cnn model to classify tweets related to flood events, while @xcite uses a few-shot learning model based on a cnn. nguyen et al. (2016) also uses a cnn model to classify related tweets and the type of information contained (e.g. infrastructure damage, affected individuals etc) from the nepal 2015 earthquake. @xcite introduces a real-time system for the japan 2011 earthquake that classifies the relatedness of the posts and extracts surface information like named entities. other approaches include bilstm models for tweet classification @xcite , event detection based on twitter streams @xcite , adversarial data augmentation for image classification @xcite and domain-adaptation across different events using an adversarial network. it is particularly important to first responders the identification of actionable information from a stream of messages as the one provided by twitter. processing information without the context of a crisis event is a bottleneck for big data crisis analytics, as discussed by @xcite . the lack of context makes the classification of messages very difficult, since the models are prone to eventspecific biases. due to the fact that we deal with real-time data, a domain-adaptation approach cannot use fine-tuning in a zero-shot scenario, which results in highly-biased models. most recent work on bias removal @xcite focuses on using adversarial learning to remove demographic bias from representations. examples include adversarial generative networks that create fair representations @xcite , metrics to quantify unintended biases @xcite and applications that show substantial improvements on traditional nlp tasks like nli @xcite , coreference resolution @xcite ) and text classification @xcite by using unbiased representations. our approach is inspired by the work of @xcite on bias removal through an adversarial attack. the authors use an adversarial setting to remove demographic information from text and construct cleaner representations. in our case, the adversarial classifier attempts to predict the event to which the tweet belongs. another difference with our work is the imbalanced data used for training the classifier of the main task. other related work includes domain adaptation based on a gradientreversal layer @xcite ), text classification based on adversarial multi-task learning @xcite , and multi-adversarial domain adaptation across multi-modal data @xcite . ## approach in this work we used data from the trec 2018 incident streams challenge 2 , which contains labels on criticality and information types @xcite . they define criticality as a score to identify posts that need to be shown to an officer immediately as an alert. the raw data and information about the specific event each tweet belongs to is extracted from the crisis nlp @xcite dataset, which contains tweets in english from disaster events that occurred during 2012-2018. the crisis events in our dataset can be split into five main groups: earthquakes, floods, typhoons, wildfires and attacks. in figure 1 , we show that the data mainly consists of multiple earthquake, flood, and typhoon events, only two wildfire events, and five diverse attacks originated by humans. in our experiments we used a labeled subset of the data formed by 18,283 tweets which are labeled into four categories according to their level of importance for the authorities: low, medium, 2 http://dcs.gla.ac.uk/ richardm/trec is/2020/oldindex.html high, and critical. the distribution of the labels is highly skewed towards the low and medium labels as shown in figure 2a . these types of tweets do not provide important information for decision-making during a disaster event. since we are aiming to sieve the actionable tweets, we grouped together the low and medium labels as non-critical, and the high and critical as critical. the new distribution of the data after relabeling is shown in figure 2b . as we see on the examples shown in ## experiments for our experiments we used two of the main popular word embeddings to represent the tokens of the tweets in the target dataset: glove @xcite embeddings, and bert @xcite embeddings. we used the 100-dimensional glove embeddings pre-trained on wikipedia and gigaword, which were made publicly available by the authors foot_0 . for extracting bert embeddings we used the python package bert-embeddings foot_1 as we built the networks for our experiments in pytorch. this package offers a pre-trained 768-dimensional hidden state transformer model with 12-layers and 12-headed attention. in our experiments, the bert model was frozen with no fine-tuning during training. throughout all of our experiments the tweet encoder h is an lstm with two layers. each of the lstms have a hidden dimension of 100, which results in a tweet embedding of size 200. both classifiers c r and c e are linear layers with output size 2 and the number of events per experiment, respectively. during our initial experimentation, we set the gradient-reversal layer scaling value lambda to different values within the range [0.1 -10]. the most stable result throughout the whole experiments was obtained with λ = 1. the models were trained using the adam optimizer (kingma and ba, 2014), with an initial learning rate 0.01, batch size 16 and trained for 40 epochs. we employed dynamic batching by padding each batch to the sequence length of the longest sample in the batch. to test the performance of the model at every epoch we calculated the micro f1 on the critical class from c r and considered as the best model the one which showed the highest critical-f1 score, since for disasters it is important to recall as many critical tweets with the highest possible precision. ## qualitative analysis we took a deeper look into our experimental results by comparing which patterns are learnt by the adversarial model but not the baseline. for this analysis, we focused on flood and earthquake event types, as they show the greatest difference in non-critical rt pal passengers to/from manila who are unable to take their flights due to floods may rebook their tickets with rebooking c ... f1 score between the baseline and the adversarial model. ## future work our experiments show that mixing data from events whose semantics are similar, like the violent mass attacks and the synthetically generated set of floods and typhoons, confuses the adversarial model. as a result, it does not show any improvement over the baseline. moreover, in some cases models trained with glove achieved better performance compared to those trained with bert. for this reason, it seems appropriate to fine-tune transformer-based language models so we could take advantage of the large amount of unlabeled data provided by the crisis nlp dataset that was not used in this work. given that our ultimate goal is to detect and use actionable information during crisis events to inform life-saving actions, an essential part of future research is to design interpretable models. an interesting work proposes a new approach to interpretable classification named deep weighted averaging classifiers (dwac) @xcite , which gives an explanation of the prediction in terms of the weighted sum of training instances. dwac could replace the importance classifier c r in our proposed adversarial model. an advantage of using dwac is that it would deliver the most relevant tweets from the training data which contributed to the detection of a critical tweet. finally, since we deal with a real-time information stream it seems appropriate to evaluate this model in an online learning scenario @xcite . ## conclusion in this work, we compared an adversarialy trained model against a baseline classifier and a multitask learning model. the main task for all the models was to predict if a tweet is critical or non-critical over four types of disaster events: earthquakes, floods, typhoons, and mass attacks in public spaces. we presented a thorough analysis on how a simple classification model trained on crisis event data can be improved through adversarial training. our results showed how the addition of an adversarial network removes the bias from specific events, allowing the network to put more attention in disaster related information rather than specificities of a particular event. in most of our experiments the adversarially trained model obtained the highest f1 score. our experimental results demonstrate the relevance of using micro-f1 scores for evaluating the detection of critical posts from an information stream such as twitter. the impact of false negatives while detecting critical tweets is larger than the false positives, since we would be missing decisive information from the data stream. hence, micro-f1 score is a more informative metric to consider instead of accuracy, or even the overall f1 score since event crisis detection usually suffers from highly skewed data towards the irrelevant samples of the dataset.