Are We Ready for this Disaster? Towards Location Mention Recognition from Crisis Tweets (2020)

Abstract: The widespread usage of Twitter during emergencies has provided a new opportunity and timely resource to crisis responders for various disaster management tasks. Geolocation information of pertinent tweets is crucial for gaining situational awareness and delivering aid. However, the majority of tweets do not come with geoinformation. In this work, we focus on the task of location mention recognition from crisis-related tweets. Specifically, we investigate the influence of different types of labeled training data on the performance of a BERT-based classification model. We explore several training settings such as combing in- and out-domain data from news articles and general-purpose and crisis-related tweets. Furthermore, we investigate the effect of geospatial proximity while training on near or far-away events from the target event. Using five different datasets, our extensive experiments provide answers to several critical research questions that are useful for the research community to foster research in this important direction. For example, results show that, for training a location mention recognition model, Twitter-based data is preferred over general-purpose data; and crisis-related data is preferred over general-purpose Twitter data. Furthermore, training on data from geographically-nearby disaster events to the target event boosts the performance compared to training on distant events.

## introduction twitter has shown to be an effective medium for gaining situational awareness and performing urgent needs assessment of the affected population during sudden onset disasters @xcite @xcite . furthermore, the platform often breaks events and thus considered a low-latency source for timely access to information when other traditional sources are not available. despite these advantages, one major issue that hinders the usability of twitter data is the lack of geolocation information. only 1-3% of tweets has gps-coordinates @xcite . response authorities and humanitarian organizations heavily rely on geolocation information for both situational awareness and response tasks. while extensive research has been conducted on processing tweets for humanitarian aid, limited focus has been given to infer and extract geolocation information from them. in this work, we focus on extracting toponyms, i.e., place or location names from tweets. we refer to this as a location mention recognition (lmr) task. two main factors that influence the robustness of a lmr system are: (i) the dataset used to train the classifier, and (ii) the learning model. in this work, we explore how the choice of a training dataset influences the performance of a lmr system in the domain of humanitarian crises where the cost and time of acquiring training data should be minimized. for this purpose, one well-established approach is to use a standard named entity recognition (ner) system trained on a general-purpose ner dataset such as @xcite . standard ner datasets provide annotated entities such as location, organization, and person from news articles or other formal web documents. however, the general-purpose ner system may not effectively extract toponyms from twitter messages due to the fact that tweets often contain informal language, misspellings, grammar mistakes, shortened words, and slangs @xcite . moreover, entities mentioned in tweets may have inconsistent capitalization, which is one of the main features standard ner systems rely on @xcite . an alternative choice is to train a system using twitter-specific ner datasets. moreover, since the focus of this work is to identify and extract toponyms from crisis-related tweets, one obvious choice is to train a system specifically on location entities and drop other entity types (i.e., org and per). furthermore, we seek to determine the performance difference between a system trained on a generalpurpose twitter data versus a system trained on disaster-specific twitter data. we also examine the difference in effectiveness when using labeled data from past disasters compared with labeled data from the current (target) disaster. as different types of disasters, such as floods and earthquakes occur in different parts of the world, we investigate whether combining labeled data from different types versus the same type help and whether having labeled data from events occurring in close-proximity versus far-away from the target event has any effect on the performance of a lmr system. considering all these diverse settings, we formulate our research questions as follows: the research on the lmr task is currently lacking answers to all those questions. in this work, we perform extensive experiments in an effort to provide answers to them. we fix our learning model to a state of the art model (i.e., bert-based) and use a variety of datasets, i.e., web-based generalpurpose, twitter general-purpose, and twitter crisis-specific. our findings suggest that the generalpurpose datasets are not suitable for lmr in crisis tweets. moreover, the types of entities (e.g., per or org) used to train a model make a difference. specifically, training using only loc entities gives better performance than using all entity types. furthermore, while twitter datasets are preferred over general-purpose datasets, we observe that twitter crisis-related datasets help achieve better performance. while labeled data from the target event yield the best performance, we note that using labeled data from disasters happened in close proximity is helpful when the target labeled data is not available. the rest of the paper is organized as follows. we summarize related work in section 2. we present an overview of the lmr problem and define it formally in section 3. we discuss the experimental setup in section 4. we thoroughly analyze the results, answer the research question, and discuss the lessons we learned in section 5. we finally conclude and list some future directions in section 6. ## related work several past studies focusing on the lmr task exploit different techniques and features to extract location mentions (lms) from text @xcite . most of these proposed approaches are gazetteerbased in which public location gazetteers are employed such as geonames foot_0 @xcite @xcite , openstreetmap foot_1 @xcite , foursquare foot_2 @xcite , official new zealand gazetteer foot_3 (gelernter and balaji, 2013), and alexandria digital library gazetteer foot_4 @xcite , among others. although the gazetteer-based models achieve relatively high precision as they verify the candidate lms by matching entries occur in gazetteers, their main drawback is the inability to detect toponyms that do not appear in the gazetteers. additionally, the mismatch between the noisy twitter stream and nonnoisy gazetteer entries is a major issue. in this work, we aim to explore the effectiveness of exploiting available in-, cross-, and out-domain training data to build lmr models that learn the patterns of location in tweets without relying on gazetteers. to tackle the challenge of noisy twitter stream, @xcite , used information retrieval based approach. they indexed gazetteers' entries by their phonetic encodings using a multidimensional binary search tree (or k-d tree, where k is the dimensionality of the search space) @xcite . this technique mitigates the misspellings challenge efficiently. furthermore, @xcite and @xcite , constructed noisy gazetteers using cross-posts on twitter from foursquare check-ins. to detect lms, they developed a linear-chain crf model and trained it over lexical, grammatical, and geographical features. differently, @xcite and @xcite retrain standfordner using tweet dataset, as it was originally trained on newswire articles (conll-2003, muc 6 foot_5 , and muc 7 foot_6 ), to effectively identify the location mentions in tweets. more recently, @xcite proposed a statistical approach to construct regional language models. their tagger identifies the lms by traversing a tree of n-grams while matching them against region-specific gazetteers. furthermore, in 2014, the topic of the fifth australasian language technology association alta shared task was on identifying lms in tweets @xcite . participants explored several techniques such as feature engineering, ensemble classifiers, rule-based classification, knowledge infusion, crfs sequence labelers, semi-supervision. as for features, they used different features including geospatial, structural, and lexical. stanfordner was also used but after retraining it using tweet dataset. among all the related work, there is no single study that explores the setups we propose in this paper. typically, the proposed approaches are trained and tested on target events, assuming the training data is available at the onset of a disaster event, which is often not true. furthermore, none of the existing studies investigate the usefulness of labeled data from past events as well as whether geospatial proximity plays any role when choosing past events for training purposes. ## problem overview at the onset of a disaster event, response organizations and first-responders relying on twitter need geolocation information of reports or tweets about the crisis event in general as well as those seeking immediate help. in this case, the expectation is to find the mention of one or more locations in the textual content of a tweet reporting an event or asking for help. this is different than looking at the location information present in the twitter user profile, which is often used to find the user's home location. accordingly, we define the location mention recognition (lmr) task as the automatic extraction of toponyms (i.e., places or location names) from text. in this work, we limit the scope from two angles; we focus on tweets, and more specifically crisis-related tweets that are shared during emergencies and natural disasters. tweet #1 [user mention] dear friends, pl help by sending boat to 54 and 58, vivekananda nagar street, nesapakkm, chennai [...] tweet #2 [user mention] fear bridge being washed away. adayar bridge saidapet. hope tvk bridge is holding up fine at malhar [url] tweet #3 65 dead in earthquake, probably more, according to john key (prime minister) on the news #christchurch #earthquake ## experimental setup in this section, we describe the details of our experimental setup. we present the datasets in section 4.1 and the experimental configurations in section 4.2. we then discuss the base lmr model in section 4.3 followed by the evaluation measures in section 4.4. ## results and analysis in this section, we thoroughly discuss the results of our experiments to answer our research questions. we explore the usefulness of exploiting "out-domain" training data with either multiple entity types (such as person and organization) alongside the location entity (section 5.1) or with location entity alone (section 5.2). we further study the performance when training on "in-domain", "cross-domain", and "out-domain" data in sections 5.3 and 5.4. we finally discuss the effectiveness of considering the geographic proximity of disaster events when choosing the training data in section 5.5.