IDRISI - RA : The First A rabic Location Mention Recognition Dataset of Disaster Tweets (2023)

Abstract: Extracting geolocation information from social media data enables effective disaster management, as it helps response authorities; for example, in locating incidents for planning rescue activities, and affected people for evacuation. Nevertheless, geolocation extraction is greatly understudied for the low resource languages such as Arabic. To fill this gap, we introduce IDRISI-RA, the first publicly-available Arabic Location Mention Recognition (LMR) dataset that provides human- and automatically-labeled versions in order of thousands and millions of tweets, respectively. It contains both location mentions and their types (e.g., district, city). Our extensive analysis shows the decent geographical, domain, location granularity, temporal, and dialectical coverage of IDRISI-RA. Furthermore, we establish baselines using the standard Arabic NER models and build two simple, yet effective, LMR models. Our rigorous experiments confirm the need for developing specific models for Arabic LMR in the disaster domain. Moreover, experiments show the promising domain and geographical generalizability of IDRISI-RA under zero-shot learning.

## introduction worldwide, and in the arab world, twitter has played a critical operational role in crisis management. the beirut explosion in 2020 is an excellent case in point, where on-site individuals started intuitively responding to each other using geotagged tweets. what makes tweets invaluable is the presence of location mentions at different granularity @xcite @xcite . response authorities exploit this geographical information to effectively manage emergencies using crisis maps. although the geographical dimension adds situational and operational values to twitter data, on 18 june 2019 twitter discontinued the geotagging feature in tweets. 1 this necessitates the need to develop automatic geolocation tools. nevertheless, the main obstacle for the arabic language, which is a low-resource language, the lmr task is severely understudied due to several factors, including the absence of a unified evaluation framework constituting annotated datasets, a representative set of baselines, and fair evaluation metrics. to address these barriers, we focus on the location mention recognition (lmr) task and introduce idrisi-ra, foot_1 the first human-labeled dataset comprising arabic tweets from 7 disaster events (gold annotations). we also introduce the first largescale automatically-labeled tweets (silver annotations) from 22 disaster events that cover all arab world. idrisi-ra covers the most occurring disaster types that happened in the arab world. more importantly, it is labeled for two annotation types that are location mentions (i.e., toponym textual spans) and their types (e.g., city, street, pois, etc.). hence, it supports type-less lmr, where the model detects toponyms at any granularity, and type-based lmr, where the model distinguishes lms types while detecting them @xcite . although adapting named entity recognition (ner) models and english datasets goes a long way towards tackling the lmr task, researchers have empirically shown that specialized lmr models yield better performance and are more effective for emergency management tasks @xcite . for the arabic language, only a few datasets exist yet suffer from their limited geographical, domain, and dialectal coverage. what exacerbates the low resources issue for the arabic language is the unavailability of lmr datasets, except a few task-specific ones, including traffic surveillance and event detection @xcite @xcite . therefore, in an effort to expedite the development of arabic lmr models and shape future directions, we perform extensive experiments to empirically answer the following research questions: rq1: are standard arabic ner models sufficient for effective lmr over disaster tweets? rq2: can lmr models trained on idrisi-ra generate generalizable lmr models that reasonably perform on unseen disaster events? rq3: can lmr models trained on idrisi-ra generate domain generalizable lmr models that reasonably perform on unseen disaster events of the same or different types? rq4: can lmr models trained on idrisi-ra generate geographically generalizable lmr models that reasonably perform on unseen disaster events that happened in different countries? our rigorous analyses and experiments necessitate the development of specialized lmr models for the disaster domain. additionally, the experiments demonstrate promising domain and geographic generalizability of idrisi-ra under zeroshot learning. the contributions of this paper are as follows: the rest of the paper is organized as follows. section 2 discusses the related work. section 3 presents an overview of the lmr problem. section 4 describes the design objective, dataset creation, and annotation. section 5 analyzes the relia- ## related work in this section, we review the arabic twitter ner and lmr datasets. we present their characteristics and issues and discuss how idrisi-ra dataset overcomes these limitations. in ## problem overview in this work, we focus on the location mention recognition (lmr) task that aims to automatically recongnize and extract toponyms (places or location names) from text. to distinguish the lmr from other tasks, we emphasize that the lmr task aims at removing geo/non-geo ambiguity of tokens in text. it is also known as location extraction or geoparsing in the literature. differently, the location mention disambiguation (lmd), which is a consecutive task for lmr, aims at removing geo/geo ambiguity between candidate lms extracted by lmr systems. the lmd task is also known as location resolution, location linking (looking up a geo-positioning database), or geocoding (assigning geo-coordinates to lms) in the literature. there are two types of lmr tasks. the first type recognizes toponyms (e.g., paris, new york) without their types (e.g., city, state) and is denoted as "type-less lmr". the second type recognizes toponyms and also distinguishes between location types (e.g., country, city, and street) and is denoted as "type-based lmr" @xcite . the latter better serves the development and evaluation of geolocation processing systems in light of the responders' needs. it enables a variety of downstream tasks (e.g., crisis maps) at different location granularity, in addition to being crucial for accurately disambiguating the toponyms. the lmr problem is formally defined as follows: given a tweet t, the lmr system aims to identify all location mentions l @xmath0; @xmath1, n t ]} in the tweet t, where l i is the i th location mention and n t is the total number of location mentions in t, if any. each location mention may span one or more tokens. in this work, we follow the bilou annotation scheme (bilou_tag) with 5 classes: "b" denotes the beginning token of an lm, "i" denotes a token inside an lm, "l" denotes the last token in an lm, "o" denotes a token outside of an lm, and "u" indicates that the lm has only one token such as "doha". therefore, we define the lmr as a multi-class classification task on the token level. a type-less lmr model predicts whether a token is part of a location mention (<@xmath2, while a type-based lmr model predicts whether a token is part of lm of specific type (<@xmath3. @xmath4.2. ## dataset construction this section discusses design objectives, data selection, and data annotation details. ## dataset description and quality in this section, we evaluate idrisi-ra datasets for reliability, diversity, and coverage. ## benchmarking experiments in this section, we discuss idrisi-ra benchmarking experiments that we conducted to provide baselines for the research community. ## dataset generalizability in this section, we empirically study the generalizability of idrisi-ra dataset. for that, we employ the best lmr model, marbert-based (refer to section 6); hereafter, we refer to it as "the model". we study three dimensions: (i) generalizability to unseen events regardless of their type and geolocation, (ii) generalizability to unseen events of the same or different disaster types (domain generalizability), and (iii) generalizability to unseen events that happened in the same or different countries (geographical generalizability). ## silver idrisi-ra given the dearth of arabic lmr datasets, we expand the size of idrisi-ra dataset by employing the best-performing lmr model to infer labels for 1.2 million arabic tweets posted during 22 disaster events. we call the resulting data as the silver version to indicate the reliability of annotations. the inference model is trained using the entire type-based and type-less train sets from all events. all development partitions are used for tuning the hyperparameters of the marbert-based model. the model applied to the tweets resulted in 884,217 lms (18,609 distinct). we note that this large dataset can potentially be useful for a variety of lmr approaches, e.g., domain adaptation, transfer learning, and semi-supervised learning. ## research use cases releasing idrisi-ra empowers research on different applications, other than the geolocation. in this sections, we discuss a few use cases. event/incident detection: people tend to mention where events/incidents take place when they report them @xcite . following @xcite and @xcite , idrisi-ra can be employed to detect event/incident on twitter by extracting lms. relevance filtering: prior studies show that the geographical references in social media messages could indicate their relevance and informativeness for a disaster event @xcite . therefore, idrisi-ra can be utilized to train relevance filtering models. displacement monitoring: extracting lms from tweets shared by displaced people and refugees can help predicting their future paths. in fact, monitor the population movement during emergencies is very useful for responders. thus, idrisi-ra can be exploited in modeling the population movement. geographical retrieval: idrisi-ra could serve the gir retrieval techniques that rely on detecting locations and spatial references in queries and documents @xcite . ## ethical considerations while twitter allows users to self-manage their privacy by disabling the geo-tagging functionalities, "even well informed and rational individuals cannot appropriately self-manage their privacy" (solove, 2012) due to lack of awareness on how such data can be collected or commercially used. however, geolocation extraction could be justified in the context of social good during natural disaster events where protecting geographical privacy could be of least importance as affected people need to rescue their lives or get basic necessities of life @xcite . the privacy of affected people during human-made disaster events is more critical because revealing their locations during conflicts and wars could risk their lives. therefore, we limit our dataset to natural disaster events and covid-19 pandemic and apply a couple of de-identification steps to protect users' privacy (refer to section a). we further limit the dataset usage to research purposes only by releasing it under the creative commons attribution 4.0 international license. 8 additionally, we emphasize that systems developed for lmr using idrisi-ra dataset should implement proper mechanism for preserving user privacy. ## conclusion we introduced idrisi-ra, the first arabic lmr twitter dataset. it contains 22 disaster events of different types that happened in the arab region. we manually-(gold) and automatically annotated (silver) about 4.6k and 1.2m tweets. both versions are annotated for location mentions and location types which form the value and uniqueness of idrisi-ra. our analysis showed that idrisi-ra is second to none in empowering research for arabic lmr. additionally, the extensive experiments emphasize the need for developing lmr-specific models for the disaster domain. the developed lmr baselines are simple yet competitive ones. the results also demonstrated the decent generalizability of idrisi-ra. for future work, we plan to extend the annotations for the location mention disambiguation (lmd) task. we further plan to explore different transfer learning, domain adaptation, and active learning techniques to tackle the lmr task.