IDRISI - D : A rabic and E nglish Datasets and Benchmarks for Location Mention Disambiguation over Disaster Microblogs (2023)

Abstract: Extracting and disambiguating geolocation information from social media data enables effective disaster management, as it helps response authorities; for example, locating incidents for planning rescue activities and affected people for evacuation. Nevertheless, the dearth of resources and tools hinders the development and evaluation of Location Mention Disambiguation (LMD) models in the disaster management domain. Consequently, the LMD task is greatly understudied, especially for the low resource languages such as Arabic. To fill this gap, we introduce IDRISI-D, the largest to date English and the first Arabic public LMD datasets. Additionally, we introduce a modified hierarchical evaluation framework that offers a lenient and nuanced evaluation of LMD systems. We further benchmark IDRISI-D datasets using representative baselines and show the competitiveness of BERT-based models.

## introduction the cost-effectiveness and efficiency of communication over social media platforms make them primary sources of information during disaster events and emergencies. an essential dimension that makes the data extracted from microblogging platforms (e.g., x platform, formerly twitter) invaluable and actionable is the geolocation information. nevertheless, users typically opt to disable the geolocation functionalities over social media platforms to preserve their own safety and privacy which necessitates the development of geolocation extraction tools for social good. in this paper, we focus on the location mention disambiguation (lmd) task over microblogs that we exemplify by x posts. an lmd system aims at matching location mentions (lms) appearing in microblogs to toponyms, i.e., place or location names, in a geopositioning database, i.e., gazetteer. unfortunately, the research community lacks access to public disaster-specific microblogging lmd datasets, especially for low-resource languages, which consequently prevents the development and comparison of robust lmd systems. for example, there are only two english lmd datasets, namely singapore @xcite and geocorpora @xcite , where the former dataset is geographically confined, lacks event context, and is not publicly available, whereas the latter one (i.e., geocorpora @xcite ) is public, it has the same issues of low geographical coverage, lacking disaster event context, and many relevant/informative posts that do not contain the tracking keywords @xcite . on the other front, there are no arabic lmd datasets to the best of our knowledge. in this paper, we fill this gap and release idrisi-d datasets foot_0 for arabic (idrisi-da) and english (idrisi-de) languages. idrisi-da is the first public human-labeled arabic (a low-resource language) dataset, constituting 2,869 posts and 3,893 lms. idrisi-de is the largest to date humanlabeled english microblogging dataset in terms of number of lms. it constitutes 5,591 posts and 9,685 lms. additionally, to alleviate the lack of context challenge for microblogs and toponyms, we asked annotators to judge different features such as hashtags, replies, and urls, among others, for usefulness for the lmd task. furthermore, to evaluate the lmd systems, accuracy (acc), precision (p), recall (r), and the f β score are typically computed @xcite @xcite @xcite . while these measures evaluate binary classification tasks, the lmd task is usually perceived as a multiclass classification where every lm has only one (or no) correct toponym in gazetteers. moreover, distance-based methods @xcite , are also used to evaluate lmd systems within a distance d that is commonly set to 161 km (100 miles). for example, acc@d is the fraction of correctly predicted lms that are within d. however, tuning the d for different location granularity was not empirically investigated. to address these shortcomings, we propose evaluating the lmd systems using ranking evaluation measures, namely the mean reciprocal rank at cutoff r (m rr@r) in a lenient hierarchical strategy @xcite where systems are evaluated at different location granularity such as country, city, street, etc. indeed, the hierarchical evaluation substitutes the distance-based measures but in discrete manner. the contributions of this work are as follows: the remainder of this paper is organized as follows. we present the related work in section 2. we then define the lmd task in section 3. we introduce idrisi-d datasets and analyze them in sections 4 and 5, respectively. we then benchmark the datasets in section 6. we next discuss the dataset use cases in section 7. we finally conclude in section 9. ## related work in this section, we discuss the lmd related studies and discuss their technical solutions (section 2.1) and evaluation (section 2.2). ## problem definition the lmd system, as illustrated in figure 1 , is given the following inputs: the lmd system aims to match every location mention l i in the post p to one of the toponyms t j in g that accurately represents l i , if exists. otherwise, the system must abstain and declare that l i is unresolvable (or unlinkable). ## dataset construction in this section, we discuss the constructing process of idrisi-d datasets. we start by describing idrisi-r datasets. we then present the sampling strategy and the annotation process. idrisi-r datasets: we extend idrisi-r location mention recognition (lmr) english (idrisi-re) @xcite and arabic (idrisi-ra) @xcite datasets that are originally sampled from humaid @xcite and kawarith (alharbi and lee, 2021) datasets, respectively. we select these datasets due to their unique characteristics as described below. idrisi-re is the largest to date lmr microblogging english dataset. it exhibits unique diversity (domain and location types), coverage (temporal and geographical), and generalizability (domain and geographical), compared to all existing datasets of its kind. it comprises around 20k human-labeled (gold) and 57k machine-labeled (silver) posts from 19 disaster events of diverse types covering wide geographical areas. the events capture the critical periods of disaster events. the annotations include spans of location mentions in the textual content alongside their location types (e.g., country, city, street). empirically, idrisi-re is the best domain and geographical generalizable dataset against all existing english datasets. idrisi-ra is the first arabic lmr microblogging dataset. it contains 22 disaster events of different types that happened in arab countries, covering various dialects reasonably. it contains 4.6k manually-annotated (gold) posts sampled from 7 disaster events, foot_1 and 1.2m automatically-annotated (silver) posts sampled from the entire dataset. both versions are labeled for location mentions and location types. empirically, the lmr models trained on idrisi-ra showed decent generalizability to unseen events and acceptable domain and geographical generalizability. dataset sampling: constrained by not overwhelming the volunteered annotators, we sampled a set of posts from every disaster event in idrisi-re while maintaining the distribution of lm types, but covering all fine-grained lms including neighborhoods, streets, and pois. in total, we sampled 8,224 posts containing 11,023 lms. on the other hand, the idrisi-ra gold version was labeled entirely, including seven events containing 2,974 having lms (the remaining 1,618 posts do not contain lms) and 5,236 lms. ## dataset analysis idrisi-d datasets inherit the geographical, domain, location types, temporal, informativeness, and dialectical (for arabic) coverage from idrisi-r datasets. in this section, we analyze the reliability of annotations and the usefulness of post features for the lmd task. reliability: to evaluate the reliability of annotations in phase 1, we compute the inter-annotator agreement (iaa) using cohen's kappa @xcite . we measure the iaa for the ability to resolve lms, i.e., whether an lm is resolvable or not. we also compute the agreement percentage on the extracted toponyms from gazetteers by the annotators for all lms. the annotators in phase 1 achieved substantial and almost perfect cohen's kappa scores of approximately 0.90 and 0.83 for idrisi-da and idrisi-de datasets, respectively. the raw agreement percentages are around 97.98% and 93.50% for idrisi-da and idrisi-de datasets, respectively. these results statistically demonstrate the high quality and reliability of annotations of idrisi-d datasets. to further increase the quality of the datasets, we resolved the disagreement cases in the subsequent annotation phases 2 and 3. usefulness of features: looking carefully at the annotations of features' usefulness, we make different observations through examples in other lms: the geo-vicinity between cooccurring lms usually represents inclusion and containment relationships, making the coarsegrained lms useful to disambiguate the finegrained lms. for instance, in post #2, " " ("beirut" is a city) which is also a hashtag is helpful for accurately disambiguating " " ("saint george hospital" is a human point-of-interest). similarly, in post #4, "nebraska" (state) was useful to distinguish "elkhorn river" (human point-of-interest) from another part of the river located in "west virginia" (state). different reasons cause the low usefulness percentages of "other lms". to elaborate, in cases where the same lm appears multiple times in the same post, the duplicates are useless for disambiguating each other. hashtags: as most hashtags indicate the disaster event (e.g., " _ _ " and ' _ " in posts #1 and #2), they are equally important to the "event" feature. replies: typically, a small number of posts get the community attention. hence, replies are rarely useful for lmd. urls: linked web pages are useful if they elaborate on the geographical context of the reported information in the post. for example, the linked web page in post #2 was useful for locating the hospital. also, "lake butler" in post #4 is challenging lm. the linked facebook page contains "lake butler, fl, united states" and "keystones heights" that helped the annotator to successfully resolve this lm by their geo-proximity. the importance percentage of urls is low as many urls are already broken or require a paid subscription. it is worth noting here that the coarse-grained lms are usually easy to disambiguate without exploiting any features (e.g., posts #3 and #6). ## benchmarking experiments in this section, we discuss the experimental setup and results of benchmarking idrisi-d. ## research use cases releasing idrisi-d enables research on disasterspecific and generic geolocation applications that we discuss below: event/incident detection: while lms indicate where events and incidents took place @xcite , idrisi-d datasets with their resolved lms could serve event/incident detect models that exploit geospatial features. relevance filtering: while lms increase the likelihood of microblogs being relevant and informative with regard to the disaster events @xcite , idrisi-d can enable research on relevance filtering approaches that utilize geospatial information. geolocation applications: while the lmp tasks play a key role in tackling all of the geolocation tasks (e.g., predicting post location @xcite , inferring user location @xcite , modeling user movement @xcite , etc.) that employ textual features @xcite , idrisi-d is an invaluable resource for tackling all these tasks. geographical retrieval: the geographical information retrieval (gir) systems are concerned with extracting spatial information alongside the relevant multimodal data to the user information need. idrisi-d could empower the gir retrieval techniques that rely on applying lmp tasks over queries and documents @xcite . ## challenges compared to gazetteers, posts over social media contain informal language, misspellings, grammar mistakes, shortened words, and slang, causing the so-called mismatch challenge @xcite . t# challenge post text 1 nicknames # _ # 2 #chennaifloods sad to see the state of city. detroit of india is suffering. hv personal experienced. misspelling _ # ... 6 medical students of shri ramchandra medical college in chennai stranded without supplies. need help. 7 shortcuts ... 8 sm 1 help providing water 50 children @lawrence charitable trust.safe.2/4,1st cross st,3rd avenue,ashoknagar-lakshmansruti #chennaifloods and " " (bridge in english), respectively. also, using "st" instead of "road", in post #8. also, using "@" symbol instead of the literal "at" prepositions in the same post. capitalization: users tend to ignore capitalization when writing posts (e.g., "chennai" instead of "chennai" in post #6. dialectics and varieties: " " (bridge in english) in post #7 is the dialectical (e.g., egyptian) form of in modern standard arabic (msa). ## conclusion this paper contributes towards a crucial task, i.e., location mention disambiguation in the crisis management domain. we introduced idrisi-d, the first arabic and the largest to date english lmd datasets.the lmd annotations that are of high reliability indicating the usefulness of the dataset. a key characteristic of idrisi-d is the annotations of features' usefulness that we anticipate to guide the development of lmd tools. our benchmarking results show the competitiveness of simple exact matching (nominatim) and the promising performance of contextual features (bert lm d ) for learning lmd. we release the datasets and the evaluation script for the research community. the future directions are two-fold: (i) enhancing the representation of lms and toponyms for robust lmd learning, and (ii) employing advanced learning algorithms.