Arukikata Travelogue Dataset with Geographic Entity Mention, Coreference, and Link Annotation (2024)

Abstract: Geoparsing is a fundamental technique for analyzing geo-entity information in text, which is useful for geographic applications, e.g., tourist spot recommendation. We focus on document-level geoparsing that considers geographic relatedness among geo-entity mentions and present a Japanese travelogue dataset designed for training and evaluating document-level geoparsing systems. Our dataset comprises 200 travelogue documents with rich geo-entity information: 12,171 mentions, 6,339 coreference clusters, and 2,551 geo-entities linked to geo-database entries.

## introduction human activities, mobility, and events are often described with natural language expressions of locations or geographic entities (geo-entities), which indicate the geographic positions in the real world. this signifies the importance of technologies for extracting and grounding geo-entity expressions for various application domains, including tourism management, disaster management, and disease surveillance (hu et al., 2022). geoparsing (leidner, 2006; gritta et al., 2020) is a fundamental technique involving two subtasks: geotagging, which identifies geo-entity mentions, and geocoding, which identifies corresponding database (db) entries for (or the coordinates of) geo-entities. notably, geoparsing, geotagging, and geocoding can be regarded as special cases of entity linking (el), named entity recognition (ner) or mention recognition (mr), and entity disambiguation (ed), respectively. this study focuses on geoparsing from the perspective of document-level analysis. geo-entity mentions that co-occur in a document tend to be geographically close or related to each other; thus, this paper presents a dataset suitable for document-level geoparsing: the arukikata travelogue dataset with geographic entity mention, coreference, and link annotation (atd-mcl). our dataset includes the three types of geo-entity information illustrated in figure 1: (1) spans and entity types of geo-entity mentions, (2) coreference relations among mentions, and (3) links from coreference clusters to corresponding entries in a geographic db (geo-db). our dataset has two desirable characteristics for document-level geoparsing. the first characteristic is that single travelogue documents in our dataset contain a rich amount of geo-entity mentions, in contrast to short documents, e.g., social media posts (matsuda et al., 2017; wallgrün et al., 2018). to leverage the inherent characteristic of the original travelogues, we have adopted an annotation policy to exhaustively markup geo-entity mentions, which refer to various locations and facilities expressed by named, nominal, and deictic expressions. the second characteristic is the geographic continuity among co-occurring mentions; that is, mentions that refer to nearby locations in the real world tend to appear near to one another within a document. because travel records reflect the actual trajectories of travelers, this characteristic is more notable in travelogues than other text genres, e.g., news articles (lieberman et al., 2010; kamalloo and rafiei, 2018; gritta et al., 2018, 2020). the potential applications of our dataset (and constructed geoparsers) include but not limited to tourism management applications. this is because geoparsing of location and facility mentions with diverse surface forms is essential for gaining a detailed understanding of where some event happened from text. for example, in disaster prevention/mitigation applications, it is crucial to specify detailed geographical positions by analyzing expressions other than named locations, utilizing geographic continuity if available, from social media posts about ongoing disasters and reports on past disasters. as a result of manual annotation, our dataset comprises 12,273 sentences from the full text of 200 travelogue documents with 12,171 mentions, 6,339 coreference clusters (geo-entities), and 2,551 linked geo-entities. foot_0 furthermore, we have conducted two types of evaluation using our dataset. first, we have measured inter-annotator agreement (iaa) for three types of information; the results indicate the practical quality of our dataset in terms of consistency. second, we have evaluated current entity analysis systems on our dataset for benchmarking baseline performance; the results demonstrate that reasonable performance can be achieved for mr and coreference resolution (cr), but performance has room for improvement in ed. we will release our annotated dataset at https://github.com/naist-nlp/atd-mcl and experimental codes at https://github.com/ naist-nlp/atd-mcl-baselines . ## dataset annotation design strategy for building geoparsing datasets, it has been challenging to achieve a high coverage for facility entity mentions mainly because of the limited coverage of public geo-dbs, e.g., geonames. @xcite to address this db coverage problem, we adopt openstreetmap (osm), foot_3 a free, editable, and large-scale geo-db of the world. the usefulness of osm has been continually increasing, as evidenced by the increase in node entries from over 1.5b in 2013 to over 80b in 2023. @xcite furthermore, we define entity types to cover broad types of location and facility mentions, including districts, buildings, landmarks, roads, and public transport lines and vehicles, as described in §2.2. annotation flow following the data preparation by the authors, annotation work was performed by native japanese annotators at a professional data annotation company according to the three-step annotation flow: (1) mention annotation, (2) coreference annotation, and (3) link annotation. ## dataset statistics the annotators first annotated 200 documents with mention information, then annotated the same 200 documents with coreference information, and finally annotated 100 documents, which were randomly sampled from the 200 documents, with link information. @xcite we call the latter 100 documents that contain link annotation set-b and refer to the remaining 100 documents without link annotation as set-a. the numbers of documents (#doc), sentences (#sent), words (#word), mentions (#men), and entities (coreference clusters) (#ent) in the atd-mcl are listed in ## inter-annotator agreement for mention, coreference, and link annotation, we requested two annotators to independently annotate the same 10, 10, and 5 documents out of the 200, 200, and 100 documents, respectively; we simply selected 10 or five documents in ascending order based on document id. @xcite we measured the interannotator agreement (iaa) for the three annotation tasks. ## experiments we conducted experiments on the atd-mcl for three tasks: mr, cr, and ed. the purpose of the experiments is to clarify the performance level of current entity analysis systems, including off-theshelf and finetuned models, on our dataset. ## conclusion this paper has described the atd-mcl dataset, which is designed for document-level geoparsing, along with the annotation criteria, iaa assessment, and performance evaluation of the baseline systems. our dataset enables other researchers to conduct reproducible experiments through the public release of our annotated data. we expect that our dataset contributes to fostering future research and advancing geoparsing techniques. in future work, we plan to (1) develop a document-level geoparser that leverages both characteristics of geo-entity mentions in text and geo-db entries, (2) enhance our dataset with additional semantic information, such as the movement trajectories of travelogue writers, for more advanced analytics, and (3) construct annotated travelogue datasets in other languages by extending our annotation guidelines.