Evaluating Language Models in Location Referring Expression Extraction from Early Modern and Contemporary J apanese Texts (2024)

Abstract: Automatic extraction of geographic information, including Location Referring Expressions (LREs), can aid humanities research in analyzing large collections of historical texts. In this study, to investigate how accurate pretrained Transformer language models (LMs) can extract LREs from historical texts, we evaluate two representative types of LMs, namely, masked language model and causal language model, using early modern and contemporary Japanese datasets. Our experimental results demonstrated the potential of contemporary LMs for historical texts, but also suggest the need for further model enhancement, such as pretraining on historical texts.

## introduction historical texts are crucial for a better understanding human and natural history because they record various events and activities of their time. from a geographic perspective, historical texts often include location referring expressions (lres), such as historical place and facility names, along with objects and events related to those locations. as representative examples of such texts, travelogues describe the experiences of the writer in the places they visited, and disaster records describe the affected regions, the scale of the damage, and peoples' situation. automatic extraction and structurization of such geography-related information by computers can support humanities scholars in analyzing large collections of historical texts. as a fundamental step for computer-aided geographic text analysis, this study addresses lre extraction from historical japanese texts. for an example sentence "名取川渡りて仙台に入る," foot_0 an lre system is required to extract two lres, "名 取川 (natorigawa)" and "仙台 (sendai)." specifically, we investigates the lre accuracy of trans-former @xcite language models (lms), which have achieved remarkable success in various natural language processing tasks @xcite @xcite . we focus on two representative types of lms: masked language model (mlm) and causal language model (clm). for model evaluation experiments, we use three datasets: an early modern japanese travelogue to which we added lre annotations, oku no hosomichi (hosomichi), foot_1 early modern japanese disaster records, the minna de honkoku dataset @xcite (minna), and contemporary japanese travelogues, the arukikata travelogue dataset (arukikata. co., ltd., 2022; ouchi et al., 2023) (arukikata). the reasons for using contemporary texts alongside historical texts are twofold: (i) comparing model performance across texts from different eras, and (ii) investigating whether contemporary texts can enhance model performance on historical texts. our experiments demonstrated the following results: ## background and related work lre extraction, also known as geotagging or toponym recognition, is a special case of named entity recognition (ner) @xcite . lre extraction has often been addressed within the task of geoparsing @xcite , which aims to estimate the geographic coordinates or geographic database entries that correspond to the locations referenced by lres. resources previous studies have constructed location-annotated historical corpora and evaluated the performance of machine learning systems in lre extraction using, for example, english news articles @xcite , english travel writings @xcite , french literary texts @xcite , and chinese historical books @xcite . for japanese, some researchers have attempted to manually annotate lres and their geographical coordinates in texts within historical disaster record databases, such as the database of materials for the history of japanese earthquakes @xcite and minna de honkoku @xcite . system evaluation many studies have investigated various methods for recognizing named entities, including locations, in historical texts @xcite . in particular, some recent studies have focused on pretrained transformer lms. labusch et al. (2019) investigated training strategies for bert suitable for ner on historical german newspaper texts. they showed that a contemporary bert model achieved the highest accuracy when both pretraining on large unlabeled historical texts and labeled contemporary texts were performed prior to fine-tuning on target labeled historical texts. tang et al. (2024) evaluated ner accuracy on ancient chinese historical documents using mlms pretrained on historical texts, and open and closed contemporary clms, with mlms achieving higher accuracy. ## results and discussion ## conclusion this study investigated the extraction accuracy of representative pretrained japanese lms using early modern and contemporary lre datasets. one of main findings from our experiments is the effectiveness of fine-tuning with both contemporary and historical labeled texts. possible future work includes (i) expanding the evaluation to cover a broader range of eras and registers, and (ii) investigating pretraining strategies using unlabeled historical texts effective for downstream tasks, including lre extraction and others.