TIMBERT : Toponym Identifier For The Medical Domain Based on BERT (2020)

Abstract: In this paper, we propose an approach to automate the process of place name detection in the medical domain to enable epidemiologists to better study and model the spread of viruses. We created a family of Toponym Identification Models based on BERT (TIMBERT), in order to learn in an end-to-end fashion the mapping from an input sentence to the associated sentence labeled with toponyms. When evaluated with the SemEval 2019 task 12 test set (Weissenbacher et al., 2019), our best TIMBERT model achieves an F1 score of 90.85%, a significant improvement compared to the state-of-the-art of 89.13% (Wang et al., 2019).

## introduction phylogeographers, who study the geographic distribution of viruses, have long linked the increase in the geographical spread of viruses @xcite to the growth in global tourism and international trade of goods. most notably, in december 2019, a pneumonia-like disease, later dubbed covid-19, that was detected in the city of wuhan, china quickly became a world pandemic mainly due to global travels @xcite . epidemiologists study the global impact of the spread of viruses by considering information on the dna sequence and structure of viruses, as well as relying on accurate metadata. although accurate localized geographical data is critical for creating maps of the locations of viruses and their migration paths, most publicly available databases, such as genbank @xcite , provide insufficient details on the matter, limited to country or state level. in a study by @xcite , it is estimated that 7% of the genbank records are missing geospatial metadata and 73% lacking detailed records. however, more fine-grained localization information on the viruses may be present in articles that describe the research work. therefore, a manual inspection of biomedical articles is vital for obtaining more detailed information about the locations of the viruses. toponym detection aims to identify the word boundaries of expressions that denote geographic names. toponym detection has been the focus of much work in recent years @xcite @xcite and studies have shown that the task is highly dependent on the textual domain @xcite @xcite @xcite . the focus of this paper is to propose a competitive deep learning based model for toponym detection that learns in an end-to-end fashion the mapping from an input sentence to the associated sentence with toponym labels. we evaluated our models using the semeval 2019 task 12 test set @xcite and report their performances based on precision, recall and f1 respectively. these metrics can be measured in two ways: strict or overlapping. the strict measures, consider a prediction to match the gold standard annotation if both point to the exact same span of text at the character level. on the other hand, the overlapping measures are more lenient as they consider a prediction to match the gold standard annotations when they share a common span of text. since the research community in toponym identification is more concerned with strict measures @xcite , we only report on the strict measures of precision, recall and f1. as reported in section 4, our best model achieves a strict f1 score of 90.85% on the semeval 2019 task 12 test set @xcite . ## previous work categorizing each word of a text as toponym or non-toponym, is the focus of toponym detection. for example, given the sentence: covid-19 was first reported in wuhan, hubei province, china. foot_0 a toponym detection system should identify wuhan, hubei province, and china as toponyms, and all other words as non-toponyms. toponym detection tackles ambiguities between toponyms and other classes of named entity recognition as well as metonymic usage of toponyms. toponym resolution in the epidemiology domain was the objective of the semeval 2019 task 12 @xcite . the majority of current approaches to toponym detection in the medical domain are based on a combination of rule-based techniques, geographical gazetteer approaches, and deep learning models @xcite @xcite . in our approach, we aim to develop a toponym detection system solely based on deep learning techniques in order to sidestep many design choices and avoid pattern mining that comes along with rule-based techniques and gazetteer approaches. contextualized embeddings previous attempts to toponym detection in the medical domain have used contextualized embeddings, specifically elmo @xcite , as the core of their models @xcite . in this paper, we experiment with a different contextualized embedding model and choose the pretrained bert model @xcite as the backbone of our network architecture. ## our proposed model the architecture of our toponym recognition model is shown in figure 1 . the wordpiece @xcite tokenization of a sentence constitutes the input of the model. these tokens are then passed to a pretrained bert network @xcite . the output of the network along with certain linguistic features (see section 4) are then concatenated and passed to a fully connected layer which determines the labels of each token. in our experiments, we used two variations of the bert model: bert-base and bert-large @xcite . the corresponding timbert models are called timbert-base and timbert-large. since the bert-large model is much more computationally expensive than bert-base, we limit the experiments involving this model. 4 experiments and results our models were evaluated on the semeval 2019 task 12 @xcite dataset. the dataset consists of 150 articles from pubmed annotated with toponym mentions. the dataset was split into 3 sections: training, validation, and test set containing 60%, 10%, and 30% of the dataset respectively. ## discussion our search for a toponym identifier for the medical domain with little to no task-specific design choices led us to the development of the timbert models. our experiments with bert as the backbone of our models detailed in section 4 confirmed that certain linguistic insights such as pos tags are seamlessly transferred to downstream tasks while others, such as orthographic features need to be integrated to the model. our experiments with the typical preprocessing techniques led to poor model performances. this suggests the need for a general agreement between the textual structure of the data during pretraining and fine-tuning. ## conclusion and future work in this work, we presented a competitive model for toponym detection in the medical domain that significantly improves the state-of-the-art performance. we developed a family of toponym detection models and used bert as the backbone of our models. in future studies, we will investigate the effects of using other language models, such as xlnet @xcite , roberta @xcite , and albert @xcite , for the backbone module. we experimented with parameter pruning to regularize our models as well as to reduce their computational complexity. in future studies, we will examine other knowledge distillation techniques in order to find competitive and resource conservative models for toponym identification in the medical domain. our experiments with precursory fine-tuning resulted in significant performance improvement of our model. further research can determine if precursory task specific fine-tuning is helpful for other nlp tasks. this can potentially lead to the development of task specific pretrained models for more efficient transfer learning, especially for tasks with small datasets.