Crossroads, Buildings and Neighborhoods: A Dataset for Fine-grained Location Recognition (2022)

Abstract: General domain Named Entity Recognition (NER) datasets like CoNLL-2003 mostly annotate coarse-grained location entities such as a country or a city. But many applications require identifying fine-grained locations from texts and mapping them precisely to geographic sites, e.g., a crossroad, an apartment building, or a grocery store. In this paper, we introduce a new dataset HarveyNER with fine-grained locations annotated in tweets. This dataset presents unique challenges and characterizes many complex and long location mentions in informal descriptions. We built strong baseline models using Curriculum Learning and experimented with different heuristic curricula to better recognize difficult location mentions. Experimental results show that the simple curricula can improve the system’s performance on hard cases and its overall performance, and outperform several other baseline systems. The dataset and the baseline models can be found at https://github.com/brickee/HarveyNER .

## related work ner research has a long history and many ner datasets have been created with certain pre-defned entity categories. general domain datasets such as @xcite and ontonotes 5.0 @xcite attend to certain common entity types including location. location mentions in these datasets are mostly coarsegrained, e.g., the u.s. (a country) or london (a city). @xcite ; @xcite focus on identifying fne-grained points-of-interest for location-based services, and their dataset is automatically constructed by mapping location inventory to tweets. @xcite ; @xcite try to identify crisis-related location mentions, but their dataset contains incomplete location mentions and is of limited use for a disaster response system. in contrast, our dataset harveyner emphasizes fne-grained locations that can map to coordinates on a map. recent approaches @xcite @xcite use neural network models like bilstm-cnn-crf @xcite and contextual embeddings like bert @xcite , and have greatly improved the ner performance. however, none of these approaches consider the diffculty of different ner cases in their model training. @xcite pointed out that using a curriculum strategy enables the model to learn from easy examples to complex ones and leads to generalization improvement. many natural language processing tasks such as machine translation @xcite @xcite , natural language understanding @xcite ), text generation @xcite and dialogue systems @xcite beneft from such curriculum learning strategies. considering that harveyner contains many long and complex location mentions, we design corresponding curricula to learn them. data cleaning we applied several strategies to flter out irrelevant tweets. first, we only keep the tweets that are related to the houston area, i.e., the geo-coordinates of the tweets or the authors' profle locations are within the bounding of houston. second, we applied our weakly-supervised event detection system @xcite to identify tweets on disaster-related topics; these tweets are likely to be related to hurricane harvey during the specifed period. we also manually fltered out remaining irrelevant tweets (such as non-english and repeated tweets) during the annotation process. in total, 6,571 tweets were selected for this study, as shown in ## curriculum arrangement in consideration of the characteristic diffculties of harveyner, we employ curriculum arrangements to help learn these hard cases. we follow the curriculum designing approach introduced by @xcite , which mainly requires specifying two functions: the curriculum learning procedure using the two functions is described in algorithm 1. ## experiments in our experiments, we use two state-of-the-art ner systems as base models and evaluate their performance on the harveyner dataset. then, we test the effectiveness of the designed curricula by applying them to train the base models. ## conclusion we introduce a fne-grained location recognition dataset, harveyner, to enable many downstream applications such as building real-time disaster response systems. this dataset contains many long and complex location mentions that feature interesting internal syntactic and semantic structures and the state-of-the-art ner systems are unable to fully recognize these hard cases. considering the clear characteristics of diffcult cases in this dataset, we experimented with two heuristic curriculum learning strategies and a novel commonness-based curriculum strategy to better recognize the diffcult location mentions. empirical results demonstrate the effectiveness of the curricula, which serve as strong baseline results in this dataset. future work may consider incorporating external knowledge or innovations on system architectures to better identify fne-grained location mentions. yuchen zhang, and zhi zhong. 2013. towards robust linguistic analysis using ontonotes. in proceedings of the seventeenth conference on computational natural language learning, pages 143-152, sofa, bulgaria. association for computational linguistics. yixuan su, deng cai, qingyu zhou, zibo lin, simon baker, yunbo cao, shuming shi, nigel collier, and yan wang. 2021. dialogue response selection with hierarchical curriculum learning. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 1740-1751, online. association for computational linguistics. erik f. tjong kim sang. 2002. introduction to the conll-2002 shared task: language-independent named entity recognition. in coling-02: the 6th conference on natural language learning 2002 (conll-2002). ashish vaswani, noam shazeer, niki parmar, jakob uszkoreit, llion jones, aidan n. gomez, lukasz kaiser, and illia polosukhin. 2017. attention is all you need. in advances in neural information processing systems 30: annual conference on neural information processing systems 2017, december 4-9, 2017, long beach, ca, usa, pages 5998-6008. benfeng xu, licheng zhang, zhendong mao, quan wang, hongtao xie, and yongdong zhang. 2020. curriculum learning for natural language understanding. in proceedings of the 58th annual meeting of the association for computational linguistics, pages 6095-6104, online. association for computational linguistics. jie yang and yue zhang. 2018. ncrf++: an opensource neural sequence labeling toolkit. in proceedings of acl 2018, system demonstrations, pages 74-79, melbourne, australia. association for computational linguistics. wenlin yao, cheng zhang, shiva saravanan, ruihong huang, and ali mostafavi. 2020. weakly-supervised fne-grained event recognition on social media texts for disaster management. in the thirty-fourth aaai conference on artifcial intelligence, aaai 2020, the thirty-second innovative applications of artifcial intelligence conference, iaai 2020, the tenth aaai symposium on educational advances in artifcial intelligence, eaai 2020, new york, ny, usa, february 7-12, 2020, pages 532-539. aaai press. mingliang zhang, fandong meng, yunhai tong, and jie zhou. 2021. competence-based curriculum learning for multilingual machine translation. in findings of the association for computational linguistics: emnlp 2021, pages 2481-2493, punta cana, dominican republic. association for computational linguistics.