Crossroads, Buildings and Neighborhoods: A Dataset for Fine-grained Location Recognition
Abstract
General domain Named Entity Recognition (NER) datasets like CoNLL-2003 mostly annotate coarse-grained location entities such as a country or a city.
But many applications require identifying fne-grained locations from texts and mapping them precisely to geographic sites, e.g., a crossroad, an apartment building, or a grocery store.
In this paper, we introduce a new dataset HarveyNER with fne-grained locations annotated in tweets.
This dataset presents unique challenges and characterizes many complex and long location mentions in informal descriptions.
We built strong baseline models using Curriculum Learning and experimented with different heuristic curricula to better recognize diffcult location mentions. Experimental results show that the simple curricula can improve the system’s performance on hard cases and its overall performance, and outperform several other baseline systems. The dataset and the baseline models can be found at https: //github.com/brickee/HarveyNER .
Introduction
The Named Entity Recognition (NER) task aims to locate and classify textual phrases as entity mentions that belong to predefned entity categories.
Location is one of the general entity categories and has been annotated in many NER datasets, including CoNLL-2003 (Tjong Kim Sang, 2002) and OntoNotes 5.0 (Pradhan et al., 2013).
However, these datasets contain mostly coarse-grained entities such as a continent (e.g., Europe), a country (e.g., the U.S.), or a city (e.g., London).
Many downstream applications require identifying fne-grained location entities from texts, such as an apartment building (e.g., Bayou Oaks ) or a specifc store (e.g., the HEB on Montrose), in order to locate the geographic places on a map, which is vital to identify actionable information
(Khanal and Caragea, 2021).
For example, in Figure 1, a food disaster happened in the Houston area and then someone tweeted the shortage of necessities in two locations.
If a disaster response system can detect the disaster-related tweets, identify the two location mentions from the text, and link them to location entities on the map, necessary help can be directly delivered to the people living in disaster-affected places.
Accurately identifying the fne-grained location mentions plays a critical role in such a system.
Several previous works have attempted to create crisis-related datasets with fne-grained location mentions, either automatically (Middleton et al., 2013) or by manual (Khanal et al., 2021) annotations.
However, these prior datasets contain many incomplete descriptions of locations that otherwise can be precisely projected to a map with certain geo-coordinates.
For example, "the corner of
Richey St and W Harris Ave in Pasadena" is an intersection of two roads
and we annotate it as a Point on the map, but previous work regard it as two
Road mentions "Richey St" and "W Harris Ave in Pasadena" , and such incomplete location mentions will affect uses in many applications.
We introduce HarveyNER, the frst dataset that annotates such coordinate-oriented location mentions.
We use the Harvey disaster in Houston as an example to demonstrate how to annotate such location mentions. Specifcally, we consider tweets about
Hurricane Harvey affecting the Houston metropolitan area in 2017 and annotate mentions of locations that exist in this city. Compared with the location mentions in prior NER datasets, HarveyNER focuses on the location mentions that can link to specifc sites on a map. We carefully constructed the annotation guidelines and trained annotators to obtain high-quality annotations. We also built strong baselines over the dataset for future reference.
The unique characteristics of HarveyNER present challenges for NER systems.
First, many location entities in this dataset are long and complex to precisely describe a place.
E.g., the above example of a Point entity contains up to 11 words, and it could be wrongly recognized as two road entities by a NER system.
Second, as an instant social medium, tweets contain many informal contents, local conventions, and even grammatical errors, which create many out-of-vocabulary (OOV) words that cannot be found in pretrained word embedding such as Glove (Pennington et al., 2014) or BERT (Devlin et al., 2019).
To improve the performance on these hard location mentions and build strong baselines for the HarveyNER dataset, we adopt Curriculum Learning (CL) (Bengio et al., 2009) to better learn diffculty samples by ordering examples during training based on their diffculty.
We design two heuristic curricula based on entity length and word complexity considering that many long and complex entities in HarveyNER are naturally diffcult (as shown in Figure 3, the performance of baseline systems is worse on these hard cases).
We further assume that the diffculty to learn may not only depend on the inherent diffculty of a type of case but also depend on how commonly seen or how well represented such cases are in the dataset.
Therefore, we propose a novel curriculum with a diffculty scoring function that comprehensively considers the two heuristic diffculty metrics as instance frequencies.
Empirical results show that all of the curricula can outperform several other baseline systems, and our novel curriculum performs the best.
We also fnd that different NER-based systems beneft from different curriculum scheduling strategies. In our experiments, the normal curriculum (training with easier samples frst) is suitable for training the neural network-based model NCRF++ without pretrained language models, while the anticurriculum (training with harder samples frst) facilitates fne-tuning of the pretrained language model BERT.
Related Work
NER research has a long history and many NER datasets have been created with certain pre-defned entity categories.
General domain datasets such as CoNLL-2003 (Tjong Kim Sang, 2002) and
OntoNotes 5.0 (Pradhan et al., 2013) attend to certain common entity types including Location .
Location mentions in these datasets are mostly coarsegrained, e.g., the U.S. (a country) or London (a city).
Li and Sun (2014); Ji et al.
(2016) focus on identifying fne-grained points-of-interest for location-based services, and their dataset is automatically constructed by mapping location inventory to tweets.
Khanal and Caragea (2021); Khanal et al.
(2021) try to identify crisis-related location mentions, but their dataset contains incomplete location mentions and is of limited use for a disaster response system.
In contrast, our dataset
HarveyNER emphasizes fne-grained locations that can map to coordinates on a map.
Recent approaches (Yang and Zhang, 2018; Li et al., 2020; Chen et al., 2021) use Neural Network models like BiLSTM-CNN-CRF (Ma and Hovy, 2016) and contextual embeddings like BERT (Devlin et al., 2019), and have greatly improved the
NER performance.
However, none of these approaches consider the diffculty of different NER cases in their model training.
Bengio et al.
(2009) pointed out that using a curriculum strategy enables the model to learn from easy examples to complex ones and leads to generalization improvement.
Many Natural Language Processing tasks such as machine translation (Platanios et al., 2019;
Liu et al., 2020; Zhang et al., 2021), natural language understanding (Xu et al., 2020), text generation (Liu et al., 2018, 2021) and dialogue systems (Su et al., 2021) beneft from such curriculum learning strategies.
Considering that HarveyNER contains many long and complex location mentions, we design corresponding curricula to learn them.
Data Split
Train Valid
Test
Total # of Tweets 3,967 1,301 1,303 6,
Tweets w/ Entity 1,
[Table]
[Table]
Tweets w/o Entity
[Table]
[Table]
Mentions
[Table]
The HarveyNER Dataset 3. Data Preparation
Data Collection We used the Twitter PowerTrack
API to retrieve the tweets posted during the time of peak disruption caused by Hurricane Harvey in the Houston area, specifcally from 5:00 a.m. August 25 to 4:59 a.m.
August 31, 2017.
In total, we collected 1,121,363 tweets excluding retweets and replies.
Data Cleaning We applied several strategies to flter out irrelevant tweets.
First, we only keep the tweets that are related to the Houston area, i.e., the geo-coordinates of the tweets or the authors’ profle locations are within the bounding of Houston.
Second, we applied our weakly-supervised event detection system (Yao et al., 2020) to identify tweets on disaster-related topics; these tweets are likely to be related to Hurricane Harvey during the specifed period.
We also manually fltered out remaining irrelevant tweets (such as non-English and repeated tweets) during the annotation process.
In total, 6,571 tweets were selected for this study, as shown in Table 1.
3. Location Entity Annotation
Location Types HarveyNER focuses on the coordinate-oriented locations so we mainly annotate Point that can be precisely pinned to a map and Area that occupies a small polygon of a map.
Considering that some disasters can affect line-like objects (e.g., a food can affect the neighbors of a whole river), we also include Road and River types.
• Point : denote an exact location that a geocoordinate can be assigned.
E.g., a uniquely named building, intersections of roads or rivers; • Area : denote geographical entities such as city [Table]
[Table] [Table]
• Road : denote a road or a section of a road; • River : denote a river or a section of a river.
Annotation Quality To train the annotators to well annotate the fne-grained location mentions, especially to distinguish the Point locations, we conduct rounds of initial annotation exercises and receptively update annotation guidelines to reduce ambiguity and subjectivity.
The detailed guidelines can be found in Appendix A.1.
We trained three annotators and calculated their Inter-Annotator Agreement (IAA) based onrandomly selected tweets they all annotated.
We pairwise calculate the Cohen’s kappa ( κ ) scores based on the token-level annotations from each pair of annotators. As suggested by Brandsen et al.
(2020), we report two scores: one calculated using all the token annotations and one only using the annotated tokens that exclude non-entity tokens.
As shown in Table 2, we observe a high average κ score of 83.64% for all tokens and an average κ score of 63.04% for annotated tokens only.
After that, the three annotators annotated the remaining tweets independently.
3. Dataset Analysis
We randomly split the annotated tweets into training, validation, and test sets for experiments with a ratio of 6:2:2.
Table 1 shows some basic data statistics.
We can see that 27.48% of the tweets contain at least one location entity mention, while the remaining tweets do not mention any location.
As for location types, Point and Area are two dominant entity types covering 38.36% and 44.66% of entity mentions respectively, while Road and River only make up 10.22% and 6.76% of entity mentions respectively.
Comparisons with CoNLL-2003 We compare HarveyNER with CoNLL-2003, a general NER dataset annotated with coarse-grained locations in news articles, and Table 3 shows the comparisons in several aspects.
First of all, entities in HarveyNER are longer [Table]
Datasets
[Table] [Figure] [Table]
[Figure] [Table]
[Figure] l [Figure]
c [Figure] i o n m e n t i o n s [Figure]
on average at both word level and character level. We observed that many location mentions of the type Point and Area are especially long to precisely describe a site or area on a map. We manually analyzed long entities in the validation set and observed that many location mentions are complex noun phrases with a conjunction or a prepositional phrase attachment. We noted down two commonly seen conjunctions and seven commonly seen prepositions, and table 4 shows location examples for each.
We calculate the percentage of complex entity mentions with one of these words, and HarveyNER has 11.8% of entity mentions fall in this category while CoNLL-2003 has few such entity mentions (0.19%).
Figure 2 shows the number of tweets with each of the words. Second, considering that the language used in tweets is informal and contains many abbreviations
[Table]
[Table] [Table]
and even grammatical errors, we calculate the outof-vocabulary (OOV) rates (in Table 3) for both datasets by counting words that are absent from the pretrained Glove 1 (Pennington et al., 2014) word lists.
We can see that the HarveyNER has a much higher OOV rate than CoNLL-2003.
The high OOV rate might degrade the performance of NER systems relying on pretrained word embeddings.
In addition, we compare the average sentence length between the two datasets. To our surprise, HarveyNER has overall longer sentences, based on both word counts and character counts.
This is counter-intuitive since the tweet content is strictly constrained to be no more than 140 characters each.
One possible reason is that short tweets are less likely to provide useful event information and have been fltered out by the event detection system (Yao et al., 2020) we used.
Lastly, we measure the density of annotated location entities in the two datasets by calculating the average number of location entity mentions per sentence and calculating the percentage of entity words out of all the words in a sentence.
We also calculate these two measures for the subset of sentences that contain at least one location mention (non-empty sentences).
The last section of Tableshows the results.
We can see that the two datasets are similar over these annotation density measures.
Curriculum Arrangement
In consideration of the characteristic diffculties of
HarveyNER, we employ curriculum arrangements to help learn these hard cases.
We follow the curriculum designing approach introduced by Bengio et al.
(2009), which mainly requires specifying two functions:
• Diffculty Scoring Function :
Given an input sample x i , this function map it to a numerical score, d ( x i ) ∈ R .
The score is used to represent the diffculty level of the corresponding sample.
Usually, the higher the score, the more diffcult the sample is. • Pacing Function :
The pacing function p ( t ) ∈ (0 , 1] specifes the input training data size at time or step t .
Normally we use p ( t ) the lowest diffculty-scored samples for training at time t , but in the anti-curriculum setting, we use p ( t ) the highest diffculty-scored samples.
Given such a subset of the dataset containing the easiest or hardest ones, we sample training batches uniformly from it for training.
The curriculum learning procedure using the two functions is described in Algorithm 1.
4. Three Diffculty Scoring Functions
We frst design two dataset-specifc heuristic curricula, based on maximum entity length and entity complexity 2 , inspired by the dataset analysis in
Section 3.3.
Then, we introduce a new metric that integrates the two heuristic metrics.
Maximum Entity Length (Max) :
As mentioned previously, our HarveyNER dataset has longer entity mentions on average compared to CoNLL-2003, and this brings many long and diffcult entities that are hard to identify.
Intuitively, we can design a corresponding curriculum based on such an entity-level diffculty.
Specifcally, given an input tweet sample x i that contains n words, x i = { w 1 , w 2 , . . . , w n } , and k ( k ≥ 0 ) entities, { E 1 , E 2 , . . .
, E k } .
| E j | represents the length of j -th entity, specifcally, the number of words in the j -th entity.
Now, we can assign each tweet sample a score using the maximum length 3 of its entities:
[Equation]
where, L i = {| E 1 | , | E 2 | , . . . | E k |} , the set of entity lengths for the i-th sample x i .
With this scoring function, we need to pay attention to the tweets with zero entity mention ( 72.52% of tweets in HarveyNER as shown in Table 1) since their diffculty scores will all be 0 .
In this case, the algorithm will provide all these tweets in one step to the curriculum, which will mislead the model to a local minimum and learn that no entity exists in the data.
We propose a remedy to this issue by randomly feeding the empty tweet samples.
Specifically, when we order our dataset by the diffculty scores, we randomly intersperse those no-entity tweet samples among the ordered samples that have entities.
Complex Entity Rate (Complex) :
Corresponding to the analysis of the complex entity rate in HarveyNER, we defne another diffculty scoring function.
Specifcally, we defne the complexity of an entity c ( E ) based on whether the entity contains one of the conjunction words or preposition words we identifed and which word the entity contains.
Heuristically, we assign a weight greater than 1 to these words, specifcally, we assign a weight of 3 to each conjunction word and a weight of 2 4 to each preposition word to refect our intuition that entities with conjunctions can be more diffcult cases.
If an entity E contains more than one “complexity” indicator, we choose the one with the highest weight.
For example, the entity example E "the corner of Richey St and W Harris Ave in Pasadena" contains the conjunction "and" and the preposition "in" , we deem the complexity of this entity c ( E ) is 3 instead of 2.
Then, one tweet sample x i can have multiple entities with different complexities
C i = {
c ( E 1 ) , c ( E 2 ) , . . .
, c ( E k ) } , we assign the maximum entity complexity value as the complexity value of the tweet sample, i.e.,
[Equation]
However, if none of the entities in a tweet sample contain complexity indicators, this scoring function will assign 0 as the diffculty score for the tweet sample.
A similar issue has been discussed for the previous scoring function, and we use the same remedy as well and randomly interspersed these zero-scored samples among the other ordered samples.
Commonness of Diffculty (Commonness) :
We further propose a novel diffculty metric that considers both of the prior metrics as well as their support.
We exploit the assumption that the diffculty to learn may not only depend on the inherent diffculty of a type of case but also depend on how commonly seen or how well represented such cases are in the dataset.
Specifcally, to assign a diffculty score for a tweet sample, we frst count the number of training samples that have the same diffculty score as the sample x i according to one of the prior two metrics and then divide it by the total number of tweet instances N .
Then, we take the reciprocal of it and get f metric . [Equation]
Where, d metric are the diffculty metrics d max or d complex .
Hence, the larger support of a diffculty level based on one of the two prior heuristics, the lower the f metric value is.
We then normalize this value to the range of [0 , 1] .
[Equation]
Then, we integrate f max and f complex , and take the L2 -norm to generate the fnal diffculty score.
[Equation]
As a result, the more common a sample is concerning its length or complexity, the smaller the L2 -norm value is, which indicates a lower diffculty based on the new metric.
In addition, we add a hyperparameter λ to balance the infuence of the two metrics.
Similar to the previous single diffculty-based curricula, the commonness diffculty score is zero when a tweet sample has no entity.
We adopt the same remedy and randomly intersperse those noentity tweet samples among the ordered ones that contain entities.
4. Pacing Function
We use the root-based pacing function introduced by Platanios et al.
in all our experiments.
[Equation]
Here p (0) defnes the proportion of samples we feed our model at the very beginning; T is the number of epochs that we apply curriculum learning to our model.
Experiments
In our experiments, we use two state-of-the-art
NER systems as base models and evaluate their performance on the HarveyNER dataset.
Then, we test the effectiveness of the designed curricula by applying them to train the base models.
5. Baselines
NCRF++ (Yang and Zhang, 2018) is an opensource Neural Sequence Labelling Toolkit.
We use the BiLSTM-CNN-CRF structure as a base model.
BERT (Devlin et al., 2019), a pretrained language model based on Transformer (Vaswani et al., 2017), has signifcantly improved many NLP tasks including NER.
We fne-tune the base-uncased version of BERT with the BiLSTM-CRF structrue for experiments.
5. Training Setup
For the NCRF++ model, we use the tweet-based version Glove as word embeddings and keep all the other hyper-parameters as default.
For the BERT model, we set learning rate as 5e-5 and set batch size as 32 .
As for the λ hyperparameter in Eq.
(5), we used grid search and set it 1 and 0 . 6 for the NCRF++ model and the BERT model respectively.
We train all the NCRF++ models for 100 epochs and train all the BERT models for 50 epochs.
For fair comparisons, we keep all the training parameters the same when conducting curriculum learning.
For the NCRF++ model, we use the normal curriculum setting and feed easier cases [Table] frst, while for the BERT model, we use the anticurriculum setting (more explanations provided in Section 5.5).
Note that we train all the experiments fve times using different random seeds to alleviate random turbulence.
5. Results
Table 5 shows the experimental results.
We can see that the BERT base model outperforms the NCRF++ base model consistently across the four location categories on this dataset.
Curriculum learning yields further performance gains for both the BERT model and the NCRF++ model, this is true for all the curricula paired with both models except the Max curriculum when applied to the BERT model, where the average performance almost stays the same. Among the three curricula,
he
Commonness curriculum achieves the best performance for both models.
Conducting curriculum learning have unequal impacts on the four location categories.
When using the NCRF++ model, curriculum learning yields a small performance improvement on Point and clear improvements on Area and River , while using the BERT model, curriculum learning yields a relatively larger improvement on Point and a clear improvement on River as well.
Note that the best-performed BERT model only achieves a micro-average F1-score of 69.67% on this dataset, which is still much lower than recently published BERT-base performance on CoNLL2003 (e.g., 92.4% as reported in (Devlin et al., 2019)).)
Meanwhile, we acknowledge that recognizing fne-grained locations is a trickier task than recognizing coarse-grained locations or many other general types of entities, even for humans as shown by the imperfect inter-annotator agreements.
How are the Diffcult Samples Learned?
In order to understand if the models have better learned the diffcult samples after applying curriculum learning, we divide the test set into “easy” and “hard” subsets based on either entity length or entity complexity and report experimental results on each subset.
In this analysis, we only consider tweet samples in the test set that contains at least one entity mention since the groupings are determined by the characteristics of the entities.
In the frst grouping, we divide tweet samples into “short” and “long” groups depending on if the maximum entity length (number of words) in a tweet sample is greater than a threshold, four words in particular ( < = 4 v.s. > 4 ).
In the second grouping, we divide tweet samples into “simple” and “complex” groups by checking if a tweet sample contains a complex entity mention 6 .
Across all the experimental settings, we report the results on two subsets separately under each grouping (Figure 3). We can see that curriculum learning indeed yields noticeable improvements in identifying those hard cases, while the improvements vary when adopting different curricula. Meanwhile, curriculum learning also achieves mild improvements on identifying the remaining relatively easy cases.
5. Curriculum v.s Anti-curriculum
We fnd that applying different curriculum settings (normal curriculum that exposes the easiest examples frst or anti-curriculum that exposes the most diffcult examples frst) results in a large performance difference between the NCRF++ model and the BERT model.
As shown in Figure 4, for the
NCRF++ model without pretrained language mod- [Figure] [Figure] els, the normal curriculum setting yields signifcantly better average F-1 scores across all the three curriculum scoring functions in comparison with the anti-curriculum setting.
However, for the pretrained language model BERT, the results are the opposite; using anti-curriculum learning consistently yields better performance than using normal curriculum learning.
One possible explanation is that the volatile gradients resulting from using anti-curriculum learning can lead to better local minima for a wellpretrained model.
Specifcally, the anti-curriculum learning will feed those “hard” samples to the model frst [Figure] and the gradients from those long-tailed hard cases will cause relatively large fuctuations compared to those from easy instances.
BERT is a pretrained language model and the pretrained parameters might constrain the model to some local regions.
The fuctuations produced by the “hard” samples from the anti-curriculum learning can enable the BERT model to reach other better local minima regions [Figure]
Conclusion
We introduce a fne-grained location recognition dataset, HarveyNER, to enable many downstream applications such as building real-time disaster response systems.
This dataset contains many long and complex location mentions that feature interesting internal syntactic and semantic structures and the state-of-the-art NER systems are unable to fully recognize these hard cases.
Considering the clear characteristics of diffcult cases in this dataset, we experimented with two heuristic curriculum learning strategies and a novel commonness-based curriculum strategy to better recognize the diffcult location mentions.
Empirical results demonstrate the effectiveness of the curricula, which serve as strong baseline results in this dataset.
Future work may consider incorporating external knowledge or innovations on system architectures to better identify fne-grained location mentions. Acknowledgements
We gratefully acknowledge support from National Science Foundation via the awards IISand IIS-1759537. In addition, this research is par- tially supported by 2021 PRISE Grant 02-243712.
We would also like to thank the anonymous reviewers for their valuable feedback and input.
A Appendix A. Annotation Guidelines
Location types can be "Area", "Point", "Road", and "River."
– “Area” refers to all the named entities of cities, neighborhoods, super neighborhoods, geographic divisions etc.
– “Point” refers to a location that is a building, a landmark, an intersection of two roads, an intersection of a river with a lake/reservoir/ocean, or a specifc address.
– "Road" refers to a road/avenue/street or a section of a road/avenue/street when the tweet does not provide an exact location on that road.
– "River" refers to a river or a section of a river when the tweet does not imply there is an intersection between the river and other places.
A section of a road/river between two detailed/precise locations should be considered as a point.
However, if the distance between the two points is very large, it might be considered as a stretch of a road/river.
A road passing through a small area can be designated as a point.
A road intersecting a very large area cannot be a point and must be denoted as a stretch of a road.
In some peculiar cases, the road takes a small detour and tangentially brushes off an area – in such specifc cases, roads can be annotated as a point.
The following locations, Lake Houston , Barker Reservoir , and Addick’s Reservoir , are annotated as areas due to their signifcant size while all other lakes/reservoirs are considered as points.
Ignore generic company/franchise names like HEB, Kroger etc.
unless it is accompanied with a precise location, for example, HEB at Kirkwood Drive .
However, non-franchised small businesses with only one unique location are considered as a point.
Ignore any locations in the Twitter username, like @HoustonABC.
However, if the @ does not refer to a Twitter account name,
please recognize the location.
For example, I am
XXX High School , “XXX High School” will be considered as a point.
For abbreviations or vague location names, always look up the tweet’s context (or even other tweets’ context) to decide if it is a location or not.
We will use search engine if it is necessary.
– Eg: Coke Ck ; Here, "Ck" refers to a creek.
This is understood when multiple such tweets point towards a creek.
Similarly, for names that can refer to different or multiple locations, like “Bellaire” can either refer to Bellaire St or the Bellaire area, we always look up the tweet’s context to decide their location types.
We annotate the mentioned location as the complete set of phrases that describes the detail of the location including the core noun and all defning relative clauses.
If a tweet mentioned the same location multiple times, they will be annotated as multiple location mentions.
Ignore the location that only contains “Houston”, “Harris County”, or “Texas” • 11.
Ignore any tweet outside Houston (like London, Dallas, etc) and all non-English tweets.
We keep the exact words in tweet context as the location name after extracting the entities.