Mapping Local News Coverage: Precise location extraction in textual news content using fine-tuned BERT based language model (2020)

Abstract: Mapping local news coverage from textual content is a challenging problem that requires extracting precise location mentions from news articles. While traditional named entity taggers are able to extract geo-political entities and certain non geo-political entities, they cannot recognize precise location mentions such as addresses, streets and intersections that are required to accurately map the news article. We fine-tune a BERT-based language model for achieving high level of granularity in location extraction. We incorporate the model into an end-to-end tool that further geocodes the extracted locations for the broader objective of mapping news coverage.

## introduction a media or news desert is an uncovered geographical area that has few or no news outlets and receives little coverage. mapping locations mentioned in news articles is the primary step in identifying news deserts. a key challenge in the process is to manually peruse the corpus of news articles, identify the location mentions and assign spatial coordinates which can then be placed on a map to identify a newsroom's coverage. while conventional named entity recognition (ner) taggers such as those offered by spacy @xcite , natural language toolkit (ntlk) @xcite and stanford nlp @xcite group contain tags to identify organizations, geo-political entities (gpe) and certain non-gpe locations such as mountain ranges and bodies of water from text, they are not able extract precise location mentions such as addresses, streets or intersections in their entirety. for example, the sentence -"the family will hold shivah from 7 to 9 p.m. thursday, oct. 13, and * equal contribution again saturday, oct. 15, at temple sinai, 5505 forbes ave., pittsburgh." passed through the stanford named entity tagger, returns "temple sinai", 'forbes ave." and 'pittsburgh" as separate location entities. however, to accurately map the location of interest, one requires the whole address -'temple sinai, 5505 forbes ave., pittsburgh.' to be returned as a single location. in recent years, there has been an advent of powerful pre-trained deep learning based language models such as google's bidirectional encoder representations from transformers (bert) @xcite , xlnet @xcite ) and openai's gpt-2 @xcite . these models can be fine-tuned for specific classification tasks in the absence of abundant training data and thus are often helpful with weak supervision @xcite . we fine-tune the bert model to extract precise location mentions in their entirety as iterated in the previous paragraph. we first build a dataset of about 10,000 sentences extracted from a corpus of 80,000 news articles spanning jan 2018 to june 2019 published in philadelphia inquirer newspaper. we use amazon mechanical turk (mturk) to label the locations of interest in the sentences and fine-tune the bert based ner tagger to classify the words in a text. finally, we incorporate geocoding of the extracted locations into the pipeline. in this work, we present an end-to-end system to extract geographic data from text. this tool could be particularly useful for newsrooms to map the coverage of their printed content helpful in identifying news deserts or by researchers and other organizations to extract precise location mention in text. our main contributions are as follows: the overall process is illustrated in figure 1 . we will first discuss our modelling approach followed by the process that we followed for geocoding the extracted locations. ## related work named entity recognition (ner) is a well studied area in the field of natural language processing (nlp). it aims to identify different types of entities such as people, organizations, nationalities and locations in text. tools trained on conventional ner models such as conditional random fields @xcite , maximum extropy (ratnaparkhi, 2016) and labeledlda @xcite have been successful in identifying common named entities. however, challenge comes when high level of granularity is of interest in extracting location entities such as specific addresses, streets or intersections. @xcite evaluated the effectiveness of existing ner tools such as stanford ner, opennlp and yahoo! placemaker on extracting locations from disaster-related tweets. @xcite presented an approach combining ner and parts-of-speech (pos) tagging to develop a set of heuristics to achieve higher granularity for location extraction in text. there has been some work around the use of end-to-end neural architecture on several sequence labelling tasks including ner @xcite and pos @xcite tagging. @xcite explored the use of external gazetteers for entity recognition with neural models showing that extracting features from a rich model of the gazetteer and then concatenating such features with the input embeddings of a neural model outperforms conventional approaches. fine-tuning pre-trained language model for domain-specific machine learning tasks has become increasingly convenient and effective. @xcite introduced biobert, a bert based biomedical language representation model for biomedical text mining and @xcite presented a fine-tuned bert model for entity and relation extraction in chinese medical text. @xcite presented advances in extractive summarization using a fine-tuned bert model. to our knowledge, no previous work has been done to fine-tune a pre-trained language-based deep learning model to achieve the level of precision and granularity in location extraction that is required for the purpose of mapping news coverage. furthermore, there does not exist an open source endto-end tool to extract and geocode precise locations from a piece of text. ## model our approach to developing a named-entity tagger for the task of precise location extraction involves fine-tuning an existing neural network on a target dataset. fine-tuning a model updates its pre-trained parameters, improving its performance on the downstream nlp task. we treat the task of named-entity tagging in a sentence as that of token classification within a sequence, assigning each word (token) in the sentence (sequence) a label. the fine-tuning process for token classification involves: (1) preparing the training dataset with expected labels for tokens within each sequence (2) loading an existing model with pre-trained weights (3) extending the model with a classification layer at the end with number of nodes equal to the number of classes in the task at hand (4) training the model on the target dataset. we will use bidirectional encoder representations from transformers (bert), developed by google ai in 2018 as the pre-trained model. bert makes use of multiple multi-head attention layers to learn bidirectional embeddings for input tokens. it is trained for masked language modeling, where a fraction of the input tokens in a given sequence are masked and the task is to predict the masked word given its context @xcite . our decision to choose bert is motivated by the fact that it is a general purpose language representation model pre-trained on millions of articles on english wikipedia and bookcorpus. given the diversity of topics present on these two training sets, we believe bert would be able to generalize well to our dataset containing news articles. bert's use of wordpiece tokenizer mitigates the out-ofvocabulary issue while tokenizing location names which are often proper nouns. with minimal architecture modification, bert can be applied to our ner task. ## mapping by deploying the fine-tuned bert model, we present a system that (1) consumes textual news content (or any other text for that matter), (2) extracts the raw text of the locations referenced and (3) returns the corresponding geocodes. the geocodes can be plotted on a map using any stan-dard mapping tool. ## conclusion and future work our work can be advanced and extended from many different perspectives. first, a more comprehensive dataset could be developed with specific tags for different location types. the bert model can be further fine-tuned using this dataset to extract different location types from the text. second, the disambigation of the extracted location can be further strengthened using contextual clues to enhance the accuracy of the geocoding process. lastly, many other state-of-art pre-pretrained language models can be fine-tuned using our dataset and a comparison can be established to select the best performing model to be used for tagging.