Named Entity Recognition in a Very Homogenous Domain (2023)

Abstract: Machine Learning models have lower accuracy when tested on out-of-domain data. Developing models that perform well on several domains or can be quickly adapted to a new domain is an important research area. Domain, however, is a vague term, that can refer to any aspect of data such as language, genre, source and structure. We consider a very homogeneous source of data, specifically sentences from news articles from the same newspaper in English, and collect a dataset of such “in-domain” sentences annotated with named entities. We find that even in such a homogeneous domain, the performance of named entity recognition models varies significantly across news topics. Selection of diverse data, as we demonstrate, is crucial even in a seemingly homogeneous domain.

## introduction supervised neural models for named entity recognition achieve high accuracy when used in-domain. when models are evaluated or adapted @xcite @xcite for out-of-domain text, or even developed for specialized domains @xcite , the term domain generally refers to broad genres such as news, social media, or biomedical text. however, text can be (dis)similar in aspects beyond genre, such as the source of the data, its structure, or the time period. @xcite distinguish two aspects of domainthe genre and the tenor, which they describe as the participants in the discourse, their relationships and their purpose. they find that even though people consider genre to be more important for domain adaptation, tenor is important as well when selecting pre-training data. the term domain encompasses more than just broadly defined genres. online comments on different platforms can be considered different domains. so can news from different newspapers or different time periods. we show that even text from the same genre and source needs to be examined finely for topical or structural differences. we collect a dataset of news articles from the new york times and annotate it for named entities. we find that the performance of ner models varies significantly even in this dataset when it is stratified based on news topics. while entities unseen in the training data can be a factor that contributes to performance degradation, we find that structural differences in sentences and entity ambiguity are the main contributors. selecting diverse data is therefore crucial even in such "in-domain" settings. we show that even a very small number of sentences from each topic can help narrow the performance gap, and selecting random sentences rather than full documents from the full corpus, will ensure that there is a good sample of diverse sentences. ## dataset the dataset is available at https://github.com/ oagarwal/nyt-ner . here we describe the process of collecting it. ## results we finetune bert-large-cased @xcite on each topic, evaluating on all others. hyperparameter details are listed in the appendix. we report micro-f1 at the span-level averaged over three runs with different seeds. the full evaluation table is shown in the appendix for reference. here we discuss the aggregated results. since domain is used to refer to the genre of text (news in this case), we use the term sub-domain to refer to the news topics. however, we still use in-subdomain (ind) and out-of-subdomain (ood) to refer to in-subdomain and out-of-subdomain training and evaluation in the following sections. ## data selection datasets are typically collected by selecting some documents and then annotating all sentences in each document. the training set in conll'03 @xcite has 15k sentences from 946 documents, wikigold @xcite has 1.7k sentences from 145 pages, and muc-7 @xcite has 3.5k sentence from 100 articles. 3 this method of data selection is reasonable and intuitive. it also supports the development of models that utilize document-level context @xcite which can help resolve the entity types in sentences such as the above example from sports. however, most commonly used models are built at the sentence level and the selection of full documents could result in performance similar to a model trained on the same sub-domain, with all sentences in a document representing the same sub-domain and fewer chances to cover rare sub-domains (types of documents). to illustrate this, we train models for ner using conll '03. we randomly select 3,000 training sentences as this is roughly the number of sentences in each of the sub-domains. we train three models with different seeds and report the average f1 in the third column of it is therefore essential to ensure a diverse set of sentences in the training data. even a small number of sentences of each sub-domain in the training data can make a vast difference. columns 'c' and 'n' in one way to select relatively diverse sentences is by data selection at the sentence level instead of the document level. first, segment each document in a corpus into sentences and then select sentences randomly. while new future domains or those that evolve significantly will still be missed, this method would result in the selection of some representative samples of each existing domain. such explicit sentence selection has been performed for domains such as twitter where explicit documents foot_2 do not exist. @xcite selects tweets from different countries and different types of user accounts for linguistic variations and topics. they also account for temporal variation taking tweets from different years, months, weeks and days. we build models with this random sentence selection scheme. we first downsample the data such that it follows the same distribution of sub-domains as the nyt corpus with 20 years of articles. this results in 10,500 training and 4,494 development sentences with 14% arts, 11% business, 3% classified, 5% editorial, 7% foreign, 11% metropolitan, 8% national, 11% sports and 30% others. we then select 3,000 training and 1,284 development sentences randomly from this set. this is roughly the average number of sentences in each of the subdomains and seeks to eliminate the impact of the training data size. every sub-domain has at least 39 sentences in the selected training set. with models trained on this dataset, the average f1 is almost the same as in-subdomain training (col rndm). ## conclusion perform fine-grained inspection of data even when it seems that the domain is homogeneous, and perform training data selection at the sentence level rather than the document level.