Detecting Entities in the Astrophysics Literature: A Comparison of Word-based and Span-based Entity Recognition Methods (2022)

Abstract: Information Extraction from scientific literature can be challenging due to the highly specialised nature of such text. We describe our entity recognition methods developed as part of the DEAL (Detecting Entities in the Astrophysics Literature) shared task. The aim of the task is to build a system that can identify Named Entities in a dataset composed by scholarly articles from astrophysics literature. We planned our participation such that it enables us to conduct an empirical comparison between word-based tagging and span-based classification methods. When evaluated on two hidden test sets provided by the organizer, our best-performing submission achieved F1 scores of 0.8307 (validation phase) and 0.7990 (testing phase).

## introduction a large body of scientific literature is published in different domains, making it difficult for researchers in their respective fields to find information or keep up-to-date. automatic information extraction, in particular named entity recognition (ner), is one of the core methods from the nlp community to assist researchers. it finds mentions of entities of interest in a given text, such as in medicine @xcite , astronomy @xcite , geology @xcite , chemistry @xcite , materials @xcite or even finance @xcite . astrophysics scientific literature has its own unique properties, raising some specific challenges for handling of the text. for example, it contains ambiguous names chosen based on the scientists names responsible for a mission or a facility name. while it is not the first time that ner for astrophysics has been studied @xcite , it is rather under-studied. deal (detecting entities in the astrophysics literature) shared task introduced as part of the aacl-ijcnlp 2022 conference has challenged the community with the release of an annotated dataset to pave the way for advancing information extraction methods in this field. we investigate two different ner methods, word-based tagging and span-based classification, on astrophysics data provided by the organisers of the deal shared task. in particular, we examine their effectiveness in extracting 31 different types of entities of interest, such as computingfacility and wavelength, and report our experimental results, which led our team to an overall third ranking among 12 teams. ## related work information extraction, and in particular ner, on scientific literature has attracted substantial research @xcite @xcite . ner refers to the task of identifying mentions of different types of entities in free-text. types of entities of interest depend on the domain of the text; for example disease names in biomedical text @xcite or numbers in finance @xcite . methods to recognise such entities should also handle different types of the text, including both formal and informal text , such as social media posts @xcite . for astronomy, there are two existing annotated datasets. @xcite created the astronomy bootstrapping corpus (abc) which is a corpus of 209 annotated article abstracts in english from the radio astronomical papers from the nasa astrophysics data system archive. it also includes a further unannotated 778 abstracts used for bootstrapping. hachey et al. experimented with active learning for ner on a then novel domain of astronomy. murphy et al. (2006) annotated 200, 000 words of text from astronomy articles published on arxiv. the dataset is manually annotated with approximately 40 entity types of such as galaxy, ... ... ... were performed on the cosmos shared memory system at damtp university of cambridge operated on behalf o o o o b-fac i-fac i-fac i-fac i-fac i-fac b-org i-org i-org o o o words ... word-based tagger star and particle. murphy et al. also propose a maximum entropy-based ner method on this dataset, reporting an f 1 score of approximately 87%. @xcite created astrobert, a language model for astronomical text provided by the nasa astrophysics data system (ads). 1 it is pre-trained on 395, 499 english documents from ads, and is benchmarked for ner, showing improvements over bert @xcite . ## method we start from splitting a long document into sentences s 1 s 2 • • • s d using a heuristic rule. that is, every full stop is used to mark the end of a sentence if the current sentence consists of more than 10 words. given a sentence s i , two neural entity recognition models are employed to recognize all entity mentions. they use the same encoder (i.e., transformers @xcite ) and differ in their classifiers, a high-level illustration of these two models is shown in figure 1 . we describe the encoder in section 3.1 and two classifiers-word-based tagger and span-based-in section 3.2 and section 3.3, respectively. ## dataset and experimental setup the deal shared task organisers released one labelled training set (1, 753 documents) and one labelled development set (20 documents), on which participants can develop their ner systems. two holdout labelled sets (validation and testing) were used to score submissions, and the labels of these holdout sets were not available to participants until the official scoring period ends. the dataset has 31 entity types, with entity 'organization' comprising 16.3% as highest and entity 'textgarbage', lowest with 0.1%. a descriptive statistics of the dataset is shown in we train our models on the first 1, 578 documents of the training set, and the remaining 175 documents are used for hyper-parameter tuning and best checkpoint selection. we use the microaverage string match f 1 score to evaluate the effectiveness of the models. the model which is most effective on these 175 documents is finally eval-uated on the development, validation, and testing sets. we repeat all experiments three times using different random seeds, and the mean scores and standard deviations are reported. in addition to the f 1 score, we report precision (p), recall (r) and matthew's correlation coefficient (mcc) @xcite metrics, calculated using the scripts provided by the shared task organizers. ## results and discussion we compare word-based and span-based entity recognition models using both roberta-base and roberta-large models. results in task-adaptive pre-training does not guarantee better performance some studies have shown that pre-trained language models are more effective when pre-training data is similar to downstream task data @xcite . task-adaptive pretraining @xcite -continue pre-training on the unlabeled training set for a given task-is a cheap adaptation technique that aims to reduce the disparities between models pre-trained on generic data and domain-specific task data. we continue pre-training roberta-large on the deal training set using masked language modeling. the total number of optimization steps is 3,000 (≈ 100 epochs), and we save checkpoints every 600 steps. during the task-adaptive pre-training stage, we observe both the training and development losses keep decreasing, however, the resulting task-adaptive pre-trained checkpoints seem to be very unstable and do not guarantee improved effectiveness (figure 2 ). note that @xcite reported improved effectiveness via taskadaptive pre-training roberta-base, whereas we use roberta-large. we conjecture the observed instability may be attributed to the optimization difficulties discussed by @xcite , when continue training large size models on small data. errors due to over-segmentation one problem we observe is that many domain-specific terminologies are split into multiple sub-tokens and then taken as input to the encoder. taking the sentence in figure 1 as an example, since the term 'cos-mos' is not in the vocabulary associated with the roberta pre-trained models, it is split into four 1 (720) 2 ( 334 ) 3 (166) 4 (92) 5 (46) >= 6 (55) # sub-tokens per word 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 word-level f1 method span-based word-based figure 3: word-level f 1 scores calculated on words that belong to entity names. number in brackets are the number of corresponding words. development validation testing orig 0.8490 0.8267 0.8034 innermost 0.8533 0.8293 0.8033 outermost 0.8491 0.8298 0.8065 we measure the impact of over-segmentation by calculating word-level f 1 score on tokens that are part of entity names and grouping words by the number of sub-tokens they are split into. figure 3 shows that both word-based and span-based methods suffer from over-segmentation, especially when words are split into three or more sub-tokens. errors due to nested predictions span-based methods were originally designed to tackle nested ner @xcite @xcite , where two entity names may nest each other. for example, the span-based method may predict both 'cosmos shared memory system' and 'cosmos shared memory system at damtp' as computingfacility entities. however, the annotations of deal shared task do not allow nested structure. we find that span-based method benefit from post-processing via resolving these nested predictions. results in ## conclusions we reported our experiments on extracting mentions of 31 different types of entities from astro-physics scientific literature. two different sets of methods based on words and spans were compared. results show that span-based method using roberta-large pre-trained models outperforms the widely used word-based sequence tagging method. potential research directions include building better span representations with the help of external knowledge base; enhancing pre-trained models with domain-specific vocabulary; and, combing the strengths of word-based and span-based models.