Astro-m T 5: Entity Extraction from Astrophysics Literature using m T 5 Language Model (2022)

Abstract: Scientific research requires reading and extracting relevant information from existing scientific literature in an effective way. To gain insights over a collection of such scientific documents, extraction of entities and recognizing their types is considered to be one of the important tasks. Numerous studies have been conducted in this area of research. In our study, we introduce a framework for entity recognition and identification of NASA astrophysics dataset, which was published as a part of the DEAL SharedTask. We use a pre-trained multilingual model, based on a natural language processing framework for the given sequence labeling tasks. Experiments show that our model, Astro-mT5, out-performs the existing baseline in astrophysics related information extraction.

## introduction extracting information about entities and their relationships from unstructured text is an important area in natural language processing (nlp). fast evolution of many scientific disciplines has led to a continuous influx of a large number of research papers into the publication repositories (e.g., arxiv foot_1 , anthology foot_2 and biorxiv foot_3 ). literature study is an important step in any scientific study. till now, this has been limited to human effort i.e., the amount of previous literature that an individual is exposed to is limited to human capabilities. this may lead to a few fundamental problems, such as, the researcher's inability to find out relevant previous works and identify suitable baselines for performance comparison. it poses a significant problem due to the limitation of human abilities, unless a framework could be designed to obtain the literature corpus by using machine learning methods. to overcome this problem, @xcite al. (2018); jain et al. (2020); hou et al. (2021 @xcite . self-supervised pre-training allows these plms to learn highly accurate linguistic, semantic, and factual information from a significant quantity of unlabeled data @xcite . while tremendous progress has been made in the field of ai, the area of astrophysics has been rarely explored as an area of application by ai researchers. the current search engine of nasa astrophysics data system (ads) shows poor performance on information retrieval (ir) tasks due to the absence of suitable kg @xcite . @xcite have recently developed astrobert, a language model pre-trained on astrophysics literature, in order to apply it to downstream tasks of nlp in the astrophysics domain. the first edition of the shared task, named detecting entities from astrophysics literature (deal) took place in 2022, for building a system that is capable of extracting fine-grained entities of different categories such as celestialobjectregion, celestialregion, instrument @xcite . we participated in deal sharedtask 2022 and proposed a neural architecture based model to identify the required entities from a collection of astrophysics articles. our proposed model namely, entity extraction from astrophysics literature using mt5 language model (astro-mt5), seeks to devise a transfer learning strategy by fine-tuning the mt5 @xcite model. furthermore, we apply conditional random field (crf) decoder to implement the entity extraction task. experimental results show that the proposed framework achieves state-of-the-art (sota) performance on this task. ## related works recently, researchers have explored many directions for applying information extraction from the unstructured text of scientific articles written in english. adding token-level classifiers or crfs above the sentence encoders is a popular strategy for ner task @xcite @xcite . @xcite empirically showed that using pre-trained word embeddings such as glove along with cnnbased @xcite and lstm-based models @xcite produced better results on the ner task. recent releases of transformer based plms such as bert @xcite , scibert @xcite , t5 @xcite , roberta @xcite , big-bird @xcite , albert @xcite , @xcite , and longformer @xcite showed a significant performance improvement in many downstream tasks in nlp such as ner and re. additionally, @xcite developed an easy-to-use interface namely flair, that allows users to fine-tune any word embedding and any plms to produce improved results on the ner task. there has been a recent surge in proposing multilingual pre-trained language models such as mbert @xcite , mbart @xcite , xlm-r (conneau et al., 2019), mt5 for achieving sota results in many downstream ie tasks in nlp. ## system description given a sentence from the astrophysics documents, we follow a sequence labeling approach for the fine-grained entity extraction task. formally, given a sentence (word sequence) @xmath0, w 2 ...w n ), the objective is to learn a function f θ (parameterized by θ) that maps an observed sequence of embedded vectors to a sequence of labels f θ : (w 1 , . . . , w n ) → (y 1 , . . . , y n ), where each w @xmath1, and each y @xmath2, i, o} stands for a label, which indicates if it is the beginning, continuation or the end of a text span (in the context of our work -predicted entity category). given a set of examples of such @xmath3, y)} sequence pairs, the parameters θ of a sequence classification models are learned by optimizing the cross-entropy loss. in our work we use flair foot_4 , a neural framework, proposed by @xcite . in this framework, we employ mt5 as a base pre-trained language model, which gets fine-tuned on our downstream entity extraction sequence labeling task. say, the internal output representation produced at the fine-tuning stage is x @xmath4. then, we pass it to the crf decoder layer and get the probability sequence over the possible sequence labels y by using the eqns. 1 and 2. here, w, @xmath5, which are trained during end-to-end training of our model. the overall architecture diagram of our proposed model, astro-mt5, has been shown in fig. 1. 4 experimental setup 4.1 data description throughout our experiments, we have used the astrophysics dataset released for the deal shared-task. the dataset mainly consists of the full texts and the acknowledgement sections of a collection of astrophysics articles. the total number of entity categories used for the sequence classification task is 32 and the frequency of the categories (ner tags) has been depicted in fig. 2. the dataset statistics has been shown in ## conclusion this study discusses a transformer-based deep neural architecture to identify named entities from an astrophysics literature dataset provided by the deal sharedtask team. our model, astro-mt5, has achieved f1-score of 80.58% and mcc of 89.54% on the test data, which remarkably outperforms previously reported models and all other competing models submitted in the deal shared-task. our future work will include more research on fine-grained ner and boundary detection in context of astrophysics to support a wide range of practical applications. we can plan to enhance our framework by introducing fine-grained ner in place of coarse-grained ner to handle a named entity with various types. we can also investigate data-driven factored modeling approaches to handle the class imbalancing problem.