Overview of the First Shared Task on Detecting Entities in the Astrophysics Literature ( DEAL ) (2022)

Abstract: In this article, we describe the overview of our shared task: Detecting Entities in the Astrophysics Literature (DEAL). The DEAL shared task was part of the Workshop on Information Extraction from Scientific Publications (WIESP) in AACL-IJCNLP 2022. Information extraction from scientific publications is critical in several downstream tasks such as identification of critical entities, article summarization, citation classification, etc. The motivation of this shared task was to develop a community-wide effort for entity extraction from astrophysics literature. Automated entity extraction would help to build knowledge bases, high-quality meta-data for indexing and search, and several other use-cases of interests. Thirty-three teams registered for DEAL, twelve of them participated in the system runs, and finally four teams submitted their system descriptions. We analyze their system and performance and finally discuss the findings of DEAL.

## introduction a good amount of astrophysics research makes use of data coming from missions and facilities such as ground observatories in remote locations or space telescopes, as well as digital archives that hold large amounts of observed and simulated data. these missions and facilities are frequently named after historical figures or use some ingenious acronym which, unfortunately, can be easily confused when searching for them in the literature via simple string matching. for instance, "planck" can refer to the person, the mission, the constant, or several institutions. automatically recognizing entities such as missions or facilities would help tackle this word sense disambiguation problem. in our deal shared task, we instigate a community initiative to extract "entities of interest" from astrophysics publications. ## astrobert baseline the shared task submissions were evaluated using f-1 score and the matthews correlation coefficient (mcc) metrics. the f-1 score is a standard measure of model quality and was computed using seqeval @xcite , which uses micro-averaging and ignores the 'o' label. the mcc takes into account every value in the confusion matrix and is generally regarded as a balanced measure; it was computed using scikit-learn @xcite . the f-1 score was computed at the entity level and the mcc score was computed at the token level. using two metrics help prevent with a submission overfitting by optimizing for a single score. as a baseline, we finetuned three bert variants on the shared task. the original bert from google @xcite , scibert from allenai @xcite , and astrobert from nasa/ads @xcite . each variant was finetuned on the training dataset for 1000 epochs (~5 hours each on dual v100 nvidia gpus). ## results and analysis we report the results of the four teams that submitted their system papers in all three systems significantly outperform the as-trobert baseline, and are built on top of preexisting publicly available language models. ## conclusion and future directions all the participant systems were built on top of existing language models (i.e. general english, not tailored to a domain), and significantly beat the baseline scores. this begs the question: how would these systems performs when built on top of a language model and tokenizer tailored to astronomy? based on the competition results, the use of crfs seems especially promising. furthermore, the wide variety in the methods used by the successful participant systems indicate that the task is far from solved, and that many improvements can be made to the astrobert baseline.