A Majority Voting Strategy of a S ci BERT -based Ensemble Models for Detecting Entities in the Astrophysics Literature (Shared Task) (2022)

Abstract: Detecting Entities in the Astrophysics Literature (DEAL) is a proposed shared task in the scope of the first Workshop on Information Extraction from Scientific Publications (WIESP) at AACL-IJCNLP 2022. It aims to propose systems identifying astrophysical named entities. This article presents our system based on a majority voting strategy of an ensemble composed of multiple SciBERT models. The system we propose is ranked second and outperforms the baseline provided by the organisers by achieving an F1 score of 0.7993 and a Matthews Correlation Coefficient (MCC) score of 0.8978 in the testing phase.

## introduction astronomy and astrophysics consist of observing and studying various cosmic phenomena such as tidal disruption events, gamma-ray bursts, and many other messengers such as neutrinos and gravitational waves @xcite . missions and observations performed by astronomical facilities worldwide significantly increase the number of astrophysics papers. most published papers are freely available and accessible through the astrophysics data system (ads 1 ), where researchers can search and access more than 15 million records covering astronomy, astrophysics, and general physics publications. however, some domain keywords can be easily confused when searching for articles in the literature. for instance, "planck" can refer to the person, the mission, the constant, or several institutions. one approach for this word sense disambiguation problem would be automatically recognised entities. named entity recognition (ner) consists of recognising mentions of entities from text belonging to predefined semantic types: person, location or organisation @xcite . it is, therefore, an es-1 https://ui.adsabs.harvard.edu/ sential technique to extract relevant information from unstructured human-written data. detecting entities in the astrophysics literature (deal) is a shared task that tackles the challenge of developing a system that identifies named entities in the astrophysics literature @xcite . the shared task was organised in two stages: validation and test. evaluation metrics used were both the conll-2000 shared task seqeval foot_0 f1-score at the entity level and scikit-learn's matthews correlation coefficient (mcc 3 ) method at the token level. organisers provided the ner system's baseline (see as part of this shared task, we used and explored an ensemble of contextual pre-trained language models (pltms) for ner purposes. the paper is organised as follows: section 2 briefly presents existing methods and approaches for named entity recognition in astrophysics and other scientific domain. section 3 provides information about the corpus. section 4 describes our system as well as the experimental setup. section 5 presents our results. ## the corpus the shared task corpus comprises full-text fragments and acknowledgements sections extracted from ads papers. three sets of corpus were accessible for participants 4 : training, development and testing sets. some statistics of the corpora are provided in the annotation guide comprises 31 named entities and covers the entities of interest, such as astronomical facilities, celestial objects, coordinates, formulae or observational techniques. detailed tags list is presented in ## results on official sets the official validation and test corpora results ( to determine the number of models to include in our ensemble, we progressively formed an ensemble consisting of the five models of the first performant combination (c2), then added the five models of the second performant combination (c5) and so on. we notice that the performance decreases beyond a certain number of models. our ensemble comprises the first six combinations that gave the best results during the grid search. this represents 30 models (6 combinations * 5 models / combination). a last submission in the validation phase ## conclusion this shared task aimed to tackle the challenge of detecting entities in the astrophysics literature by proposing a ner system. we exposed in this paper our approach, which first consists of identifying the different hyper-parameters combination giving the highest f1-score. to do so, we proceeded to do a grid search on our internal training and validation sets. in the second stage, we built an ensemble of classifiers based on the top 6 combinations identified during the grid search. our submissions on the official validation and test sets show that adopting a majority voting strategy of an ensemble of scibert-based classifiers gives better results than a single model approach. finally, we ranked sec-ond, achieving an f1 score of 0.7993 and an mcc coefficient of 0.8978 using an ensemble of 32 scib-ert models.