SOMD 2025: Fine-tuning M odern BERT for In- and Out-of-Distribution NER and Relation Extraction of Software Mentions in Scientific Texts (2025)

Abstract: Software mentions are ubiquitous yet remains irregularly referenced among scientific texts. In this paper, we utilized the dataset and evaluation criteria defined by SoftwareMention Detection (SOMD 2025) competition to solve the problem of Named Entity Recognition (NER) and Relation Extraction (RE) in input sentences from scientific texts. During the competition, we achieved a leading F1 SOMD score of 0.89 in Phase I by first fine-tuning ModernBERT for NER, and then using the extracted entity pairs for RE. Additionally, we trained a model that jointly optimizes entity and relation losses, leading to an improvement in F1 SOMD score to 0.92. Retraining the same model on an augmented dataset, we achieved the second best F1 SOMD score of 0.55 in Phase II. In the Open Submission phase, we experimented with adapative fine-tuning, achieving an F1 SOMD score of 0.6, with the best macro average for NER being 0.69. Our work shows the efficiency of fine-tuning a niche task like software mention detection despite having limited data and the promise of adaptive fine-tuning on Out of Distribution (OOD) dataset.

## introduction software is an integral part of scientific research and scholarly documents @xcite . despite its significance, software is often referenced informally rather than being formally cited in academic publications @xcite . automatically identifying these inconsistencies in mentioning software and their associated attributes can enhance the transparency, *all authors contributed equally accessibility, and reproducibility of scientific research @xcite . the software mention detection 2025 @xcite challenge aimed to advance the development of robust models that can jointly detect software entities and classify their relations in scholarly documents. the goal of this competition was to automatically extract structured information about software used in scientific research, improving understanding and reproducibility. named entity recognition (ner) and relation extraction (re) are two of the most important classical problems in natural language processing (nlp) @xcite . since relation depends on the inherent named entity, these two problems had been posed as a joint tasks in the challenge @xcite . the somd 2025 shared task aimed to tackle this problem by offering two phases as below: in this paper, we experiment with various architectures and identify the most effective ones for the joint extraction of named entities and their relations. we focus on approaches that not only perform well in the given tasks but could also generalize to similar joint extraction problems in the future. to this end, we fine-tuned encoder-only transformer models and evaluated their performance. in phase i, our best-performing model was optimized by first performing ner through fine-tuning mod-ernbert @xcite , followed by identifying relationships between the extracted entities. a different model trained after phase i, referred to as the modified joint model (see section 4.1.4), later surpassed the performance of the previously best-performing model. in phase ii, we augmented the training data using large language model (llm) generated examples and retrained the modified joint model. during the open submission phase, we applied adaptive finetuning @xcite , which led to improvement in scores obtained during phase ii. all relevant code and implementations have been made publicly available on github 3 for reproducibility and further research. ## related studies ner and re are key tasks in information extraction, with several models and datasets developed to improve their performance @xcite @xcite @xcite . @xcite introduced bert, an encoder-only transformer model, which became a standard for extracting contextual embeddings in nlp tasks. building on bert, dygie++ proposed a dynamic span graph approach for joint ner, re, and event extraction, achieving f1 scores of 67.5% for ner and 48.4% for re on the sci-erc dataset. @xcite presented spert, an attention-based model for span-based joint entity and relation extraction, which achieved f1 scores of 70.33% for entity recognition and 50.84% for relation extraction on scierc using scibert. @xcite introduced rebel, a sequence-to-sequence model based on bart, which achieved an f1 score of 92.02% for re tasks on the nyt dataset by linearizing triplets into text sequences. @xcite proposed onerel, a joint entity and relation extraction model, which treats the task as a triple classification problem and achieved an f1 score of 92.9% on the nyt dataset. @xcite introduced iter, an encoder-based model utilizing flan-t5, which performed ner and re in three parallelizable steps, achieving f1 3 https://github.com/ekbanasolutions/somd-2025 scores of 91.7% for ner and 71.9% for re on ace05. modernbert was introduced as a state-ofthe-art encoder-only model, optimizing bert with modern enhancements @xcite . trained on 2 trillion tokens, it showed significant improvements across various domains, though its application in ner and re remained unexplored. @xcite explored scideberta, a model specialized for scientific text, in somd to extract entities and relations with domain-aware contextual embeddings. in somd 2024, a three-stage framework based on xlm-roberta achieved a macro-averaged f1 score of 67.8%, ranking third in sub-task i (ner), with steps for entity classification, extraction, and type classification @xcite . in this paper, we experiment with modernbert and scideberta for generating contextual embedding in a joint model with an added fine-tuning layers for ner and re, aiming to enhance the state of art performance specifically in software mention detection tasks. ## dataset description phase-i the given train dataset @xcite consisted of 1149 sentences with their corresponding entity and relation labels, while the test set (indistribution) had 203 sentences. the class distributions of entities and relations in the somd 2025 train dataset are illustrated in figure 1 and figure 2 respectively. ## methods the methodologies employed in phase i, phase ii and the open submission track are detailed in the following sections. for each phase, multiple methods are presented to reflect the step-by-step progression of our approach. these variations reflect iterative improvements made based on experimental results, showing the gradual development of our approach. ## results after the phase was completed, we experimented with multiple approaches which resulted in improvement of the f1 score as mentioned in the all the macro f1 score for ner and re are computed using exact match via seqeval library @xcite .