SOMD 2025: A Challenging Shared Tasks for Software Related Information Extraction (2025)

Abstract: The use of software in acquiring, analyzing, and interpreting research data underscores its role as an essential artifact of scientific inquiry.Understanding and tracing the provenance of software in research helps in reproducible and collaborative research works.In this paper, we present an overview of our second iteration of the So ftware M ention D etection (SOMD) shared task as a part of the Scholarly Document Processing (SDP) workshop, that will be held in conjunction with ACL in 2025. We intend to foster among participants to brainstorm for optimized software mention detection and additional attributes and relation extraction tasks in the provided gold standard benchmark. Our shared task has two phases of challenges. First, the participants focus on implementing a joint framework for NER and RE for the given dataset. At the same time, the second phase includes the out-of-distribution dataset to evaluate the generalizability of the methods proposed in Phase I. The competition (March-April 2025) attracted 18 participants and spanned two months. Four teams have finished the competition and submitted full system descriptions. Participants applied various approaches, including joint and pipeline models, and explored data augmentation with LLM-generated samples.The evaluation was based on a macro-F1 score for both NER and RE, with the average reported as the SOMD-score.The winning teams achieved a SOMD-score of 0.89 in Phase I and 0.63 in Phase II, demonstrating the challenge of generalization.

## introduction scientific research is becoming progressively datacentric, and software plays an important role across disciplines by enabling the analysis, processing, and modeling of research data. as such, it has emerged as a key scholarly artifact, essential not only for conducting research but also for ensuring the reproducibility and advancement of scientific knowledge. to ensure transparency and reproducibility of scientific work, it is essential to identify the software used and trace its provenance, thus encouraging collaboration among scientists/researchers. software mentions in scholarly publications are heterogeneous, informal, and in widespread use. therefore, identifying and disambiguating software mentions, while attending to its metadata, is an essential yet challenging task. various knowledge graph resources, such as openaire @xcite and soft-warekg @xcite , link open-access articles to the software used, supporting the need for robust methods to identify, extract, link, and disambiguate software mentions. various existing citation principles regarding software usage and mentions @xcite promote knowledge sharing and innovation. however, these principles are not always strictly followed in all works, resulting in informal and incomplete information regarding the software mentioned or used @xcite . robust information extraction (ie) methods help to detect and disambiguate software mentions and related metadata. somesci @xcite is a manually curated gold standard corpus about software mentioned in scientific articles, providing training samples for named entity recognition (ner), relation extraction(re), entity disambiguation(ed), and entity linking(el). based on this dataset, the somd2024 shared task was organized to advance research on automatic detection and analysis of software mentions in scholarly articles. the task challenged participants to develop methods for (i) detecting software mentions, (ii) identifying associated attributes, and (iii) classifying the relations between software and their attributes. in this paper, we present the software mention detection shared task (somd2025)-the successor to somd2024 @xcite . the goal is to advance the field through communitydriven development and evaluation of new methods. somd2025 builds on the success of the previous edition. but while the first iteration focused on establishing ner, attribute detection and re for software mentions in separate subtasks, somd2025 emphasizes a joint evaluation of these subtasks. our task advances the development of a pipeline for ie components (ner; re) for scientific knowledge. these pipelines serve as an initial step for functions such as metadata enrichment, semantic linking, and knowledge graph construction from scholarly articles, aligning with nfdi4ds's foot_0 and berd@nfdi's foot_1 broader mission of supporting the research data lifecycle and providing infrastructures. we focus on the discovery and traceability of the software mentioned in research publications-a crucial step in the reproducibility of research. in addition to learning and evaluating the joint ner and re framework, we introduce an out-ofdistribution (ood) test set to assess the generalizability of the models-a significantly more challenging benchmark compared to the in-distribution data. we hosted the two subsequent phases of the competition in the codabench platform @xcite . phase i aims at model development, where we provide gold-standard training and test splits to the participants. phase ii challenges participants to apply their models from phase i on an out-ofdistribution dataset comprised of scholarly documents that were not part of the training or test set used in phase i. although 18 participants registered, only three teams submitted for phase i and five teams made submission for phase ii. four of them submitted a system description for the workshop proceedings. to encourage future research, we have transformed phase 2 into an open submission phase that will allow further development of ie systems for our task. 3 we provide the competition details in the rest of the paper. we include the task description and the evaluation metrics in section 3 and a description of the dataset for both phases in section 3.2. we summarize results in section 5, where we compare the methods of different participants. joint ner and relation extraction. joint learning of ner and re has emerged as a robust alternative to traditional pipeline approaches, which often suffer from error propagation. integrated models have demonstrated improved accuracy and efficiency by simultaneously extracting entities and their relations @xcite @xcite . while these models have been widely adopted in general and biomedical domains, only a few efforts-such as somesci and softwarekgexplicitly address relation-level modeling in the software domain. their contributions underscore the growing importance of joint models in domainspecific information extraction. prior studies consistently show joint frameworks outperform pipeline systems @xcite , addressing limitations in earlier methods @xcite . ## task description we focus on the discovery and traceability of the software mentioned in research publications-a crucial step to ensure the reproducibility of research. for this purpose, we propose information extraction of software and related metadata, including named entity recognition and relation extraction. we approach the concept of software as a form of research artifact, with softwarerelated ie serving as a foundational element in the construction of research knowledge graphs (rkgs) @xcite . these rkgs, in turn, are built upon a foundation of scholarly articles, thereby facilitating the aggregation and organization of research findings. we encourage participants to build robust and generalizable nlp methods, i.e., models for software mentions, attribute detection, and relation extraction. an instance of a sentence with annotated software mentions, attributes, and relations is illustrated in figure1. somd2024 @xcite had hosted these problems as three independent subtasks. somd2025 combines these three subtasks into an end-to-end setup for training and evaluation. we endorse jointly learning the automatic extraction of software mentions, its attributes, and their relations from scholarly documents. our task belongs to the well-known problem in information extraction, i.e., joint learning paradigm for ner and re @xcite @xcite . we have two phases of competition. we provide the labeled dataset for phase i, supporting model training. it contains three aligned files per instance: a tokenized text file, a ner label file, and a relation label file, each line corresponding to a sentence. the ner file uses iob2 ## participants and approaches a total of 18 teams registered for the somd2025 shared task. three teams participated fully by submitting results in both phase i and phase ii, as well as providing a system description. these teams were the tu graz data team (tugraz), a team from the nepal-based company ekbana, and one participant from the universidade de aveiro (uaveiro). additionally, there was one late participation, consisting of a master's student from the georgia institute of technology and an independent researcher (psr123), who submitted only for phase ii and provided a system description. these four teams are referred to as the final participants in this paper. one further participant submitted results in phase ii but did not provide a system description and is therefore not discussed further. all final participants used the open submission phase to further test and refine their approaches after the conclusion of phase ii. in this section, we introduce the four final approaches alongside two baseline models. ## results in this section, we present the results of the somd2025 shared task, including performance scores for both phases of the competition. for this section, we focus on the more challenging phase ii test set because it better illustrates the generalization capabilities of the used ie models. we compare the results of all final participating teams, highlight the top-performing systems, and contrast them with baseline models. additionally, we include results from the non-competitive open submission phase as including unpublished codabench results reported in the corresponding system descriptions of the teams. this provides further insight into model improvements beyond the official evaluation period. the main results can be found in ## conclusion the somd2025 shared task addressed the challenge of extracting software mentions, attributes, and relations from scientific articles using a joint ner and re framework. with two evaluation phases, including an out-of-distribution test set, the task emphasized both extraction accuracy and model generalizability. participating teams employed diverse strategies, including pretrained language models, graph-based architectures, and data augmentation using llms. results show that while current methods perform well on in-distribution data, generalization remains a significant challenge. the top systems showed improvements over previous baselines, particularly in phase i, and provided valuable insights into joint learning strategies and training data choices. the shared task supports sustained progress in software-related information extraction from scholarly texts by continuing with an open, ongoing submission phase.