Extracting Software Mentions and Relations using Transformers and LLM -Generated Synthetic Data at SOMD 2025 (2025)

Abstract: As part of the SOMD 2025 shared task on Software Mention Detection, we solved the problem of detecting and disambiguating software mentions in academic texts. a very important but under appreciated factor in research transparency and reproducibility. Software is an essential building block of scientific activity, but it often does not receive official citation in scholarly literature, and there are many informal mentions that are hard to follow and analyse. In order to enhance research accessibility and interpretability, we built a system that identifies software mentions and their properties (e.g., version numbers, URLs) as named entities, and classify relationships between them. Our dataset contained approximately 1,100 manually annotated sentences of full-text scholarly articles, representing diverse types of software like operating systems and applications. We fine-tuned DeBERTa based models for the Named Entity Recognition (NER) task and handled Relation Extraction (RE) as a classification problem over entity pairs. Due to the dataset size, we employed Large Language Models to create synthetic training data for augmentation. Our system achieved strong performance, with a 65% F1 score on NER (ranking 2nd in test phase) and a 47% F1 score on RE and combined macro 56% F1, showing the performance of our approach in this area.

## introduction somd 2025 foot_0 shared task focuses on software mention detection. software which is firmly integrated into the fabric of contemporary scientific inquiry-not just as an experimental and analytic tool but also as a subject of theoretical discussion. in spite of its ubiquity, software regularly fails to receive systematic and formal citation in scholarly papers. closing this gap calls for automated systems with the ability to detect and comprehend software mentions and their corresponding context in scholarly texts. in this paper, we introduce a system to address this task based on named entity recognition (ner) and relation extraction (re) methods. our solution is built upon a heavily annotated dataset of 1,150 sentences @xcite of research articles, covering a wide variety of software-related entities and relationships. these are not limited to simple identifiers such as software names, but also encompass more involved constructions including versions, developers, licenses, and usage scenarios. the hierarchical annotation structure and bio tagging scheme of the dataset allow for fine-grained entity recogni- ## background software plays a fundamental role in research across many scientific disciplines, facilitating experimentation, simulation, analysis, and reproducibility. despite its centrality, software is usually only informally discussed in academic literature and mostly absent of reference citations or proper metadata, which presents issues for subsequent indexing, reuse and reproducibility @xcite . in order to address these issues, it is now common practice to automate the identification of software mentions in academic writings and the various attributes of that mention. this commonly involves two natural language processing tasks: named entity recognition (ner) and relation extraction (re). commonly ner and re have been looked at independently using pipeline-based approaches which lead to iterative errors, e.g. misidentified entities will lead to misidentified relationships @xcite ; @xcite . in response to these limitations researchers have begun to shift away from independent modeling tasks to modeling both tasks, in a joint fashion, simultaneously and in turn improving performance and efficiency. ## performance analysis experimentation phase. in the experimentation stage, we investigated several modeling methods for the named entity recognition (ner) task, mainly utilizing the deberta-v3-large @xcite model with a token classification method. we started with a vanilla deberta-v3-large model and used a 4-fold cross-validation configuration. in this setup, we observed that the model trained on fold 1 achieved an f1 score (macro average) of 60% to enhance generalization, we supplemented our training data with synthetically generated datasets produced using different large language models (llms), namely gemma-2-9b-it @xcite , mistral-7b-instruct-v0.1 @xcite , and qwen2.5-7b-instruct @xcite . among these, only the synthetic data generated by mistral-7b-instruct-v0.1 @xcite led to a significant improvement of 5% in the f1 score. the datasets generated by qwen2.5-7b-instruct and gemma-2-9b-it @xcite of-fered a modest 2% gain our best-performing system was the final model: a deberta-v3-large @xcite trained on the complete original training dataset, augmented with synthetic data from mistral-7b-instruct-v0.1 @xcite . this configuration achieved a macro-averaged f1 score of 65% we also experimented with xlm-roberta @xcite using the same setup, training it on the full dataset along with the additional synthetic data. however, it achieved only a macro-averaged f1 score of 28% ## acknowledgments we would like to thank the organisers of the software mention detection (somd) shared task and the scholarly document processing (sdp) workshop for running this competition. we also thank the anonymous reviewers for their insightful and constructive comments, which helped raise the standard of this manuscript considerably. ## conclusion our approach exhibits decent performance in addressing the task of re and ner within the software space. in the case of ner, our model has a macro f1-score of 65% overall, our findings highlight the potential of combining pretrained models with synthetic data to tackle complex information retrieval problems in scholarly text. our hope is that this work will help to further develop better software mention detection systems.