From In-Distribution to Out-of-Distribution: Joint Loss for Improving Generalization in Software Mention and Relation Extraction (2025)

Abstract: Identifying software entities and their semantic relations in scientific texts is key for reproducibility and machine-readable knowledge graphs, yet models struggle with domain variability and sparse supervision. We address this by evaluating joint Named Entity Recognition (NER) and Relation Extraction (RE) models on the SOMD 2025 shared task, emphasizing generalization to out-of-domain scholarly texts. We propose a unified training objective that jointly optimizes both tasks using a shared loss function and demonstrates that joint loss formulations can improve out-of-domain robustness compared to disjoint training. Our results reveal significant performance gaps between in- and out-of-domain settings, prompting critical reflections on modeling strategies for software knowledge extraction. Notably, our approach ranked 1st in Phase 2 (out-of-distribution) and 2nd in Phase 1 (in-distribution) in the SOMD 2025 shared task, showing strong generalization and robust performance across domains.

## introduction software is crucial to scientific work, but identifying its mentions and relations in text is difficult due to ambiguity, limited supervision, and domain variation @xcite . the software mention detection (somd) shared task series addresses these challenges through benchmark datasets and evaluation frameworks. while the 2024 edition focused on pipeline approaches using full-text articles from the somesci corpus @xcite , the 2025 task shifts to joint modeling of ner and re at the sentence level to reduce cascading errors @xcite @xcite ). in this work, we evaluate joint ner and re models for software knowledge extraction under domain shift. we compare span-based (gliner @xcite ), encoder-based (bert, scibert, and deberta @xcite @xcite ), and instruction-tuned architectures (gemini and llama @xcite ) on the somd 2025 benchmark. our central research question is: does a joint loss objective improve generalization in multitask ner and re models? we find that joint loss boosts in-distribution performance and consistently mitigates degradation in out-of-distribution settings. this highlights its utility as a simple yet effective mechanism for improving robustness in extractive multitask learning. ## related work extracting software mentions and their semantic relations from scientific texts is crucial for reproducibility and knowledge organization, yet software is often referenced informally, posing challenges for automatic identification and disambiguation @xcite . early approaches treated ner and re as separate pipeline stages @xcite , but this modularity frequently led to cascading errors, particularly when entity boundaries were misidentified @xcite . to mitigate these issues, recent work has shifted toward joint models that unify ner and re within a single architecture. span-based methods leverage contextualized representations to jointly encode entities and their relations @xcite , while generationbased architectures such as rebel @xcite and iterative decoding frameworks @xcite aim to improve expressiveness and compositional generalization. in addition, a growing body of work has focused on discourse-aware and document-level models, which extend the context window beyond a single sentence. for example, @xcite introduce a span-based architecture that propagates information across sentences using global context graphs, while @xcite use levitated markers to retain global coreference and discourse signals. models like scirex @xcite further highlight the importance of integrating paragraph-level and document-level context to improve entity linking and relation reasoning in scientific documents. these approaches demonstrate that sentence-local models are often insufficient for resolving longrange dependencies, a limitation we also observe in our results. while much of the literature emphasizes architectural integration, fewer studies explore the role of joint loss optimization, i.e., coupling ner and re learning via a shared objective @xcite @xcite . we extend this line of research by comparing joint and disjoint loss formulations across model families and analyzing their impact on generalization, particularly under distribution shift. in the somd 2024 shared task, participants explored alternatives to traditional pipeline systems. others investigated few-shot adaptation with gpt-3.5/4 @xcite or token-level fine-tuning with falcon-7b @xcite , highlighting the challenges of aligning llms with structured tasks. building on these efforts, our work unifies evaluation across model types and emphasizes generalization under domain shift, responding to recent calls for more robust extraction frameworks @xcite . ## task and dataset the somd 2025 task involves a two-phase sentence-level joint ner and re on annotated scientific texts from the somesci corpus @xcite . the annotation schema includes 14 entity types and 11 relation types that can be found in appendix a. in phase 1 (in-distribution validation set) models are trained and evaluated on 1,432 annotated sentences (train) and 256 test sentences, while in phase 2 (out-of-distribution test set) 457 new test sentences from unseen domains are released without gold labels. models are evaluated via leaderboard submissions, with an additional focus on generalization. ## conclusion we evaluate joint ner and re models for extracting software mentions and relations in scientific texts, with a focus on out-of-distribution generalization. our results show that a shared loss objective consistently boosts performance across architectures, indicating that multitask learning benefits not only from architectural integration but also from coupled optimization. this joint loss approach reduces error propagation and enhances robustness, making it a simple yet effective strategy for structured scientific information extraction.