Enhancing Domain-Specific Encoder Models with LLM -Generated Data: How to Leverage Ontologies, and How to Do Without Them (2025)

Abstract: We investigate the use of LLM-generated data for continual pretraining of transformer encoder models in specialized domains with limited training data, using the scientific domain of invasion biology as a case study. To this end, we leverage domain-specific ontologies by enriching them with LLM-generated data and pretraining the encoder model as an ontology-informed embedding model for concept definitions. To evaluate the effectiveness of this method, we compile a benchmark specifically designed for assessing model performance in invasion biology. After demonstrating substantial improvements over standard MLM pretraining, we investigate the feasibility of applying the proposed approach to domains without comprehensive ontologies by substituting ontological concepts with concepts automatically extracted from a small corpus of scientific abstracts and establishing relationships between concepts through distributional statistics. Our results demonstrate that this automated approach achieves comparable performance using only a small set of scientific abstracts, resulting in a fully automated pipeline for enhancing domain-specific understanding of small encoder models that is especially suited for application in low-resource settings and achieves performance comparable to masked language modeling pretraining on much larger datasets.

## introduction transformer encoder models such as bert @xcite and its successors (e.g., @xcite @xcite have consistently achieved state-of-the-art results across a wide range of nlp tasks. these successes are largely driven by large-scale pretraining on generaldomain corpora such as wikipedia and bookcorpus @xcite , using objectives like masked language modeling (mlm) or replaced token detection @xcite . while transformer encoders offer an optimal balance between performance and efficiency, their full effectiveness in specialized domains -such as scientific text processing -is often enabled by additional pretraining on domain-specific corpora @xcite , proven highly effective in fields where extensive domain-specific data is available (e.g., biomedical text processing @xcite . however, in specialized disciplines with limited resources, the potential of this approach diminishes, highlighting the need for alternative methods of injecting domain knowledge during pretraining. to address this challenge, we propose a novel method for continual pretraining of transformer encoder models that leverages a set of domainrelevant concepts and their corresponding definitions as the core training resource. these concepts can be drawn from domain-specific ontologies (i.e., data structures containing precise, domain-specific and structured knowledge curated by domain experts @xcite ; @xcite ; algergawy et al. ( 2025 )) or extracted from texts using llms. using this resource, we pretrain the model as an embedding model for concept definitions, encouraging definitions of identical or related con-cepts to occupy nearby positions in the embedding space and thus enabling the model to develop a structured understanding of domain-specific entities and their interconnections. in this process, we perform an extensive analysis of the effectiveness of incorporating different types of information, moving from using domainrelevant concepts extracted from the ontologies towards a fully unsupervised pipeline with llmextracted concepts. contributions: ## related work continual pretraining is an effective and efficient approach to make lms robust against new, everchanging data that differs from its original pretraining @xcite @xcite , enhances an llm's domain specific effectiveness @xcite @xcite and improves knowledge transfer to downstream tasks @xcite . ontologies and knowledge graphs (kgs) have been explored as resource for continual pretraining since they provide a structured representation of domain knowledge in the form of unique entities and precise relations between them, contrasting the distributed and often less precise knowledge representation within neural networks. to bridge this gap, various methods have been proposed to integrate structured knowledge into transformer models. while some approaches incorporate kg information during inference @xcite @xcite , the majority of approaches focus on creating kg-informed pretraining methods, for example by performing mlm pretraining that incorporates knowledge about entities @xcite , performing mlm pretraining on sentences derived from kg triples @xcite @xcite @xcite , designing auxiliary classification tasks based on ontological knowledge @xcite or by creating contrastive ontology-informed sentence embedding methods @xcite . our approach aligns most closely with the latter but extends it into a broader framework that incorporates not only relationships between concepts but also llm-derived knowledge about individual concepts by incorporating synthetic concept definitions, thus creating a more informative and flexible pretraining process that is not reliant on the presence of ontologies. using synthetic data for model pretraining and/or fine-tuning is an appealing prospect @xcite , especially in specialized domains with little available training data. many studies explore the potential of llm-generated or llm-annotated data to enhance task-specific performance, both for encoder @xcite @xcite and decoder architectures @xcite . beyond task-specific fine-tuning, synthetic data has also been investigated for task-agnostic pretraining. while this approach has shown promise for general-domain models @xcite @xcite , its application in domain-specific pretraining remains relatively underexplored (e.g., @xcite . despite its advantages, synthetic data introduces risks, including potential performance degradation compared to human-generated data -a phenomenon known as model collapse @xcite , prompting studies aimed at mitigating this effect, especially for autoregressive llms @xcite @xcite . in section 6, this phenomenon will be further discussed in the context of our own experiments. ## method we propose a method for injecting domain knowledge into transformer models through continual pretraining. this section provides a general overview of our approach, while section 4 and section 5 detail and evaluate its application to datasets derived from ontologies and scientific abstracts. ## ontology-informed pretraining this section details the application and evaluation of our proposed method, using domain-specific ontologies for dataset creation. our experiments focus on the scientific domain of invasion biology, a specialized subfield of biodiversity research that investigates non-native species, their introduction pathways, ecological impacts, and management strategies to mitigate their effects on ecosystems (jeschke and heger, 2018). ## using llm-extracted keywords in the previous section, we explored the performance improvements achieved by combining our proposed similarity loss on ontology-derived data with traditional mlm pretraining. while this approach is highly valuable in domains with available ontologies, many fields may lack such structured resources. to address this limitation, we explore the feasibility of using an llm for constructing a dataset of domain-relevant concepts, definitions, and relations using only a small set of scientific abstracts. we compare results achieved on our original dataset of 5,000 abstracts with those using ontology-derived data and also evaluate how well our approach scales with increasing dataset size. ## conclusion in this study, we investigated the use of llmgenerated, synthetic data for continual pretraining of domain-specific encoder models, demonstrating how to utilize domain specific ontologies or derive domain information through llm-extraction from scientific abstracts for domains where ontologies may not be available. our results demonstrate that the proposed pretraining approach produces strong synergistic effects when combined with masked language model-ing training. this leads to significant performance improvements in low-resource settings and results in a model surpassing other specialized models from the broader biomedical domain, despite being trained on orders of magnitude less data. given the minimal data requirements, our approach has the potential to be widely applicable beyond the domain explored in this study. furthermore, its robustness against model collapse despite using synthetic data represents a meaningful advancement in leveraging llm-generated data for training specialized models.