Automated Collection of Evaluation Dataset for Semantic Search in Low-Resource Domain Language (2025)

Abstract: Domain-specific languages that use a lot of specific terminology often fall into the category of low-resource languages. Collecting test datasets in a narrow domain is time-consuming and requires skilled human resources with domain knowledge and training for the annotation task. This study addresses the challenge of automated collecting test datasets to evaluate semantic search in low-resource domain-specific German language of the process industry. Our approach proposes an end-to-end annotation pipeline for automated query generation to the score reassessment of query-document pairs. To overcome the lack of text encoders trained in the German chemistry domain, we explore a principle of an ensemble of “weak” text encoders trained on common knowledge datasets. We combine individual relevance scores from diverse models to retrieve document candidates and relevance scores generated by an LLM, aiming to achieve consensus on query-document alignment. Evaluation results demonstrate that the ensemble method significantly improves alignment with human-assigned relevance scores, outperforming individual models in both inter-coder agreement and accuracy metrics. These findings suggest that ensemble learning can effectively adapt semantic search systems for specialized, low-resource languages, offering a practical solution to resource limitations in domain-specific contexts.

## introduction in nlp, a low-resource language lacks sufficient linguistic data, resources, or tools for effective model training and development @xcite . domain-specific german, especially in areas with professional jargon, codes, acronyms, and numeric data, qualifies as a low-resource language because large, publicly accessible datasets for such specialized domains are scarce. as a result, few language models are figure 1 : an example of a mocked text log from a shift book in the german language. the logs contain a log of domain-specific terms, which require domain knowledge in the area and know specifics of the production process trained specifically for these areas. while general german has extensive nlp resources, specialized sublanguages often demand unique datasets that are difficult to gather and typically limited in volume. shift logs in the process industry are detailed records maintained by operators or technicians during their work shifts (see figure 1 foot_0 ). they document key operational activities, system statuses, production metrics, equipment performance, process parameters, maintenance activities, safety observations, product quality, and any incidents or anomalies. the process industry produces and transforms raw materials into finished products through chemical, physical, or biological processes. the complexity of parsing and interpreting professional terminology and industry-specific syntax requires models trained on annotated datasets tailored to the domain, which are often non-existent or proprietary. this lack of accessible, high-quality datasets makes it difficult to build, fine-tune, or adapt existing nlp models for these specialized uses. without significant efforts in curating and labeling domain-specific data, language models will struggle with accurate interpretation and generation in these fields. collecting and annotating text collections for se-mantic search in low-resource languages presents several significant challenges. first, finding qualified annotators for this task who are both fluent in the language and trained in linguistic annotation can be extremely difficult. moreover, the complexity of semantic search requires annotations beyond basic syntactic labeling, such as entity recognition and coreference resolution, which demand specialized knowledge and increase the task's difficulty. second, standalone general language models trained on high-resource languages can collect the test data to a certain extent but do not transfer well to these low-resource contexts and lack accurate language representation of the domain language. this paper explores the principle of ensemble learning to create test collections for semantic search in domain-specific german language. ensemble learning is a machine learning technique that combines multiple individual models, often called "weak learners," to create a more powerful and accurate predictive model by mitigating each other's weaknesses @xcite . our experiments demonstrate that combining an ensemble of multiple encoders with a generative llm (gpt-4o in our case) to reassess relevance scores significantly improves the quality of test collections for semantic search evaluation. specifically, this approach increases inter-coder agreement (measured by krippendorff's alpha) by nearly four times and improves the f1-score by 1.5 times. ## related work ensemble learning improves machine learning performance by combining predictions from multiple models, thus enhancing accuracy, reducing variance, and mitigating bias (mienye and sun, 2022). ensemble learning is popular across domain-specific domains and applications, such as medical diagnosis and fraud detection. it has started evolving from being used with machine learning algorithms to deep learning models. llms have already been widely used for data annotation, specifically for domain-specific tasks requiring specialized domain knowledge, where human annotations are costly but crucial @xcite . multiple studies have evaluated llms in biomedicine @xcite , law and education @xcite , and financial sector @xcite . while llms are a powerful tool for data annotation, the studies show that standalone llms perform worse than human annotators @xcite . to mitigate the drawbacks of llm annotations, new methods were proposed to involve reasoning, reevaluating the assigned labels, or involving collective decisions. one of the state-of-the-art techniques is to use a human-in-loop annotation process and help human annotators by augmenting them with the fast llm-pre-annotated labels @xcite . the most recent development employs an ensemble of llms for annotation @xcite or utilizes a synergy of thoughts across multiple smaller-scale lms @xcite , similar to ensemble learning with "weak" models. ## methodology ensemble learning is widely used in practice because it can improve model robustness and accuracy and reduce variance, especially when individual models are prone to errors or have high variability. the central idea is that by aggregating the predictions of several models, the ensemble can outperform any single model, reducing the risk of overfitting and improving generalization. ensemble methods leverage the strengths of different models while compensating for their weaknesses, leading to better performance on complex tasks @xcite . in stacking of ensemble learning, different models (often of different types) are trained, and their predictions are used as input to a "meta-model," which learns how to combine these predictions to make the final decision. the methodology of the ensemble for annotating a test collection for semantic search comprises two main parts: (1) document indexing and (2) creation of the query-document pairs. the key aspect of document indexing is using a set of encoders with various architectures and training strategies. the goal is to combine different aspects of the document similarity that each encoder has learned. re-ranking combines the relevance score based on the document similarity with the score generated by a generative llm. llm assesses the relevance of the query-document pair independently from the score used for the retrieval, thus allowing the combining of another "point of view" to the querydocument relevance. figure 2 depicts the proposed methodology. ## evaluation we evaluated our approach against the manually assigned relevance scores to the retrieved documents @xcite . the goal was to evaluate how the proposed approach agreed with how a human assessed the query-document pairs. ## conclusion this paper investigates a principle of ensemble learning with "weak" text encoders to create a test collection for the semantic search evaluation. we combined multiple text encoder models for document retrieval. we experimented with creating a test collection for semantic search evaluation in the domain of the german process industry. the experiments showed that computing the final relevance score by combining the average score of the ensemble of text encoders and an independent relevance score created by an llm for each querydocument pair increases the inter-coder agreement and accuracy metrics for several datasets. we invite the research community to apply further and investigate the proposed methodology across additional languages and domains.