E xpert G en QA : Open-ended QA generation in Specialized Domains (2025)

Abstract: Generating high-quality question–answer (QA) pairs for specialized technical domains is essential for advancing knowledge comprehension, yet remains challenging. Existing methods often yield generic or shallow questions that fail to reflect the depth and structure of expert-written examples. We propose ExpertGenQA, a generation protocol that combines few-shot prompting with dual categorization by topic and question style to produce more diverse and cognitively meaningful QA pairs. ExpertGenQA achieves twice the efficiency of standard few-shot methods while maintaining 94.4% topic coverage. Unlike LLM-based judges, which often favor surface fluency, Bloom’s Taxonomy analysis shows that ExpertGenQA better captures expert-level cognitive complexity. When used to train retrieval systems, our questions improve top-1 accuracy by 13.02%, demonstrating their practical value for domain-specific applications.

## introduction in high-stakes domains like law, transportation, and finance, expert-written questions capture how professionals reason, prioritize, and apply knowledge. they emphasize concepts essential for learning and grounded decision-making @xcite @xcite . different from generic questions that target factual recall, domain-specific questions support deeper understanding and better reflect real-world scenarios. they are also important for training ai systems in tasks such as information retrieval and question answering, where diverse, informationrich questions expose models to complex semantic structures and improve generalization to unseen human-written queries @xcite . however, generating such questions at scale requires substantial domain expertise and time, making automatic generation an attractive solution. while domain-adapted llms such as bloomberggpt @xcite , fingpt @xcite , ecomgpt @xcite , biogpt @xcite , and med-palm @xcite are designed to answer domain-specific questions, they are not optimized for generating them. prompt-based approaches like med-prompt @xcite show that simply asking better questions can sometimes outperform fine-tuned models on domain benchmarks. still, generating high-quality, domain-specific questions remains underexplored. existing methods often default to generic, surface-level prompts @xcite ) that fail to capture the depth and structure of expert-authored questions. for instance, a legal professional gains little from a basic query like "what is a subpoena?" when their work requires complex, scenario-driven questions that synthesize statutes, precedents, and regulations. evaluating the usefulness of generated questions also remains a challenge. while llms have advanced in comparing generated answers, existing evaluation methods are less effective for questions. reward models and llm-as-judge approaches @xcite @xcite often prioritize fluency and syntactic form over semantic depth and task relevance. as a result, questions that score highly with llm judges frequently perform poorly in downstream retrieval tasks, failing to meet the practical needs of domain experts. we introduce a question generation pipeline that learns to produce domain-specific questions by imitating a small set of expert-written examples. our approach focuses on generating question-answer pairs that are not only comprehensive in topic coverage but also capture the cognitive complexity and practical needs of domain experts. to achieve this, we ground generation in expert-written faqs and apply a dual-categorization strategy based on question style and topic. our proposed pipeline, ex-pertgenqa (figure 1 ), significantly outperforms standard few-shot prompting and template-based methods (e.g., mdcure @xcite ), improving top-1 document retrieval accuracy on human-written test queries from 23.96% to 36.98%. our analysis shows that these improvements stem from diversity, cognitive load @xcite , and topic coverage, all of which strongly correlate with retrieval performance. expertgenqa also doubles the generation efficiency of baseline few-shot prompting while maintaining 94.4% topic coverage. why retrieval as an evaluation metric? llmbased evaluation metrics often reward surface-level fluency rather than task relevance. in contrast, retrieval accuracy provides a task-grounded measure of question quality by evaluating whether a question helps models retrieve conceptually relevant content. this better reflects expert reasoning, where questions encode meaningful structure, connect related concepts, and highlight contextually important information. models trained on highquality, expert-style questions learn to attend to these signals, resulting in stronger generalization to human-written queries. we use retrieval accuracy as the primary evaluation metric, supported by auxiliary analysis on diversity, cognitive load, and topic coverage @xcite . ## methodology this work aims to generate synthetic questions in specialized domains that have practical utility for domain experts. to achieve this, we collect a small set of expert-written questions to serve as exemplars. our expertgenqa pipeline learns domainspecific patterns from these examples and generates new questions that closely align with expertwritten questions while maintaining high diversity and comprehensive coverage of source documents. ## evaluation and experimental setup while a plethora of methods exist for evaluating llm generated answers @xcite , research into evaluating llm generated questions is limited. furthermore, we uncover key shortcomings in existing methods @xcite ) that use llms or reward models in our pilot experiments. as such, we opt for a retrievalbased evaluation framework in which we assess question quality through its impact on downstream retrieval performance, using it as a proxy for practical utility. this setup directly measures how well generated questions facilitate document navigation and comprehension in highly technical domains. ## analysis while directly finetuning a retrieval language model provides the most accurate measure of synthetic question effectiveness, it is impractical during pipeline development due to the compute required. we investigate alternative evaluation metrics using the larger fra domain to determine which ones meaningfully correlate with improvements in retrieval language model performance. these metrics could serve as more practical proxies for assessing question quality during the development process. ## conclusion this work introduces expertgenqa, a protocol for combining structured categorization with fewshot learning to generate high quality domainspecific questions. our evaluation reveals limitations in current automated assessment methods: both reward models and llm-as-judge approaches struggle to meaningfully evaluate technical content quality. the cognitive complexity analysis shows expertgenqa better preserves the distribution of expert-level thinking demands compared to template-based methods. retrieval lms using expertgenga achieves improved retrieval performance even compared to the much larger nvembed-2-70b, although the modest absolute performance highlights ongoing challenges in technical domain retrieval. in the future, we will extend this approach to other specialized fields where expert knowledge is crucial but limited.