W e QA : A Benchmark for Retrieval Augmented Generation in Wind Energy Domain (2025)

Abstract: Wind energy project assessments present significant challenges for decision-makers, who must navigate and synthesize hundreds of pages of environmental and scientific documentation. These documents often span different regions and project scales, covering multiple domains of expertise. This process traditionally demands immense time and specialized knowledge from decision-makers. The advent of Large Language Models (LLM) and Retrieval Augmented Generation (RAG) approaches offer a transformative solution, enabling rapid, accurate cross-document information retrieval and synthesis. As the landscape of Natural Language Processing (NLP) and text generation continues to evolve, benchmarking becomes essential to evaluate and compare the performance of different RAG-based LLMs. In this paper, we present a comprehensive framework to generate a domain relevant RAG benchmark. Our framework is based on automatic question-answer generation with Human (domain experts)-AI (LLM) teaming. As a case study, we demonstrate the framework by introducing WeQA, a first-of-its-kind benchmark on the wind energy domain which comprises of multiple scientific documents/reports related to environmental aspects of wind energy projects. Our framework systematically evaluates RAG performance using diverse metrics and multiple question types with varying complexity level, providing a foundation for rigorous assessment of RAG-based systems in complex scientific domains and enabling researchers to identify areas for improvement in domain-specific applications.

## introduction in recent years, the advancements in llm have revolutionized various natural language processing tasks, including text and response generation. however, text generation using llm often encounters challenges such as generating irrelevant or incoherent outputs, perpetuating biases ingrained in the training data, and struggling to maintain context and factual accuracy @xcite . these issues pose significant obstacles to achieving humanlevel performance in automated text generation systems. rag effectively mitigates these common challenges by incorporating retrieved information to enhance coherence and factual accuracy, thus minimizing the generation of fictitious or irrelevant content @xcite . furthermore, concurrent works suggest rag is the most sought approach for adapting models towards accelerating repetitive and data intensive tasks in niche scientific domain such as nuclear, renewable energy, environmental policy, etc. @xcite . while rag-based systems have demonstrated promising capabilities in streamlining document analysis tasks across various professional domains, their integration into critical decision-making processes like permitting wind energy projects remains constrained due to legitimate concerns about trust and reliability. in this work, we create benchmarks to assess rag-based llm performance in the domain of permitting wind energy projects. environmental impact statements (eis) represent the cornerstone documentation within this permitting landscape, serving as comprehensive analyses that evaluate the potential environmental consequences of proposed wind energy developments. these documents play a pivotal role in promoting informed decision-making by ensuring transparency and incorporating diverse stakeholder perspectives into the approval process @xcite . by providing detailed evaluations of environmental effects, alternatives analysis, and mitigation measures, eis documentation facilitates the responsible development of wind energy infrastructure while building public trust at the same time. as rag-based llms gain traction for domainspecific applications such as wind energy permitting, their effectiveness must be rigorously assessed through robust benchmarks to ensure its practical utility and reliability @xcite . establishing high-quality benchmarks is essential to evaluate their abilities to perform regulatory-focused reasoning, accurately interpret complex eis documents, and support logical deductions grounded in the documents. such benchmarks facilitate systematic assessment of how well rag-based llms can handle the nuanced requirements of the domain @xcite . a robust evaluation framework allows researchers and practitioners to investigate the impact of retrieval strategies, model architectures, and training data, on the performance of rag, while building confidence in automated tools for critical environmental decision making @xcite . in benchmarking rag for wind energy project permitting applications, it is crucial to evaluate its performance across a diverse set of questions that reflect the complexity and variability of real-world permitting scenarios @xcite . a set of well curated and diverse questions enable a comprehensive assessment of rag's ability to interpret eis documents, analyze environmental impacts, evaluate regulatory compliance, and generate coherent responses to permitting-related queries that practitioners encounter during wind energy project review processes. to generate such questions, automated methods leveraging nlp techniques can be employed, including rule-based approaches that capture language patterns from relevant documents, template filling methods that incorporate wind energy terminologies, and neural network-based models that can efficiently create diverse question sets by leveraging the semantic relationships inherent in eis and other documents related to wind energy projects. human-curated questions offer a level of linguistic richness and contextual relevance that may be challenging to achieve solely through automated generation methods, particularly in specialized domains such as wind energy project permitting @xcite . by leveraging human expertise and domain knowledge, curated question sets can encompass a broader spectrum of linguistic variations, domain-specific considerations, and nuanced semantics @xcite , providing a more comprehensive evaluation of rag's performance across diverse scenarios and applications @xcite . combining automated generation with human curation for benchmarking rag offers a synergistic approach to ensure both efficiency and quality in question sets. this hybrid approach leverages the strengths of both automated and human-driven processes, that provide efficient and robust evaluation metrics for rag's performance. in this work, we present a hybrid workflow to benchmark rags, which combines rapid question generation through automated methods, augmented with properly designed human prompts to generate diverse set of questions. our proposed benchmarking framework is used to generate questions from eis and other research documents related to environmental impact of wind energy projects. the extensive question-answer dataset serve as a tool to evaluate the performance of rag-based llms, which are designed to answer queries related to these extensive and comprehensive documents. given the vast amount of information contained in these documents, manually reviewing them is impractical, making rag-based llms essential for generating accurate responses to specific queries. our benchmarking framework assesses the effectiveness of these models in accurately retrieving and responding to queries, ensuring that they can reliably process and provide relevant information from the documents. contributions the paper introduces a novel benchmark dataset for question-answering (qa) task in a specific domain and also proposes a generic framework to evaluate the rag-based llm responses to different entries in the benchmark. this framework is designed to be adaptable across various domains, with a specific focus on documents related to wind energy project permitting in this study. the contributions of this research are as follows: ## related works there have been a lot of work in the field of benchmarking, particularly for question answering (qa) task. these can be broadly divided into general qa and domain-specific qa. general qa benchmarks. these benchmarks have established foundational evaluation frameworks for reading comprehension and knowledge retrieval tasks. notable general qa benchmarks include reading comprehension datasets such as the stanford question answering dataset (squad) @xcite and mctest @xcite , reasoningfocused benchmarks like the ai2 reasoning challenge (arc) @xcite , and comprehensive evaluation suites such as glue @xcite and big bench @xcite . additional benchmarks targeting open-domain knowledge include commonsenseqa @xcite , triviaqa @xcite , search qa @xcite , and newsqa @xcite . domain-specific qa benchmarks. recognizing the limitations of general benchmarks for specialized applications, researchers have developed domain-specific evaluation frameworks that capture the unique linguistic patterns, technical terminology, and reasoning requirements of particular fields. while scientific benchmarks such as mmlu @xcite , scibench @xcite , sciq @xcite , scirepeval @xcite , sciqa @xcite , and qasa @xcite are used for multi-disciplinary scientific qa evaluations, fieldspecific benchmarks include theoremqa @xcite for mathematics, emrqa @xcite for medicine, bioread @xcite and biomrc @xcite for bi-ology, lawbench @xcite for legal, and nuclearqa @xcite for nuclear domains. for environmental assessment specifically, benchmarks such as enviroexam @xcite for environmental science qa and nepaquad @xcite for environmental impact statement (eis) documents have emerged. however, to our knowledge, no benchmarks exist specifically for wind energy project permitting, making the proposed weqa benchmark the first comprehensive benchmarking effort in this critical domain. ## dataset creation in this paper, we focus on wind energy-related documents to enable the rag-based llms to answer questions specific to this field. we gather pdf documents, including research articles and environmental impact studies published by the department of energy (doe) under the national environmental policy act (nepa). accessing information from this vast database is not straightforward, necessitating the need for a trained llm to accurately retrieve and answer questions from the provided context. the challenge is to ensure that the model's responses are based on the actual documents and do not hallucinate information. by using rag-based llms, we aim to enhance the reliability and accuracy of responses related to wind energy, leveraging the rich information within our extensive document collection. this approach ensures that the information provided is both relevant and grounded in the sourced material. we constructed a data extraction and curation pipeline to extract text, image, and table information from wind energy-related documents as depicted in the 'data curation pipeline' in figure 1 . utilizing large language model (llm) based methods such as the unstructured.io tool @xcite , we efficiently extracted information and converted it into json elements. to ensure data quality, we implemented a filtering step to remove images without meaningful content, such as decorative elements or blank spaces. these filtered json elements were then organized into a schema, creating a page-wise assortment of text, table, and image elements. this structured format ensures that the extracted data is easily accessible and can be accurately referenced during model training and evaluation. ## methodology while past works have generally preferred to use crowdsourcing as a way to craft datasets and benchmarks @xcite , we choose to automated methods for benchmark question generation. automatically generating benchmarking questions using gpt-4 allows for efficient and scalable evaluation of other llms and rag. however, this approach can introduce errors, leading to poor quality of questions being generated. this makes it essential to incorporate a humanin-the-loop for reviewing and refining the questions and responses. this paper proposes hybrid approaches, where automated methods are combined with human curation to ensure the accuracy and reliability of the benchmarking process. by leveraging both machine and human expertise, we can achieve more robust and comprehensive benchmarking framework. figure 1 provides an overview of the proposed llm benchmarking framework. the core of the benchmarking framework is the question generation aspect, where automatic generation of questions forms the foundation. we combine this with human curation to select high-quality questions, ensuring relevance and clarity. corresponding answers to these questions are then validated by humans, establishing a reliable ground truth. this curated set of questions and validated answers is used to evaluate the responses of other llms and rag models. different question types. we generate multiple types of questions, including closed, open, comparison, evaluation, recall, process, and rhetorical questions. this diversity ensures a comprehensive benchmarking process, as each question type assesses different aspects of the models' capabilities. by incorporating a wide variety of questions, we can more effectively evaluate and compare the performance of llms and rag models across various dimensions. this approach provides a holistic view of their strengths and weaknesses. each of these question type evaluates different capabilities of the llm under test. open questions require models to generate detailed, free-form responses, testing their ability to construct coherent and informative answers. comparison questions ask models to compare and contrast different concepts or entities, assessing their analytical and comparative reasoning skills. evaluation questions require models to make judgments or provide assessments, gauging their ability to evaluate information critically. recall questions focus on the model's ability to retrieve and reproduce specific information from memory, testing their factual accuracy. process questions ask models to explain processes or sequences of actions, evaluating their understanding of procedures and logical progression. rhetorical questions are used to test the models' grasp of nuances in language and their ability to recognize and appropriately respond to questions that may not require direct answers. we present two complementary approaches for hybrid question generation to support comprehensive llm benchmarking. the hybrid prompt approach employs engineered prompts to generate high-quality, curated questions, while the hybrid context approach leverages text summarization to create questions that require broader contextual understanding. the detailed prompts used for question generation across both approaches are provided in the appendix. hybrid prompt approach. we utilize gpt-4 to automatically generate questions from given text chunks through carefully designed prompts tailored to each question type. to enhance question quality, we implement a manual curation process where domain experts identify exemplary questions that effectively assess llm capabilities for benchmarking purposes. this curation is performed systematically across all question types, ensuring that each category incorporates appropriate grammatical structures and complexity levels. these curated questions subsequently serve as few-shot examples to guide the automatic question generation framework, improving the overall quality and consistency of generated questions. hybrid context approach. the initial approach primarily generates questions at the sentence level by substituting subjects or objects with interrogative words, which proves adequate for 'closed', 'open', and 'recall' type questions where answers can be directly extracted from the text. however, 'process', 'evaluation', and 'comparison' questions require deeper inferential reasoning across larger text segments. to address this limitation, we first employ gpt-4 to summarize extensive text chunks (typically exceeding 15 sentences) into concise summaries containing 5-8 sentences. we then generate questions from these summarized chunks using the hybrid prompt methodology combined with curated sample questions, ensuring that the resulting questions necessitate comprehensive understanding and synthesis of broader contextual information. questions from tables. an essential component of benchmarking rag-based llms within research articles and reports involves evaluating their capability to retrieve and interpret tabular information. tables represent critical content elements within research documents, frequently containing comprehensive summaries and key quantitative data that encapsulate the essence of entire figure 2 illustrates the diverse question-answer pairs generated from the introduction section of a document @xcite using our proposed methodology. we demonstrate the hybrid context approach where the section content is first summarized into a concise form, and subsequently, targeted qa pairs are generated from this summarized context to ensure comprehensive coverage of key concepts. ## results and discussion experimental setup. we conduct a comprehensive evaluation of three state-of-the-art llms-gpt-4, gemini, and claude-on our weqa benchmark within a rag framework knowledge extraction is performed from wind energy documents to create vector embeddings as shown in the data-curation pipeline in figure 1 , which are subsequently stored in a vector database to enable retrieval-augmented generation capabilities. we employ the ragas evaluation framework, leveraging judge llms to provide systematic assessment of model performance across multiple dimensions. the evaluation encompasses key metrics including answer correctness, context precision, and context recall, offering comprehensive insights into each model's proficiency in both retrieving relevant information and generating accurate responses from the provided context. for the judge llm component, we utilize both gpt-4 and gemini-1.5pro to ensure robust and unbiased evaluation of the assessed models' performance. figure 3 presents the answer correctness score, while the context precision and context recall depicted in specifically, "evaluation" and "comparison" type questions yield nearly zero answer correctness scores for all models, highlighting their difficulty in responding. recall that, these challenging questions were crafted from summaries of text chunks rather than the text chunks themselves, further complicating the models' ability to generate correct answers. this underscores the complexity and rigor of the benchmarking process, emphasizing the need for models to improve their understanding and contextual extraction capabilities. observation 2 there is an alignment in evaluations made by the two judge llms used within the ragas framework, particularly visible for 'closed' type questions. this similarity arises because the answers to these questions are objective ('yes' or 'no'), leading to equivalent correctness evaluations by both models. although there are some mismatches in the evaluations made by the two judge llms, the number of these discrepancies is insignificant compared to the number of matching evaluations. figure 4 displays the confusion matrix illustrating the evaluations made by the two judge llms (gpt-4 and gemini-1.5pro) on the responses provided by the rag-based claude and gpt-4 models to the benchmarking questions. in this context, a true positive occurs when the judge llm correctly identifies the model response as matching the ground truth. conversely, a false positive arises when the judge llm incorrectly states that the model response matches the ground truth, while it does not. this matrix helps visualize the accuracy and reliability of the evaluations conducted by the llms, when used within the ragas framework. we note that majority of evaluations made by either judge llm matches the actual evaluation which indicates that both of them are reliable. observation 3 comparison between 'closed' and 'open' type questions within the same section reveals a higher answer correctness for responses to 'open' type questions than 'closed' type questions. from this observation, we conclude that ragbased models generate more accurate subjective responses to 'open' questions than objective ('yes' or 'no') responses for 'closed' questions. this phenomenon may stem from the inherent design of llms, which are optimized for generating extensive text sequences and may struggle with the precision required for definitive binary responses. this suggests that these models perform better when tasked with generating detailed, context-rich an-figure 3 : answer correctness scores computed using the ragas scoring framework with gpt-4 and gemini-1.5pro as judge models for response generated by all three models used. observation 5 the answer correctness scores for 'rhetorical' questions are lower than those for other question types. this is because 'rhetorical' questions lack definite answers in the documents, making it challenging for the models to retrieve the appropriate context and provide correct responses. the absence of clear, concrete answers in the source material complicates the models' ability to generate accurate and relevant responses, leading to lower correctness scores for this question type. observation 6 evaluations made by gemini-1.5pro on the responses generated by all three llms are higher than the evaluations made by gpt-4, with the responses from gemini llm receiving significantly higher scores. figure 5 shows the scores computed by the judge llms for the responses generated by the three rag-based llms. the gemini-1.5pro judge tends to rate high scores even when the llms refuse to answer. an example is listed below: evaluation comparison of llm generated answer question: in the html table that estimates the annual number of bird collisions at different percentages of avoidance, what is the estimated number of collisions per year at 98.0% avoidance? expected answer: the estimated number of collisions per year at 98.0% avoidance is 152. gemini generated answer: this question cannot be answered from the given source. while the text discusses bird collisions with wind turbines, it does not provide any data on estimated collision numbers at different avoidance rates. gpt-4 score: 0.194 gemini-1.5pro score 0.813 conversely, in instances where the llms generate correct answers, gemini-1.5pro has been observed to evaluate them as incorrect; such as: evaluation comparison of llm generated answer question: who is the gis technician in the 'study participants' table? expected answer: jr boehrs gemini generated answer: saif nomani jr boehrs was the gis technician. gpt-4 score: 0.703 gemini-1.5pro score: 0.200 ## conclusion in conclusion, this paper presents a versatile framework for evaluating the performance of rag-based llms across various question types and document sections. we showcase this by introducing a hybrid, automated question-generation method that ensures comprehensive coverage of both objective and subjective queries, and implement this for the use case of wind energy related document and present the weqa benchmark, which is a first of its kind benchmark in wind energy domain. however, the usefulness of our work goes beyond this niche domain as our approach is domain-agnostic, meaning it can be used for creating benchmark for any domain. additionally, our use of the ragas scoring framework allows for a thorough evaluation of model performance, offering a holistic assessment of llm capabilities, while also having the advantage of being easy for other researchers to adapt this approach for their own work.