M inos E val: Distinguishing Factoid and Non-Factoid for Tailored Open-Ended QA Evaluation with LLM s (2025)

Abstract: Open-ended question answering (QA) is a key task for evaluating the capabilities of large language models (LLMs). Compared to closed-ended QA, it demands longer answer statements, more nuanced reasoning processes, and diverse expressions, making refined and interpretable automatic evaluation both crucial and challenging. Traditional metrics like ROUGE and BERTScore struggle to capture semantic similarities due to different patterns between model responses and reference answers. Current LLM-based evaluation approaches, such as pairwise or listwise comparisons of candidate answers, lack intuitive interpretability. While pointwise scoring of each response provides some descriptions, it fails to adapt across different question contents. Most notably, existing methods overlook the distinction between factoid and non-factoid questions. To address these challenges, we propose MinosEval , a novel evaluation method that first distinguishes open-ended questions and then ranks candidate answers using different evaluation strategies. For factoid questions, it applies an adaptive key-point scoring strategy, while for non-factoid questions, it uses an instance-aware listwise ranking strategy. Experiments on multiple open-ended QA datasets, including self-built ones with more candidate responses to complement community resources, show that MinosEval better aligns with human annotations and offers more interpretable results.

## introduction open-ended question answering (qa) is a fundamental task type in comprehensive large language modeling (llm) evaluation benchmarks and plateforms @xcite @xcite @xcite ). unlike closed-ended questions, * equal contribution. in recent years, evaluation methods have shifted toward leveraging llms for automatic evaluation. these methods typically involve employing pairwise comparisons for elo @xcite , listwise comparisons where llms rank responses based on relevance or quality, and pointwise approaches @xcite that score responses on predefined dimensions. meanwhile, some studies conducted pairwise and pointwise supervised fine-tuning to obtain dedicated llms for evaluation @xcite . while promising, these methods have notable limitations. pairwise methods involve multiple comparisons, leading to o(n 2 ) complexity for n responses. pointwise methods, though scoring on predefined criteria, e.g., "fluency" and "truthfulness", often fail to be adapted to each specific question context. additionally, pairwise and pointwise methods suffer from the tie-breaking problem. in addition, pairwise and listwise methods lack intuitive explanations, making their rankings difficult for human evaluators to interpret. more notably, these llm-based approaches overlook the distinction between factoid and nonfactoid questions in open-ended qa. factoid @xcite and nonfactoid @xcite questions exhibit a clear distinction. factoid questions aim to elicit answers based on objective, real-world facts or entities, with a relatively fixed and clear scope, e.g., "list the five highest mountains on earth and the average elevation of their peaks". of course, additional sentences can be included in the answer to ensure semantic coherence. in contrast, non-factoid questions offer more creative freedom, with the primary requirement being to meet the demands of the question without necessarily including critical, fact-based information, e.g., "please help me write homophone joke". by failing to account for this distinction, such a one-size-fits-all strategy undermines the precision and effectiveness of evaluation. in this paper, we propose minoseval, a novel two-stage approach for evaluating open-ended qa and providing corresponding interpretive information. it distinguishes between factoid and nonfactoid questions based on their semantic and content differences, employing tailored evaluation strategies for each type of question. for factoid questions, we use an adaptive key-point scoring strategy that extracts key factoid points from a given reference answer. it then compares how each response entails these key points using a natural lan-guage inference (nli) model, inspired by @xcite . for non-factoid questions, we apply an instance-aware listwise ranking approach, generating five levels of silver answer instances to enhance the llm's performance in directly ranking these more creative and diverse responses. our approach tackles the challenges of existing methods in open-ended qa evaluation and offers several advantages. first, it distinguishes between different types of open-ended questions, enabling a more tailored evaluation strategy. it also provides clear ranking guidelines based on the characteristics of factoid and non-factoid questions, including key points and silver instances to enhance interpretability. moreover, the guidelines are adaptive to the specific questions, and the entire process is fully automated, requiring no manual intervention. we conducted extensive experiments on four datasets. following the approach of @xcite , we performed five sets of sampling to construct antique_s5 and trec-dl-nf_s5 from the publicly available open quality assurance datasets antique @xcite and trec-dl-nf @xcite . additionally, we created two self-built datasets, align-bench_minos and gaokaobench_minos, which contain a larger number of candidate responses based on alignbench @xcite and gaokaobench @xcite to supplement the resources of the research community. the results demonstrate that our method outperforms existing llm-based evaluation approaches for openended qa. we also present case studies in which the key points for factoid questions and the silver answer instances for non-factoid questions provide a valuable interpretive foundation. we hope this work contributes to the effective evaluation of llms' performance in open-ended qa and promotes further research within the llm community. our datasets, evaluation results, and code for minoseval are publicly available 1 . question answering (qa) is a key task in natural language processing, aimed at providing accurate answers to satisfy the user's information need or request. depending on the scope of the answer, qa systems are typically divided into closedended and open-ended types. closed-ended qa includes formats like multiple choice @xcite , true/false @xcite , and close question @xcite ), while open-ended qa allows for more diverse, unbounded responses, typically divided into factoid and non-factoid questions @xcite . common tasks in open-ended qa include reading comprehension @xcite , summarization @xcite , and writing @xcite . the advent of large language models (llms) such as chatgpt (openai, 2022) and gemini @xcite ) has made open-ended qa a popular task in llm evaluation benchmarks. this task is essential for evaluating the ability of llms to generate complex, creative, and contextually relevant responses, which require advanced reasoning. moreover, open-ended qa has been used to explore llm limitations in understanding nuanced language @xcite and capturing human intent and emotions @xcite , further highlighting its importance as a benchmark for evaluating reasoning and cognitive abilities @xcite ). 1 https://github.com/johnny-fans/ minoseval 2.2 open-ended qa evaluation traditional methods: matching-based evaluation methods for open-ended qa include bleu @xcite and rouge @xcite . these methods primarily focus on matching the n-grams between the generated and reference texts, which often overlooks the semantic meaning. for example, @xcite highlights that rouge is ineffective for long-text qa tasks. lm-based methods aim to capture semantic similarity better. bertscore @xcite easuring semantic similarity between embeddings. further work introduces bem @xcite , a bertbased model for assessing semantic equivalence between candidate and reference answers, and pedants @xcite , which employs rule-guided rubrics and lightweight neural scoring. however, they struggle with complex questions and diverse answers. human evaluation: human evaluation remains the golden standard in open-ended qa @xcite , providing more accurate and comprehensive feedback @xcite . however, it is resource-intensive and difficult to scale, limiting its use in large-scale evaluations. ## methodology in this section, we begin by formally defining the open-ended qa evaluation task, explaining the principles of distinguishing between factoid and non-factoid questions, and then presenting the details of minoseval. ## conclusion in conclusion, we present minoseval, an llmbased approach for open-ended qa evaluation. by distinguishing between factoid and non-factoid questions, minoseval tailors evaluation strategies to suit each type, employing adaptive key-point scoring for factoid questions and instance-aware listwise ranking for non-factoid ones. we constructed two datasets with more candidate responses and greater difficulty, and conducted experiments across four datasets. the results validate the effectiveness of the individual modules, demonstrate the robustness of the method, and analyze its computational efficiency. minoseval shows improved alignment with human annotations and offers more interpretable results. we plan to opensource this project in hopes of contributing to more effective llm evaluation research.