AHP -Powered LLM Reasoning for Multi-Criteria Evaluation of Open-Ended Responses (2024)

Abstract: Question answering (QA) tasks have been extensively studied in the field of natural language processing (NLP). Answers to open-ended questions are highly diverse and difficult to quantify, and cannot be simply evaluated as correct or incorrect, unlike close-ended questions with definitive answers. While large language models (LLMs) have demonstrated strong capabilities across various tasks, they exhibit relatively weaker performance in evaluating answers to open-ended questions. In this study, we propose a method that leverages LLMs and the analytic hierarchy process (AHP) to assess answers to open-ended questions. We utilized LLMs to generate multiple evaluation criteria for a question. Subsequently, answers were subjected to pairwise comparisons under each criterion with LLMs, and scores for each answer were calculated in the AHP. We conducted experiments on four datasets using both ChatGPT-3.5-turbo and GPT-4. Our results indicate that our approach more closely aligns with human judgment compared to the four baselines. Additionally, we explored the impact of the number of criteria, variations in models, and differences in datasets on the results.

## introduction large language models (llms) have shown remarkable capabilities in a wide range of tasks, such as natural language generation @xcite , summarization @xcite , translation @xcite ) and text classification @xcite . question answering (qa) tasks have been widely studied and can be utilized to assess the breadth of knowledge and logical comprehension abilities of llms. these tasks are designed to probe the models not only for their factual accuracy but also for their ability to infer answers from complex queries, thereby reflecting their understanding of both context and content. most studies on qa tasks focus on close-ended questions @xcite @xcite , which have definitive correct answers. for example, the following are three close-ended questions @xcite : q: who played will on as the world turns? (a: jesse soffer.) q: how many episodes in season 2 breaking bad? (a: 13.) q: if a person who always tells lies tells you that the destination is on the left, should you go left or right? (a: right.) answering these questions requires knowledge and logical reasoning capabilities in a variety of topics. the answers to these questions are often unique and clear, answering them does not necessarily require creativity. on the other hand, open-ended questions are more vague and abstract, and do not have clearly correct answers; even experts with specialized knowledge will need to give consideration before giving an answer. examples of open-ended questions are q: what are the best practices for conducting load tests on applications that expose rest apis? q: what is a feature you would not like in your product? q: how can we prevent participants from arriving late to weekly meetings? the answers to these questions can vary widely. for instance, for the question, "how can we prevent participants from arriving late to weekly meetings?", the following are three possible answers: a: setting reminders: set alerts or reminders 15 or 30 minutes before the meeting. a: time verification: ensure that the meeting time is convenient for everyone and adjust if necessary. a: agenda sharing: share the agenda the day before the meeting, allowing partici-pants to prepare. open-ended questions are essential for generating new ideas, encouraging creativity, and facilitating deeper understanding, but require detailed and multi-faceted consideration for evaluation. the superiority or inferiority of an answer depends on which evaluation criteria are used, and an overall decision must be made by synthesizing evaluations based on different evaluation criteria, which can be a very complex decision-making task. in this study, we consider the use of llms to evaluate responses to open-ended questions. in order to have llms evaluate answers from multiple perspectives, we focus on the analytic hierarchy process (ahp) @xcite , which is known as a mathematical decision-making tool in operations research, and execute the decisionmaking procedure using llms. ahp evaluates candidates according to multiple evaluation criteria and makes a final decision by weighting them according to the relative importance of the evaluation criteria. by having llms perform the ahp process, we expect to enhance the ability of llms to evaluate multiple aspects objectively. specifically, our proposed method could be divided into two phases: the criteria generation phase and the evaluation phase. we first have an llm generate multiple evaluation criteria for answers to a question and determine the relative importance of these criteria through pairwise comparisons in the criteria generation phase. we then perform pairwise comparisons of candidate answers for each criterion in the evaluation phase. finally, these results are integrated to arrive at a final decision. our results demonstrate that multiple criteria can effectively evaluate answers to open-ended questions. we conducted experiments on 4 datasets on chatgpt-3.5-turbo and gpt-4. compared to baseline methods, our approach performs better on quantitative indicators, concordance index, and soft concordance index. the contributions of this study are summarized as follows: svoboda and lande (2024) utilized the ahp and gpt-4 to generate answers for open-ended questions as an automated decision-making process. however, the proposed method lacked validation and quantitative evaluation metrics. given the inherent uncertainty of open-ended questions, quantitative evaluation poses significant challenges. our research focuses on developing and implementing quantitative evaluation metrics for our proposed methodology and evaluating the effectiveness of the combined use of ahp and gpt-4 in answering open-ended questions. similarly, uto and uchida (2020) proposed the use of lstm networks for scoring answers of students to open-ended questions. such supervised learning methodology could properly assess the ability of students but might not facilitate the discovery of new knowledge or identify better answers to open-ended questions without prior knowledge. the presence of ground truth labels implies that the superior answers are predetermined, limiting the potential for innovation in question answering. our proposed method can be directly applied to llms without the need for training or ground truth labels. by quantitatively assessing the responses to open-ended questions, our approach allows for the discovery of new insights. ## conclusion in this study, we proposed an evaluation approach to evaluate open-ended responses by ahp-powered llm reasoning based on multiple different criteria. we conducted experiments on four datasets and with two types of llms, and we used quantitative indicators, concordance index, and soft concordance index, to demonstrate that our method outperforms the baselines empirically. three takeaways can be summarized as follows based on the results.