A Methodology for Identifying Evaluation Items for Practical Dialogue Systems Based on Business-Dialogue System Alignment Models (2025)

Abstract: This paper proposes a methodology for identifying evaluation items for practical dialogue systems. Traditionally, user satisfaction and user experiences have been the primary metrics for evaluating dialogue systems. However, there are various other evaluation items to consider when developing and operating practical dialogue systems, and such evaluation items are expected to lead to new research topics. So far, there has been no methodology for identifying these evaluation items. We propose identifying evaluation items based on business-dialogue system alignment models, which are applications of business-IT alignment models used in the development and operation of practical IT systems. We also present a generic model that facilitates the construction of a business-dialogue system alignment model for each dialogue system.

## introduction traditionally, in the dialogue systems research community, user satisfaction @xcite @xcite and user experience @xcite @xcite have been widely used as metrics for evaluating dialogue systems. with recent advancements in dialogue system technology, particularly the development of large language models (llms), it has become possible to develop dialogue systems with high scores in these metrics @xcite . however, in developing and operating practical systems, it is necessary to consider various factors other than the aforementioned metrics. for instance, a chatbot using retrieval-augmented generation (rag) @xcite can generate natural responses based on the contents of a database, but there is still a possibility of generating responses that are inconsistent with the database contents. therefore, there are risks associated with using such a system for customer service. additionally, when using an llm on one's own hardware, substantial hardware resources are required, resulting in high running costs. consequently, if the anticipated benefits do not exceed these costs, it is difficult to continue operating the system. in addition to llms, various new technologies have been proposed for dialogue systems, but not all are used in practical systems. we suspect that one reason for this is the difference between the evaluation metrics used in the research community and those used to evaluate practical systems. so it is crucial to identify evaluation items for building and operating practical systems. @xcite and mctear (2004) mention requirements for dialogue systems in explaining dialogue systems development life cycles. @xcite discusses the need for considering requirements from not only users but also operators, but how to list all the requirements is not discussed. @xcite categorize evaluation items for dialogue systems from the system owner's perspective into benefits, costs, and risks, and they include items that do not have a positive correlation with user satisfaction or user experience. however, the methodology for identifying all evaluation items for individual dialogue systems has not been presented. in this paper, we apply business-it alignment models @xcite to dialogue systems. business-it alignment models are widely used to link business goals, business processes, and applications to facilitate the examination and evaluation of business systems by various stakeholders. we call the results of the application of business-it alignment models to dialogue systems business-dialogue system alignment models (hereafter business-ds alignment models). by applying these models to individual dialogue systems to create a business-ds alignment, it becomes possible to list evaluation items specific to each dialogue system. furthermore, to facilitate the creation of the business-ds alignment model for an individual dialogue system, this paper proposes a generic model for business-ds alignment. by applying this generic model to individual dialogue systems, it is possible to create an alignment model tailored to each system, which can then be used to identify the corresponding evaluation items. it should be noted that, while this paper uses the term business, it is not limited to the narrow sense of business. instead, it encompasses all practical dialogue system development and operation. for example, the same analytical approach can be applied to systems developed and operated by nonprofit organizations or local governments. ## limitations and discussion although the case studies suggested that our approach is promising, there may be values, risks, and costs that have not been considered, necessitating continuous review. particularly with advancements in technology like llms, which enable more natural conversations, new risks that were previously unconsidered may arise. as stated earlier, academic research has often used user satisfaction and user experience as evaluation metrics. roughly speaking, user satisfaction relates to functional value, self-expressive value, and social value. user experience relates to emotional value, must-be value, attractive value, nonmaleficence, justice and fairness, and transparency. our analysis identified evaluation items beyond these, so it became possible to consider user satisfaction, user experience, and other evaluation items all at once. we hope this leads to new research themes. in planning the actual system development, it is necessary to balance values, risks, and costs. for example, while showing many advertisements might increase business value, it could decrease emotional value and pose risks to hinder nonmaleficence. similarly, using a low-performance model to reduce costs can decrease must-be value. a balanced system design considering all evaluation items is necessary, and our approach enables such a balanced design by identifying evaluation items from various perspectives. in some cases, it is desirable to integrate these evaluation items into a single-dimensional evaluation scale. however, the prioritization of these items must be determined by the consensus of various stakeholders, including the system owner. we hope business-ds alignment models help the facilitation among the stakeholders. the evaluation items obtained using the methodology proposed in this paper do not necessarily allow for a quantitative assessment of dialogue systems. however, in many cases, various it-related technologies are proposed and utilized without quantitative evaluation. in addition, focusing only on quantifiable evaluation items and ignoring other items have the risk of falling into the well-known mcnamara fallacy (also known as the quantitative fallacy). we believe that instead of focusing solely on fields where quantitative evaluation is feasible through small-scale experiments, dialogue system researchers should also consider evaluation items that are difficult to quantify. this approach may lead to the development of more practical technologies. while it is practically impossible to quantitatively demonstrate the superiority of our methodology, we aim to showcase its effectiveness by applying it to the development of a variety of practical dialogue systems and evaluating it from multiple perspectives. business-it alignment models on which our methodology is based may not be familiar to dialogue system engineers, making it potentially challenging to construct a business-ds alignment model. therefore, we believe it is effective to present a simpler model. as an alternative approach, it is also possible to consider developing human resources who can construct business-ds alignment models while communicating with various stakeholders. ## concluding remarks this paper proposed a methodology to identify evaluation items for dialogue systems based on business-ds alignment models. although the methodology presented in this paper needs improvement through more case studies. nevertheless, we believe that it serves as a useful first step. besides the future work already mentioned, we plan to analyze the issues that prevent commercializing systems in the research stage.