Paper: DynaEval: A Dynamic Interaction-based Evaluation Framework for Assessing LLMs in Real-world Scenarios (ICLR-2024)

Reviewer comment (Writing):
Even though EGPI seems to be a central concept in their paper, the authors do not even explain what it is. They simply brush over it by adding a citation. I wouldn't expect more than a small minority of the audience at ICLR to be familiar with this concept.

Author response:
A1. a. Thank you for your question. We explained EGPI in our paper in p3 (after the citation), and introduced the formal definition of EGPI in Appendix A.2. Due to the page limitation, we did not place the definition and detailed explanation of EGPI in the main body of this paper.

Reviewer comment (Presentation):
The equations (1) and (2) seem totally out of place in the paper. Honestly, I'm not sure what role they play in explaining any of the concepts. Are they just there to explain the concept of sampling, or are they there simply to provide a "scare factor"?

Author response:
A1.c. Thank you for your doubt. The equations (1) and (2) aim to model and alleviate the uncertainty of LLMs’ performance in dynamic interaction-based evaluation, which is significant in LLM evaluation. As shown in our experiments, the performance of LLMs is unstable and can fluctuate given different interaction environments. Unlike traditional supervised signal-based evaluations where the environment (context) is static, the environment of dynamic interaction-based evaluation is itself dynamic, thus magnifying the uncertainty of LLMs’ performance. Therefore, it is indispensable to formally model the performance of LLMs with “uncertainty” and to alleviate the effect of uncertainty on evaluation results. As a result, rather than modeling the performance of LLMs by deterministic values, we find that modeling by random variables is more suitable for this problem and utilize equations (1) and (2) to present how our method models and alleviates the uncertainty of LLMs’ performance in the evaluation process.

Reviewer comment (Evaluation):
Looking at Figure 3. Both Figures 3.a and 3.b show a set of highly overlapping distributions. The medians of the top contenders in 3.a are barely distinguishable even, however the conclusion that section 3.2 notes is that GPT-4 performs best is mode 1. It seems to me like the authors are making claims without regard to statistical significance of their results.

Author response:
A2.a. Thank you for your constructive suggestion. Statistical significance can be used to test whether specific features of random variables (e.g., expectation) are different. In our paper, statistical significance is not used for two reasons. First, one indispensable assumption of most hypothesis testing methods (e.g., t-test and chi-square test) is that random variables should follow a family of distributions like normal distribution. Although this assumption is valid in many real-world scenarios (e.g., a person’s height), it is unconfirmed for the random variable of an LLM’s performance, and there is still a lack of research on the statistical property of LLMs’ performance. As a result, we cannot ensure whether the statistical significance itself is correct. Second, rather than to “rank the ability of different LLMs”, we aim to use DynaEval to “discover and depict the ability of different LLMs using evaluation”. Therefore, we presented the difference in LLMs’ performances in terms of average score and stability (the range of the box) in the text of Section 3.2. Following your constructive suggestion, we will discuss more about the performances of LLMs in the next version of our paper.

Reviewer comment (Evaluation):
The same is true in Table 2: the difference between GPT-4 (8.77) and ChatGPT (8.74) seems negligible, but that is not reflected in the conclusions of the text.

Author response:
A2.b. Thank you for your constructive suggestion. As explained in A2.a, we will supplement more discussion about performance of LLMs in Code G&R (especially Table 2) in the next version of our paper.

Reviewer comment (Novelty):
I'm not convinced that DynaEval framework itself brings anything to the table. I get the utility of using games as evaluation methods, which is a good idea depending on the implementation, but the framework itself doesn't quite bring anything to the table IMO.

Author response:
A3.a. Thank you for your doubt. As you mentioned, using games as evaluation methods depends on the implementation. One challenge in this process is that games can differ a lot in many aspects such as the number of participants, role design, game rule, etc., and are hard to standardize. As a result, many existing works about LLMs’ performance in games (e.g., [1], [2]) only focus on a specific game and cannot be extended to other games or scenarios. Indeed, one contribution of DynaEval is that it provides a standardized modeling method of all kinds of games using “interaction goals” and “interaction rules” and provides a general framework for running all these games and evaluating the ability of LLMs, as mentioned in A1.b. With this framework, researchers need not waste their time on designing a specific system for running a specific game, but focus more on the properties of the game itself such as what to evaluate (which can be reflected in the interaction goals and interaction rules).

Reviewer comment (Experiments):
The games seem odd, and it's not obvious what capability of the model they are trying to measure. For example, I'm not even sure it makes sense for the Code G&R and the Machine Translation games to actually be turn-based games. As I said, I do see why games can be useful, but the devil is in the details.

Author response:
A3.b. Thank you for your doubt. As mentioned above, DynaEval is a general framework for running all kinds of games and evaluating LLMs, and “what ability to measure” depends on **the design of games**, which is decided by users of DynaEval. Generally, we can categorize the abilities of LLMs into “fundamental” abilities and “domain-specific abilities”. Fundamental abilities are necessary for all kinds of dynamic interactions between LLMs, such as the ability to understand and reasoning. Domain-specific abilities are required by specific domains or tasks, such as the ability to generate logically correct and well-styled codes in code generation. The design of evaluation tasks can decide the domain-specific abilities to evaluate. For example, as introduced in Section 2.4, p6 in our paper, the Code G&R evaluates the code generation ability of programmer LLMs and the code review ability of reviewer LLMs. In real-world scenarios, users of DynaEval can design their own evaluation tasks to evaluate their required domain-specific abilities of LLMs.

Reviewer comment (Evaluation):
The proposed DynaEval framework lacks a specific evaluation goal since the "ability of interaction" is too big. DynaEval focuses on the interaction BETWEEN LLMs, whereas the "ability of interaction" of LLMs should focus more on the interaction BETWEEN LLMs and HUMAN USERs. In other words, the more relevant concern is whether an LLM could understand user intentions and help to execute tasks. Therefore, the ability of interaction should include the ability of understanding, questioning, reasoning, or even tool using. It is unclear what specific ability DynaEval can evaluate. While the authors may argue that DynaEval is a general framework to evaluate all of these abilities, this may not address the core challenge of evaluating LLMs.

Author response:
Thank you for the question. We agree with the opinion that LLM evaluation should focus on specific abilities. As for DynaEval, what should be noticed is that **the ability to evaluate depends on the design of the evaluation task, which is determined by the user of DynaEval**. On the other hand, DynaEval itself provides a platform/framework for users to design and implement their evaluation task that evaluates the task-specific abilities of LLMs in dynamic interaction scenarios. For example, in our experiment, the Code G&R evaluates the code generation ability of the programmer LLM and the code review ability of the reviewer LLM. The reviewer may doubt that this task cannot disentangle the understanding and reasoning ability of LLMs because they are also required by this task. However in real-world dynamic interaction scenarios, LLMs always need to use multiple abilities to solve users’ problems, and the understanding and reasoning ability is the basis of them because they need to optimize their solution based on their understanding and reasoning of user feedback. Therefore, as Code G&R and Machine Translation, users of DynaEval can design their own evaluation tasks to evaluate task-specific or domain-specific abilities of LLMs. In our paper, we also introduced in **Appendix A.4** how to design evaluation tasks based on users’ own requirements.

Reviewer comment (Novelty):
There appears to be little novelty in DynaEval except for the referee component. The paper attempts to standardize the interaction process and emphasizes fairness and stability, proposing a synchronous interaction algorithm that ensures anonymity and multiple independent runs. However, many of these considerations are common sense in evaluation tasks. For example, when evaluating several recommendation models, it is natural to use the same test set (fairness) and to run multiple times (stability). Therefore, the novelty of DynaEval is questionable.

Author response:
Thank you for your question. Indeed, fairness and stability are only easy to achieve in traditional supervised signal-based evaluation or human-based evaluation because the response of each LLM is **independent of and isolated from others**. However, In the dynamic interaction between LLMs, things are much more complicated. LLMs might expose their real identity because of a lack of supervision, and the synchronicity is easy to be violated because the response time of different LLMs can vary a lot. Moreover, the synchronicity also depends on game rules. For example, in the public goods game, LLMs are required to respond simultaneously in each round. But in Code G&R, LLMs are required to respond one by one. In real-world scenarios, the game rule can be much more complex. As a result, to standardize the complex interaction process, we propose the **referee** to help guarantee the fragile but significant anonymity and rate LLMs’ performance, and we propose the **message pool** which uses the synchronous interaction algorithm to compulsively guarantee the synchronicity of the dynamic interaction process.

Reviewer comment (Theory):
The paper builds a close relation between DynaEval and extensive-form games with perfect information (EGPI), but the necessity of this connection is unclear. The authors claim that game theory helps to overcome fairness and stability issues by introducing anonymity and synchronicity, but this is unconvincing because anonymity and synchronicity are not exclusive to game theory. Since EGPI can describe a broad range of interaction scenarios, it is not surprising that DynaEval's procedures fit into EGPI. However, the connection would be more interesting if DynaEval shared specific characteristics of EGPI, such as the existence of equilibrium.

Author response:
Thank you for your question. As mentioned in A2, the dynamic interaction process between LLMs is much more complicated than the traditional supervised signal-based or human-based evaluation process. As a result, we utilize the EGPI to standardize the interaction process in all kinds of tasks and scenarios and introduce the fairness and stability condition of DynaEval. As for shared properties of DynaEval and EGPI such as the existence of equilibrium (Do you mean Nash equilibrium?), since we have proved that DynaEval belongs to EGPI, it is theoretically easy and convenient to analyze the property of DynaEval in terms of game theory. As this paper mainly focuses on LLM evaluation in dynamic scenarios, such an analysis exceeds the range of research of this paper. We plan to analyze the theoretical properties of DynaEval in our next research.

Reviewer comment (Evaluation):
In the example of IDIOMS SOLITAIRE, it is unclear which specific ability is being tested. From one perspective, the winning LLM may simply have encountered more idioms previously, allowing it to answer easily. It is questionable whether this relates to "interaction".

Author response:
As introduced in Section 2.4, p6 in our paper, we test the ability to retrieve and use the Chinese vocabulary of LLMs. Interaction is significant in this task, because LLMs’ responses are based on the context, and Chinese idioms are limited and it is easy for LLMs to acquire a host of idioms. If two participants have seen a similar amount of idioms, they need to use specific strategies to win the game, such as coming up with a Chinese idiom with the last character complex and rare, which makes it hard for the opponent LLM to come up with a valid Chinese idiom, even though it has seen a lot of Chinese idioms.

Reviewer comment (Novelty):
Overall, I feel the technical contribution of this paper is limited because the method fails to come up with a practical and novel fairness condition or stability condition. The presented fairness condition and stability condition are both standard practice, with limited new insights. Anonymity, synchronicity, and independent runs are all typical practices during LLM evaluation. As the method is tackling dynamic interaction of LLMs, I do not see any specific part of the algorithm being customized to the specific property of dynamic interaction. Also, I feel the concepts of referee and message pool are not novel either.

Author response:
Thank you for your doubt. The technical contribution of this paper is that: 1. We noticed and proved the relationship between the dynamic interaction process of LLMs in real-world scenarios and EGPI; 2. Based on EGPI, we proposed DynaEval which is a **general** framework for dynamic LLM evaluation that can cover most interaction processes in real-world scenarios, which is much more complex than traditional signal-based evaluation methods. Anonymity, synchronicity, and independent runs are fundamental yet indispensable in all dynamic interaction-based LLM evaluation. However, they are only easy to achieve in traditional supervised signal-based evaluation or human-based evaluation because **the response of each LLM is independent of and isolated from others**. In the dynamic interaction between LLMs, things are much more complicated. **LLMs might expose their real identity because of a lack of supervision, and the synchronicity is easy to be violated because the response time of different LLMs can vary a lot. Moreover, the synchronicity also depends on game rules.** For example, in the public goods game, LLMs are required to respond simultaneously in each round. But in Code G&R, LLMs are required to respond one by one. In real-world scenarios, the game rule can be much more complex. As a result, we propose the **referee** to help guarantee the fragile but significant anonymity and rate LLMs’ performance, and we propose the **message pool** which uses the synchronous interaction algorithm to compulsively guarantee the synchronicity of the dynamic interaction process.

Reviewer comment (Evaluation):
The discussion on the proposed evaluation method compared to existing ones is quite weak. It is unclear how the proposed one can tackle the dynamic nature of the LLMs interaction while the existing evaluation methods cannot.

Author response:
As presented in **the third paragraph of Section 1**, we discussed the reason why existing evaluation methods cannot tackle the dynamic nature of the LLMs interaction. In the fourth paragraph, we further illustrated how the proposed DynaEval can solve this problem. For traditional supervised signal-based methods, their evaluation environments are static, and hard to capture the complexity of dynamic real-world situations. Moreover, although some evaluation methods support multiple-round conversations, they still fall short in providing a dynamic interaction environment and lack generality.