Paper: DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks (ICLR-2024)

Reviewer comment (Theory):
The claim on "co-evolution" is not clear. I do not quite understand what co-evolution means. It seems that the evaluation process is not dependent on the LLM, then how they are correlated from each other.

Author response:
### Explain "co-evolution"

We apologize for any misunderstanding of the term "co-evolution". By this term, we mean that evaluation protocols should also evolve to be more complex to test the rapid development of LLMs. We noticed that current benchmarks remain **static**, i.e., most of them will not be updated once they are released to public; while it is the fact the LLMs do keep evolving (the network becomes larger, with more training data). So, the dilemma happens: LLMs are becoming stronger while evaluation benchmarks stays static, leading to an unfair evaluation.

Therefore, we design DyVal to be not only a dynamic benchmark, but also a **flexible** one that can evolve with the development of LLMs. For example, as shown in the following table, both ChatGPT and GPT-4 achieved 100% accuracy in math and logic reasoning tasks in BigBench. It can also be observed that in our experiments: in the simplest arithmetic and boolean logic tasks, the performance gap between GPT-4 and ChatGPT is marginal. Yet, as the complexity of tasks increases, this gap becomes increasingly pronounced. In our framework, DyVal can evolve the evaluation benchmarks by increasing the complexity of tasks, preventing them from plateauing in performance. As shown in Sec. 4.2, the performance of ChatGPT significantly drops as tasks becoming more challenging. This observation validates the effectiveness of DyVal in providing a evolving test bed for LLMs.

Reviewer comment (Experiments):
The data contamination problem is not clear. Notably, the data generated by the proposed method is rather limited type as it can not generate narrative generation tasks and others related to common sense. I am wondering how the existing datasets have the contamination problem. I think such a problem may not happen frequently in the logical reasoning and algorithm domains (especially, these abilities may be majorly from finetune from code and scientific papers). However, they are much easier to happen on those storytelling data.

Author response:
### Present data contamination in existing benchmarks
>The data contamination problem is not clear. Notably, the data generated by the proposed method is rather limited type as it can not generate narrative generation tasks and others related to common sense. I am wondering how the existing datasets have the contamination problem. I think such a problem may not happen frequently in the logical reasoning and algorithm domains (especially, these abilities may be majorly from finetune from code and scientific papers). However, they are much easier to happen on those storytelling data.

We answered this question in the general response about data contamination. Now we give more references to show how serious this problem is.

There are already several studies [1,2,3,4,5,6,7,8] (before and after our work) demonstrated the data contamination issue. Further, our observations reveal a notable performance gap between LLMs like phi-1.5 [9], Xwin-13B [10], wizard-math [11] on standard benchmarks (e.g., GSM8K [12]) and the arithmetic tasks generated by our DyVal framework. Despite their reported high performance on  GSM8K, these models exhibit significantly lower efficiency on DyVal tasks. This discrepancy points towards potential data contamination in existing benchmarks, where LLMs might have been inadvertently trained on test set data, leading to artificially inflated performance. As outlined in Section 3.2.1, DyVal's dynamic generation process is designed to prevent any overlap with training datasets, thereby providing a more accurate assessment of LLM capabilities.

Finally, existing evidence has shown that the logical and math problems are also having the contamination problem (e.g., the training data of phi-1.5 contains a lot of such reasoning tasks). This shows the necessity of our work.
Moreover, we do not limit ourselves in reasoning tasks only. As shown in the general response, our framework is flexible to other NLP tasks, which may undergoes more serious data contamination.

[1] GPT-4 performs significantly worse on coding problems not in its training data. https://brianlovin.com/hn/35297067

[2] No, GPT4 can’t ace MIT. https://flower-nutria-41d.notion.site/No-GPT4-can-t-ace-MIT-b27e6796ab5a48368127a98216c76864

[3] Kocoń, Jan, et al. "ChatGPT: Jack of all trades, master of none." Information Fusion (2023): 101861.

[4] Zečević, Matej, et al. "Causal parrots: Large language models may talk causality but are not causal." Transactions on Machine Learning Research 2023.

[5] Wei, Tianwen, et al. "Skywork: A More Open Bilingual Foundation Model." arXiv preprint arXiv:2310.19341 (2023).

[6] Zhou, Kun, et al. "Don't Make Your LLM an Evaluation Benchmark Cheater." arXiv preprint arXiv:2311.01964 (2023).

[7] Li, Yucheng. "An Open Source Data Contamination Report for Llama Series Models." arXiv preprint arXiv:2310.17589 (2023).

[8] Golchin, Shahriar, et al. "Data Contamination Quiz: A Tool to Detect and Estimate Contamination in Large Language Models." arXiv preprint arxiv:2311.06233 (2023).

[9] Li, Yuanzhi, et al. "Textbooks are all you need ii: phi-1.5 technical report." arXiv preprint arXiv:2309.05463 (2023).

[10] Luo, Haipeng, et al. "Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct." arXiv preprint arXiv:2308.09583 (2023).

[11] Xwin-LM. https://github.com/Xwin-LM/Xwin-LM

[12] Cobbe, Karl, et al. "Training verifiers to solve math word problems." arXiv preprint arXiv:2110.14168 (2021).

Reviewer comment (Reproducibility):
The potential bias may exist in the graph generation. The paper focuses on how to conduct constraints for the graph to avoid illegal ones. Nonetheless, there may be lacked of details on how the graph is generated to meet those constraints. I am concerned that the graph generation algorithms remain biased. Therefore, there will be bias in the generated text, leading to the potential issue.

Author response:
### Lacked of details for constraint construction and potential bias in evaluation
> The potential bias may exist in the graph generation. The paper focuses on how to conduct constraints for the graph to avoid illegal ones. Nonetheless, there may be lacked of details on how the graph is generated to meet those constraints. I am concerned that the graph generation algorithms remain biased. Therefore, there will be bias in the generated text, leading to the potential issue.


Thanks for your valuable concerns. We have submitted the code in the revision. As an example, to prevent illegal operations like division by zero in arithmetic tasks, we implement a detection and regeneration mechanism. If a node representing divident, e.g., 'C' in a division operation 'A=B/C', is initially zero, it is automatically regenerated until a non-zero value is obtained. 

```

   A(\)                          A(\)
   / \      -> Regenerate C      / \        
B(3) C(0)                     B(3) C(1)
```


We would greatly appreciate further details about the "potential bias". If the potential biases here refers to issues of dataset imbalance, as pointed by Reviewer iwJv, our algorithm can easily satisfy the balance requirement by meticulously controlling the flexible dynamic generation process. For example, in reachability task, we can drop the generated evaluation samples with 'False' labels until we generate a sample with 'True' label. In fact, we presented the **results of ChatGPT and GPT4 in balanced datasets** in Table 1. The results in balanced datasets are similar to our initial findings: (1) ChatGPT consistently predicted all questions as "True.", resulted in a uniform accuracy rate of 50%. (2) GPT-4 demonstrated excellent performance. It maintained significantly higher accuracy rates across all complexity levels.

| **Model**      | **ChatGPT** |   |   |   | **GPT4** |   |   |   |
|----------------|-------------|-------|-------|-------|----------|-------|-------|-------|
| **Complexity** | D1          | D2    | D3    | D4    | D1       | D2    | D3    | D4    |
| **Balanced**   | 50          | 50.00 | 50.00 | 50.00 | 84.54    | 79.03 | 73.50 | 72.41 |
| **Imbalanced** | 63.87       | 54.27 | 53.73 | 52.33 | 85.33    | 83.00 | 67.67 | 74.33 |

Reviewer comment (Theory):
A common challenge associated with this framework is the need to manually specify a problem as a computation graph with valid constraints. This requirement is only understandable if LLM is intended to acquire specific skills written in these formats.

Author response:
This relates to the importance of reasoning tasks in LLM and the flexibility of our approach. We have answered this question in the general response. Now we briefly answer it.
Reasoning ability is widely recognized as the core of both human and artificial intelligence, our focus on constructing reasoning tasks mirrors the intricate and multi-step nature of human reasoning [1, 2, 3], building reasoning benchmarks is a crutial step to help LLMs towards intelligence. 
Furthermore, our proposed dynamic evaluation protocol, including graphs and trees, can also be applied to other tasks such as natural language understanding (NLU) tasks. Take sentiment analysis as an example, We can abstract a sentence into a syntax tree and form tree templates which can be used to generate diverse sentences. Take 'I am happy today' as an instance, it can be formalized as:
    
```
     Subject (I)
         |
     Predicate
    /        \
Verb (am)   Complement
            /        \
  Adjective (happy)  Temporal Phrase (today)
```
    

Following this template, we can create variations by replacing the corresponding words, like 'He was sad yesterday', 'She is excited now'.

Backing to the reasoning tasks, it is reasonable to solve a arithmetic, or logical reasoning problem in the daily interactions with LLMs. And our generated samples, although a little bit straightforward, are similar to those real-world benchmarks. Here is an example of arithmetic problem:

```
Here is a description of an arithmetic problem:

The value of aaa is 9.
The value of aad is 4.
aae gets its value by taking the square root of the value that aad has.
The value of aab is 3.
aac gets its value by adding together the value of aaa and aab.
aaf gets its value by subtracting the value of aae from the value of aac.

Compute the result of aaf.
```

[1] Brody, Nathan. "What is intelligence?." International Review of Psychiatry 11.1 (1999): 19-25.

[2] Lohman, D., & Lakin, J. (2011). Intelligence and Reasoning. In R. Sternberg & S. Kaufman (Eds.), The Cambridge Handbook of Intelligence (Cambridge Handbooks in Psychology, pp. 419-441).

[2] Sawada, Tomohiro, et al. "Arb: Advanced reasoning benchmark for large language models." arXiv preprint arXiv:2307.13692 (2023).

Reviewer comment (Novelty):
Before reading this paper, I believed that generating a large number of mathematical problems of specific types and evaluating LLMs on them was primarily for debugging specific LLM capabilities, such as compositionality, rather than as an evaluation framework. I'm not sure if these types of problems are fundamental questions about LLMs. In fact, prior studies, such as those by Dziri et al., have already highlighted the limitations of transformers in these settings, using a very similar setup for demonstration.

Author response:
- First of all, mathematical problems are indeed a fundamental evaluation problem in LLMs, as shown in GPT-4 report and many other benchmarks. We refer the reviewer to general response for more explanation of reasoning tasks.
- Second, while the "faith and fate" paper claimed Transformer architecture faces limits in compositionality, current LLM benchmarks are still relying on the evaluation of math problems to show their strength. So, such evaluation is still valid. 
- Finally and most importantly, the goal of DyVal is totally different with the computational graphs analyzed in Dziri et al.' work. DyVal aims to mitigate the data contamination and static nature of current benchmarks. While Dziri et al.' work indicated that LLMs learned spurious correlation and tried to match subgraphs during reasoning process.

Reviewer comment (Experiments):
It's not clear if LLMs are losing some skills when fine-tuned on DyVal as DyVal examples and the chosen existing benchmarks are from very similar domains. The generalization of the fine-tuned model on DP is interesting though.

Author response:
Thanks for your valuable suggestions. First of all, there is no unanimous option in the community about the ability before and after fine-tuning: they could be stronger on some tasks and weaker on other tasks. This is still an open question. Therefore, we do not have a concrete answer in all models and tasks to this question. 

To resolve your concerns, we fine-tuned GPT-3.5-turbo-0613 using our generated data on abductive logic and reachability datasets, as GPT-3.5 performs worst on these two datasets. Specifically, we generated 100 samples across complexity levels D1, D2, and D3 for each task. We compared the performance of original and fine-tuned models on several benchmark tasks in GLUE dataset. The performance of abductive logic and reachability task are tested on D4 task (different from fine-tuning dataset). As shown below, performance on WNLI and QNLI datasets dropped for the fine-tuned model. However, fine-tuned model achieves better results on CoLA, QQP, and MRPC datasets. Despite the mixed results, the overall improvement in several datasets suggests that fine-tuning on our generated datasets *does not* necessarily hurt the general language understanding ability.

|  | Abductive Logic | Reachability | SST-2 | CoLA | WNLI | QNLI | QQP | MRPC |
|------------------------------|----------------------------------------------|-------------------------------------------|------------------------------------|-----------------------------------|-----------------------------------|-----------------------------------|----------------------------------|-----------------------------------|
| GPT-3.5                       | 55.27                                        | 50.00                                     | **93.29**                              | 77.00                             | **59.15**                             | **80.00**                             | 76.50                            | 73.0                              |
| GPT-3.5-FT                    | **85.10**                                        | **96.53**                                     | 93.23                              | **78.00**                             | 45.07                             | 72.50                             | **78.00**                            | **77.5**                              |

Reviewer comment (Writing):
The title is somewhat misleading. The evaluation tasks in this paper are mostly about reasoning on maths, logic, algorithms, etc. However, the title reflects no information about this point. The abstract could be also clearer if this point can be mentioned earlier.

Author response:
Thank you for your suggestions. We have revised the title to better reflect the focus on reasoning tasks. However, we would like to highlight that our proposed dynamic evaluation protocol can go beyond reasoning tasks in this paper to other tasks such as natural language understanding (NLU) tasks. Take sentiment analysis as an example, We can abstract a sentence into a syntax tree and form tree templates which can be used to generate diverse sentences. Take 'I am happy today' as an instance, it can be formalized as:
    
```
     Subject (I)
         |
     Predicate
    /        \
Verb (am)   Complement
            /        \
  Adjective (happy)  Temporal Phrase (today)
```
    

Following this template, we can create variations by replacing the corresponding words, like 'He was sad yesterday', 'She is excited now'. 

For more information, please refer to the general response.

Reviewer comment (Evaluation):
For the fine-tuning results in Section 5, I wonder when these LLMs are fine-tuned for the reasoning tasks proposed in this method, will the general abilities be influenced? Or to what extent will they be influenced?

Author response:
Thanks for your valuable suggestions. First of all, there is no unanimous option in the community about the ability before and after fine-tuning: they could be stronger on some tasks and weaker on other tasks. This is still an open question. Therefore, we do not have a concrete answer in all models and tasks to this question. 

To resolve your concerns, we fine-tuned GPT-3.5-turbo-0613 using our generated data on abductive logic and reachability datasets, as GPT-3.5 performs worst on these two datasets. Specifically, we generated 100 samples across complexity levels D1, D2, and D3 for each task. We compared the performance of original and fine-tuned models on several benchmark tasks in GLUE dataset. The performance of abductive logic and reachability task are tested on D4 task (different from fine-tuning dataset). As shown below, performance on WNLI and QNLI datasets dropped for the fine-tuned model. However, fine-tuned model achieves better results on CoLA, QQP, and MRPC datasets. Despite the mixed results, the overall improvement in several datasets suggests that fine-tuning on our generated datasets *does not* necessarily hurt the general language understanding ability.

|  | Abductive Logic | Reachability | SST-2 | CoLA | WNLI | QNLI | QQP | MRPC |
|------------------------------|----------------------------------------------|-------------------------------------------|------------------------------------|-----------------------------------|-----------------------------------|-----------------------------------|----------------------------------|-----------------------------------|
| GPT-3.5 | 55.27  | 50.00 | **93.29**  | 77.00 | **59.15**   | **80.00**   | 76.50 | 73.0  |
| GPT-3.5-FT                    | **85.10**                                        | **96.53**                                     | 93.23                              | **78.00**                             | 45.07 | 72.50 | **78.00**                            | **77.5**                              |

Reviewer comment (Evaluation):
As the samples for evaluation are dynamic, the comparison may be unfair when the generated data are different in different evaluation stages.

Author response:
Thank you for this valuable insight. Indeed, unfair comparison could happen without care. In our work, we have taken careful measures to ensure that our evaluation is fair and consistent across different LLMs.
- First, the generation mechanism indicates that there is no same evaluation sets for each algorithm. That's why we need to introduce randomness and multiple runs in the experiments. We never said that the results in a single run can be used to reflect the performance, but on multiple runs.
- Specifically, our experiments are done in multiple runs to eliminate such bias and unfairness. For each level of complexity (denoted as D1 to D4 in our paper, with D1 being the simplest), we generate a set of 500 samples. We run each evaluation *three* times. We present the mean and standard deviation of the performance metrics in Appendix Tables 4, 5, and 6. The standard deviations are *minor*, typically ranging between 0.5 to 1.5.
- Finally, we acknowledge there is always room for fairness enhancement such as increasing the number of evaluation samples and the frequency of repetitions can further refine the accuracy and fairness of our assessments.

Reviewer comment (Evaluation):
Can you discuss the influence of fine-tuning on reasoning tasks on the general language understanding ability?

Author response:
Thanks for your valuable suggestions. First of all, there is no unanimous option in the community about the ability before and after fine-tuning: they could be stronger on some tasks and weaker on other tasks. This is still an open question. Therefore, we do not have a concrete answer in all models and tasks to this question. 

To resolve your concerns, we fine-tuned GPT-3.5-turbo-0613 using our generated data on abductive logic and reachability datasets, as GPT-3.5 performs worst on these two datasets. Specifically, we generated 100 samples across complexity levels D1, D2, and D3 for each task. We compared the performance of original and fine-tuned models on several benchmark tasks in GLUE dataset. The performance of abductive logic and reachability task are tested on D4 task (different from fine-tuning dataset). As shown below, performance on WNLI and QNLI datasets dropped for the fine-tuned model. However, fine-tuned model achieves better results on CoLA, QQP, and MRPC datasets. Despite the mixed results, the overall improvement in several datasets suggests that fine-tuning on our generated datasets *does not* necessarily hurt the general language understanding ability.

|  | Abductive Logic | Reachability | SST-2 | CoLA | WNLI | QNLI | QQP | MRPC |
|------------------------------|----------------------------------------------|-------------------------------------------|------------------------------------|-----------------------------------|-----------------------------------|-----------------------------------|----------------------------------|-----------------------------------|
| GPT-3.5 | 55.27  | 50.00 | **93.29**  | 77.00 | **59.15**   | **80.00**   | 76.50 | 73.0  |
| GPT-3.5-FT                    | **85.10**                                        | **96.53**                                     | 93.23                              | **78.00**                             | 45.07 | 72.50 | **78.00**                            | **77.5**                              |

Reviewer comment (Evaluation):
Can you provide how to fairly evaluate the different models, especially if this evaluation method is released as a public leaderboard?

Author response:
Thank you for this valuable insight. Indeed, unfair comparison could happen without care. In our work, we have taken careful measures to ensure that our evaluation is fair and consistent across different LLMs.
- First, the generation mechanism indicates that there is no same evaluation sets for each algorithm. That's why we need to introduce randomness and multiple runs in the experiments. We never said that the results in a single run can be used to reflect the performance, but on multiple runs.
- Specifically, our experiments are done in multiple runs to eliminate such bias and unfairness. For each level of complexity (denoted as D1 to D4 in our paper, with D1 being the simplest), we generate a set of 500 samples. We run each evaluation *three* times. We present the mean and standard deviation of the performance metrics in Appendix Tables 4, 5, and 6. The standard deviations are *minor*, typically ranging between 0.5 to 1.5.
- Finally, we acknowledge there is always room for fairness enhancement such as increasing the number of evaluation samples and the frequency of repetitions can further refine the accuracy and fairness of our assessments.

Reviewer comment (Evaluation):
By the nature of the benchmark, it focuses on problems that can be expressed as (currently small) compute graphs or data graphs and are somewhat artificial. It only tests a very limited field of LLM functionality.

Author response:
Thank you for your valuable review. The importance of reasoning tasks in LLM evaluation can be found in "general response" section. Here we emphasize the flexibility of our DyVal framework by constructing a natural language understanding task: Take sentiment analysis as an example, We can abstract a sentence into a syntax tree and form tree templates which can be used to generate diverse sentences. Take 'I am happy today' as an instance, it can be formalized as:
    
```
     Subject (I)
         |
     Predicate
    /        \
Verb (am)   Complement
            /        \
  Adjective (happy)  Temporal Phrase (today)
```

Following this template, we can create variations by replacing the corresponding words, like 'He was sad yesterday', 'She is excited now'.

Reviewer comment (Evaluation):
There are benchmarks for all of the tasks that are implemented in this framework already. The paper states that its performance results contradict the ones on some of these benchmarks, but does not say which ones and, perhaps more importantly, does not provide any insight into why this is the case.

Author response:
Thank you for your valuable feedback. We acknowledge the presence of existing benchmarks and the need to clarify how our work differs and contributes beyond them. In response to your concerns, we offer the following insights:

1. **Data contamination issue:** As we discussed in related work, current benchmarks are mostly public collected datasets from Internet, DynaX series using crowd-source to generate dynamic dataset. We observed a remarkable discrepancy between the performance of certain Large Language Models (LLMs), such as phi-1.5 [1], wizard-math [2] and Xwin-13B [3] on widely-used benchmarks such as GSM8K [4], and their performance on the math tasks generated by our framework. These models show nearly zero efficiency on our tasks, despite claiming comparable (even state-of-the-art) performance on GSM8K. This discrepancy suggests a severe data contamination issue [5,6,7,8,9,10,11,12]. LLMs might be indirectly trained on parts of these test sets, inflating their genuine performance. As mentioned in Sec. 3.2.1, the dynamic generation process of DyVal ensures no overlap with training datasets, guarantee robust and accurate evaluation.

    [1] Li, Yuanzhi, et al. "Textbooks are all you need ii: phi-1.5 technical report." arXiv preprint arXiv:2309.05463 (2023).

    [2] Luo, Haipeng, et al. "Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct." arXiv preprint arXiv:2308.09583 (2023).

    [3] Xwin-LM. https://github.com/Xwin-LM/Xwin-LM

    [4] Cobbe, Karl, et al. "Training verifiers to solve math word problems." arXiv preprint arXiv:2110.14168 (2021).

    [5] GPT-4 performs significantly worse on coding problems not in its training data. https://brianlovin.com/hn/35297067

    [6] No, GPT4 can’t ace MIT. https://flower-nutria-41d.notion.site/No-GPT4-can-t-ace-MIT-b27e6796ab5a48368127a98216c76864

    [7] Kocoń, Jan, et al. "ChatGPT: Jack of all trades, master of none." Information Fusion (2023): 101861.

    [8] Zečević, Matej, et al. "Causal parrots: Large language models may talk causality but are not causal." arXiv preprint arXiv:2308.13067 (2023).

    [9] Wei, Tianwen, et al. "Skywork: A More Open Bilingual Foundation Model." arXiv preprint arXiv:2310.19341 (2023).

    [10] Zhou, Kun, et al. "Don't Make Your LLM an Evaluation Benchmark Cheater." arXiv preprint arXiv:2311.01964 (2023).

    [11] Golchin, Shahriar, and Mihai Surdeanu. "Time travel in llms: Tracing data contamination in large language models." arXiv preprint arXiv:2308.08493 (2023).

    [12] Yang, Shuo, et al. "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples." arXiv preprint arXiv:2311.04850 (2023).

2. **Static nature of current benchmarks:** For advanced LLMs such as ChatGPT and GPT-4 , they tend to achieve near-perfect scores on static benchmarks, rendering these evaluations less effective for measuring progress. For example, as shown in the following table, both ChatGPT and GPT-4 achieved 100% accuracy in math and logic reasoning tasks in BigBench. In our framework, DyVal can evolve the evaluation benchmarks by increasing the complexity of tasks, preventing them from plateauing in performance. As shown in Sec. 4.2, the performance of ChatGPT significantly drops as tasks becoming more challenging. This observation validates the effectiveness of DyVal in providing a evolving testbed for LLMs.



> To what extent the propose benchmark can do new things?

This is the main contribution of our benchmark: dynamic generation of testing samples to overcome the data contamination issue, which is completely new. This also relates to the importance of this research, and the reviewer is suggested to refer to the general response section.

Reviewer comment (Experiments):
The data generation strategies used by existing benchmarks are not discussed.

Author response:
Thank you for your valuable feedback. We acknowledge the presence of existing benchmarks and the need to clarify how our work differs and contributes beyond them. In response to your concerns, we offer the following insights:

1. **Data contamination issue:** As we discussed in related work, current benchmarks are mostly public collected datasets from Internet, DynaX series using crowd-source to generate dynamic dataset. We observed a remarkable discrepancy between the performance of certain Large Language Models (LLMs), such as phi-1.5 [1], wizard-math [2] and Xwin-13B [3] on widely-used benchmarks such as GSM8K [4], and their performance on the math tasks generated by our framework. These models show nearly zero efficiency on our tasks, despite claiming comparable (even state-of-the-art) performance on GSM8K. This discrepancy suggests a severe data contamination issue [5,6,7,8,9,10,11,12]. LLMs might be indirectly trained on parts of these test sets, inflating their genuine performance. As mentioned in Sec. 3.2.1, the dynamic generation process of DyVal ensures no overlap with training datasets, guarantee robust and accurate evaluation.

    [1] Li, Yuanzhi, et al. "Textbooks are all you need ii: phi-1.5 technical report." arXiv preprint arXiv:2309.05463 (2023).

    [2] Luo, Haipeng, et al. "Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct." arXiv preprint arXiv:2308.09583 (2023).

    [3] Xwin-LM. https://github.com/Xwin-LM/Xwin-LM

    [4] Cobbe, Karl, et al. "Training verifiers to solve math word problems." arXiv preprint arXiv:2110.14168 (2021).

    [5] GPT-4 performs significantly worse on coding problems not in its training data. https://brianlovin.com/hn/35297067

    [6] No, GPT4 can’t ace MIT. https://flower-nutria-41d.notion.site/No-GPT4-can-t-ace-MIT-b27e6796ab5a48368127a98216c76864

    [7] Kocoń, Jan, et al. "ChatGPT: Jack of all trades, master of none." Information Fusion (2023): 101861.

    [8] Zečević, Matej, et al. "Causal parrots: Large language models may talk causality but are not causal." arXiv preprint arXiv:2308.13067 (2023).

    [9] Wei, Tianwen, et al. "Skywork: A More Open Bilingual Foundation Model." arXiv preprint arXiv:2310.19341 (2023).

    [10] Zhou, Kun, et al. "Don't Make Your LLM an Evaluation Benchmark Cheater." arXiv preprint arXiv:2311.01964 (2023).

    [11] Golchin, Shahriar, and Mihai Surdeanu. "Time travel in llms: Tracing data contamination in large language models." arXiv preprint arXiv:2308.08493 (2023).

    [12] Yang, Shuo, et al. "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples." arXiv preprint arXiv:2311.04850 (2023).

2. **Static nature of current benchmarks:** For advanced LLMs such as ChatGPT and GPT-4 , they tend to achieve near-perfect scores on static benchmarks, rendering these evaluations less effective for measuring progress. For example, as shown in the following table, both ChatGPT and GPT-4 achieved 100% accuracy in math and logic reasoning tasks in BigBench. In our framework, DyVal can evolve the evaluation benchmarks by increasing the complexity of tasks, preventing them from plateauing in performance. As shown in Sec. 4.2, the performance of ChatGPT significantly drops as tasks becoming more challenging. This observation validates the effectiveness of DyVal in providing a evolving testbed for LLMs.



> To what extent the propose benchmark can do new things?

This is the main contribution of our benchmark: dynamic generation of testing samples to overcome the data contamination issue, which is completely new. This also relates to the importance of this research, and the reviewer is suggested to refer to the general response section.

Reviewer comment (Novelty):
The extent to which the benchmark can be used to really do new things (beyond existing benchmarks) is not discussed.

Author response:
Thank you for your valuable feedback. We acknowledge the presence of existing benchmarks and the need to clarify how our work differs and contributes beyond them. In response to your concerns, we offer the following insights:

1. **Data contamination issue:** As we discussed in related work, current benchmarks are mostly public collected datasets from Internet, DynaX series using crowd-source to generate dynamic dataset. We observed a remarkable discrepancy between the performance of certain Large Language Models (LLMs), such as phi-1.5 [1], wizard-math [2] and Xwin-13B [3] on widely-used benchmarks such as GSM8K [4], and their performance on the math tasks generated by our framework. These models show nearly zero efficiency on our tasks, despite claiming comparable (even state-of-the-art) performance on GSM8K. This discrepancy suggests a severe data contamination issue [5,6,7,8,9,10,11,12]. LLMs might be indirectly trained on parts of these test sets, inflating their genuine performance. As mentioned in Sec. 3.2.1, the dynamic generation process of DyVal ensures no overlap with training datasets, guarantee robust and accurate evaluation.

    [1] Li, Yuanzhi, et al. "Textbooks are all you need ii: phi-1.5 technical report." arXiv preprint arXiv:2309.05463 (2023).

    [2] Luo, Haipeng, et al. "Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct." arXiv preprint arXiv:2308.09583 (2023).

    [3] Xwin-LM. https://github.com/Xwin-LM/Xwin-LM

    [4] Cobbe, Karl, et al. "Training verifiers to solve math word problems." arXiv preprint arXiv:2110.14168 (2021).

    [5] GPT-4 performs significantly worse on coding problems not in its training data. https://brianlovin.com/hn/35297067

    [6] No, GPT4 can’t ace MIT. https://flower-nutria-41d.notion.site/No-GPT4-can-t-ace-MIT-b27e6796ab5a48368127a98216c76864

    [7] Kocoń, Jan, et al. "ChatGPT: Jack of all trades, master of none." Information Fusion (2023): 101861.

    [8] Zečević, Matej, et al. "Causal parrots: Large language models may talk causality but are not causal." arXiv preprint arXiv:2308.13067 (2023).

    [9] Wei, Tianwen, et al. "Skywork: A More Open Bilingual Foundation Model." arXiv preprint arXiv:2310.19341 (2023).

    [10] Zhou, Kun, et al. "Don't Make Your LLM an Evaluation Benchmark Cheater." arXiv preprint arXiv:2311.01964 (2023).

    [11] Golchin, Shahriar, and Mihai Surdeanu. "Time travel in llms: Tracing data contamination in large language models." arXiv preprint arXiv:2308.08493 (2023).

    [12] Yang, Shuo, et al. "Rethinking Benchmark and Contamination for Language Models with Rephrased Samples." arXiv preprint arXiv:2311.04850 (2023).

2. **Static nature of current benchmarks:** For advanced LLMs such as ChatGPT and GPT-4 , they tend to achieve near-perfect scores on static benchmarks, rendering these evaluations less effective for measuring progress. For example, as shown in the following table, both ChatGPT and GPT-4 achieved 100% accuracy in math and logic reasoning tasks in BigBench. In our framework, DyVal can evolve the evaluation benchmarks by increasing the complexity of tasks, preventing them from plateauing in performance. As shown in Sec. 4.2, the performance of ChatGPT significantly drops as tasks becoming more challenging. This observation validates the effectiveness of DyVal in providing a evolving testbed for LLMs.



> To what extent the propose benchmark can do new things?

This is the main contribution of our benchmark: dynamic generation of testing samples to overcome the data contamination issue, which is completely new. This also relates to the importance of this research, and the reviewer is suggested to refer to the general response section.

Reviewer comment (Theory):
The paper focuses solely on DAGs, but it's unclear why this is done for data graphs (e.g., reachability, max-sum).

Author response:
- > Why focusing on DAGs?
	- DAG is a fundamental structure in computer science used to model a wide range of problems where entities (like cities) are represented as nodes and connections as links. The decision to focus on DAGs was based on their prevalence and applicability in real-world scenarios. For instance, they are crucial in scenarios like project scheduling, dependency resolution, and more, thus providing a practical context for evaluating LLMs.
- > Unclear why this is done for data graphs
	- Reachability and Max-sum are fundamental tasks in graph algorithms, each with significant applicability in various real-world contexts, such as ransportation and network analysis.

Reviewer comment (Evaluation):
It is unclear whether graph size is the right complexity measure. For example, for reachability, the task appears easier if source and target are neighbors, no matter how large the graph.

Author response:
- > Why graph size is the right complexity measure?
	-  Graph size is generally used in the domain of graph learning to represent certain degree of complexities. Larger graphs require LLMs to efficiently manage and interpret more extensive information, thus challenging their retrieval and search capabilities. The scenario that the source and target are neighbors becomes increasingly less probable as the graph size grows. Therefore, larger graphs still offer a more robust challenge due to the reduced likelihood of neighboring nodes.

Reviewer comment (Experiments):
The system does not seem to generate balanced datasets. For example, the paper reports in the appendix that the proportion of true answers for reachability is not controlled, leading to "paradoxical" results.

Author response:
- > The system does not seem to generate balanced datasets?
	- Our algorithm can easily satisfy the balance requirement. We acknowledge the importance of balanced datasets in avoiding skewed results. This can be easily achieved by meticulously controlling the flexible dynamic generation process. For example, we can drop the generated evaluation samples with 'False' labels until we generate a sample with 'True' label. In fact, we presented the **results of ChatGPT and GPT4 in balanced datasets** in Table 1. The results in balanced datasets are similar to our initial findings: (1) ChatGPT consistently predicted all questions as "True.", resulted in a uniform accuracy rate of 50%. (2) GPT-4 demonstrated excellent performance. It maintained significantly higher accuracy rates across all complexity levels.

        | **Model**      | **ChatGPT** |   |   |   | **GPT4** |   |   |   |
        |----------------|-------------|-------|-------|-------|----------|-------|-------|-------|
        | **Complexity** | D1          | D2    | D3    | D4    | D1       | D2    | D3    | D4    |
        | **Balanced**   | 50          | 50.00 | 50.00 | 50.00 | 84.54    | 79.03 | 73.50 | 72.41 |
        | **Imbalanced** | 63.87       | 54.27 | 53.73 | 52.33 | 85.33    | 83.00 | 67.67 | 74.33 |

Reviewer comment (Reproducibility):
It is important for benchmarking papers such as this one to make all code, datasets, prompts, results, etc. public. The paper currently does not provide any resources (or, at least, the reviewer did not see them).

Author response:
We have submitted our code in the revision.

Reviewer comment (Evaluation):
The insight that can be drawn from the experiments is somewhat limited. While the reviewer does not count this against the paper, it only shows exposed limitations of LLMs and prompting strategies, and that the generated tasks are useful for fine-tuning.

Author response:
With all due respect, we *cannot* agree with this comment saying that we provide "limited insight" in this study. We would like the reviewer to refer to the general response to understand the importance of this work in fighting against the data contamination issue: we are the first dynamic algorithm for LLM evaluation with some interesting findings that were *never* demonstrated before. 
- First of all, our primary goal is not merely to introduce another challenging benchmark. Instead, we aim to establish a dynamic evaluation protocol trying to address two critical issues in evaluating LLMs: data contamination and the static nature of current benchmarks. By dynamically generating and evolving our evaluation tasks, we intend to present a more accurate and robust method for assessing LLMs' true capabilities.
- Second, our experimental results demonstrate some **new** findings that were never revealed before: 
    - Data contamination issue: There exists a significant discrepancy in the performance of some LLMs on static benchmarks versus our dynamic DyVal benchmarks. This contrast provides strong evidence of the data contamination issue in existing benchmarks, where models may inadvertently be trained on test data, thus skewing their performance.
    - Efficacy of DyVal as an evolving benchmark: Our results also demonstrate the potential of DyVal as an evolving benchmark. It successfully challenges LLMs with tasks that increase in complexity, ensuring that the models can be continuously tested against more advanced and varied problems.
    - Prompt engineering can help in certain tasks, but still fails in our benchmarks.

We do hope the reviewer can have a better understanding of our contributions and findings.