Repository: EvolvingLMMs-Lab/lmms-eval
Issue: [Question about Dataset] Question About Randomness and Prompt Consistency in VDC Benchmark
Root cause: Documentation Deficiency

@Espere-1119-Song 

Hello,

I have a question regarding the [VDC benchmark](https://github.com/EvolvingLMMs-Lab/lmms-eval/tree/main/lmms_eval/tasks/vdc), specifically about the use of randomness in prompt selection. In the current implementation, prompts are selected using random.choice. While I understand that the default random seed in lmms_eval/evaluator.py is set to 0, ensuring some level of reproducibility, slight variations might still occur due to differences in Python versions or operating systems.

Additionally, I noticed some inconsistencies in the phrasing of prompts for the same task. For example, some prompts include phrases like “in detail,” others specify “a few sentences,” while some are simple questions without such qualifiers.

I’m curious whether these factors—randomness in prompt selection and variations in prompt phrasing—have been thoroughly accounted for in the scoring methodology. These aspects seem potentially impactful, so I would appreciate clarification on how they are addressed.

Also, I noticed that the evaluation model was changed from GPT-4o to a local model in this [pull request](https://github.com/EvolvingLMMs-Lab/lmms-eval/pull/411), which differs from [your original paper](https://arxiv.org/pdf/2410.03051). Could you explain the rationale behind this change?
It seems that the API cost was an important factor in this decision. Could you provide an estimate of the total cost for the overall evaluation?


Thank you for your thoughtful questions and observations regarding the VDC benchmark! Your curiosity highlights important aspects of our work, and I’m happy to provide some insights:

**Randomness and Variations in Prompt Selection:**
We found it fascinating that some models are highly sensitive to the phrasing of instructions. This inspired us to ensure that models are not only capable of understanding instructions but also robust to variations in their expression. To achieve this, we introduced random prompt selection and intentionally varied prompt phrasing to test how well models perform across diverse linguistic contexts. These prompts were carefully crafted using GPT-4, with each evaluation aspect designed to maintain a consistent underlying goal. While prompts may differ in length or wording, our scoring methodology was built to be invariant to these differences, ensuring evaluations are both fair and consistent. This approach adds an exciting layer to the benchmark by testing models' adaptability and linguistic robustness!

**Change of Evaluation Model:**
Regarding the transition from GPT-4 to a local model, this decision was driven by practicality and the desire to ensure a smoother user experience. While GPT-4 is undeniably powerful, we noticed some intriguing inconsistencies during our leaderboard expansion and broader user evaluations. These were linked not only to the well-known version changes in GPT-4 but also to subtler factors such as API invocation methods (e.g., proxy usage) and even call timing, which occasionally led to unpredictable variations.

Furthermore, accessibility is a key priority for us. Not all users can easily access GPT-4’s API, and we wanted the benchmark to be both open and cost-free. After extensive testing, we selected LLaMA3.1-8B, an open-source model that performs on par with GPT-4 for this task while offering greater stability. This change ensures that the benchmark remains robust, accessible, and enjoyable to use for the community.

If you have further suggestions or observations, please don’t hesitate to share them! Your feedback plays an invaluable role in refining and enhancing this benchmark.

Thank you for your fast and kind response. 👍 
All my questions have been answered.