Repository: modelscope/evalscope
Issue: [RFC]  Supporting Standard Dataset Formats for Accuracy Validation
Root cause: Unimplemented Feature Gap

## Feature Description

Hey everyone in the community,  

I think EvalScope is a very practical tool—I use it for stress testing and accuracy validation. However, I feel the current version still has some limitations:  

1. Most of the current evaluation datasets focus on open-ended Q&A, which tests the model’s general intelligence. But for quantized models, it’s hard to precisely compare accuracy issues.  
2. Models deployed across different frameworks (vLLM, sglang) and hardware (GPU, NPU) may have accuracy discrepancies. I’d like to evaluate these differences in more detail. I have my own datasets, but the supported formats for custom datasets ([link](https://evalscope.readthedocs.io/en/latest/index.html)) are somewhat limited—for example, I might want to use multiple images or define custom request formats, which isn’t fully supported yet.  
3. HuggingFace datasets come in all kinds of formats, but ultimately they need to be converted into OpenAI-compatible formats to call the API. This requires a lot of conversion work.  Therefore, we don't actually need to adapt to various datasets. We only need to use a general - purpose dataset to request the OpenAI API.

So, I think we need a standardized dataset format for accuracy validation. Here’s an idea I’d like to share:  

For example, can we follow OpenAI’s standard? Each request is essentially an array of messages. What if we structure it like this: a CSV file where each row represents a real request and only contains the message content?

``` csv
messages answers
[{"role": "user", "content": [{"type": "text", "text": "What's in this image?"}, {"type": "image_url", "image_url": {"url": "https://example.com/street.jpg"}}]}]  "answers"
[{"role": "user", "content": [{"type": "text", "text": "First image:"}, {"type": "image_url", "image_url": {"url": "https://example.com/img1.jpg"}}, {"type": "text", "text": "Second image:"}, {"type": "image_url", "image_url": {"url": "https://example.com/img2.jpg"}}, {"type": "text", "text": "Compare both images."}]}] "answers"
[{"role": "system", "content": "You are a medical AI assistant."}, {"role": "user", "content": [{"type": "text", "text": "Analyze this medical image:"}, {"type": "image_url", "image_url": {"url": "https://example.com/xray.jpg", "detail": "high"}}]}] "answers"

```

More specifically, the URL `"url": "https://example.com/street.jpg"`  of the image here can be either an address or a Base64 - encoded string. This way, we can adapt to various multimodal content.


This way, we won't need to accommodate various formats or worry about whether it's multiple-choice or open-ended questions.  
We only need to modify the parameter list in EvalScope to support this standard format. This will allow us to intuitively evaluate accuracy issues across different devices and frameworks.  

If you think this idea is promising, we can continue discussing more details.



Thanks for the suggestion! 

Currently, EvalScope already supports custom text-based Q&A datasets using the message format. You can refer to the documentation here: https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset/llm.html#question-answering-format-qa

We are actively working on extending support for multimodal data. Please stay tuned for updates!

Hi! Thanks for your reply!
I've read the relevant documentation. Currently, this custom evaluation is used for LLM models, and the support for VLM is still insufficient.
1. For example, currently, the function of attaching 12 images to a request is not supported.
2. Essentially, the LLM custom evaluation also defines a data format. Why don't we directly use the original OpenAI message format? We can even use such a dataset to request VLM without any issues.


@pandalee99 Hi! Thanks for your feedback!

Custom multimodal evaluation is now supported! Please use the `main` branch and refer to this documentation:
https://evalscope.readthedocs.io/en/latest/advanced_guides/custom_dataset/vlm.html#multimodal-large-models

Feel free to let us know if you have any issues!

@Yunnglin Hey! Thanks a lot for your support! 
I think this Feature meets my expectations perfectly. Thank you for your help!