Repository: open-compass/VLMEvalKit
Issue: why do we have VLMEvalKit dedicated for VLM evaluations?

Hi team, 

Thanks for the efforts for open sourcing such great work to the community!

I see you also have `OpenCompass` for evaluating performance of LLMs. May I ask why do we need to have this framework dedicated for VLM? 
in my point of view, there are no fundamental difference in evaluating large text based language model or visual language model. it would be great if you could clarify the scope of these 2 evaluation framework? Thanks~

Hi,

Thank you for your interest in VLMEvalKit!

It's important to note that Multimodal Large Language Models (MLLMs) differ significantly from standard Large Language Models (LLMs). The primary distinction lies in their architecture, where MLLMs integrate visual modules that process and understand images or other visual data. These visual components vary widely from one model to another, leading to significant differences in how they handle multimodal inputs.

For evaluation purposes, the modes used in multimodal benchmarks also differ substantially. Each benchmark may require different methods to assess how well the models process and interpret visual information alongside textual data.

In summary, the key difference lies in the handling of visual components, making VLMEvalKit essential for comprehensive and fair evaluations of MLLMs.

If you have any further questions or need more details, feel free to reach out.

Best regards

Hi, thanks for sharing your insights and sorry for the late response. I see your point that vlmevalkit is essential in the aspect of model architecture and handling of multi-modal benchmarks. As I also a former user of opencompass, may I ask some follow-up questions:

- in the aspect of model architecture, although we would have to handle more encoders for different modality, minor effort should be required and it does not seems a big issue to implement it in opencompass. in my point of view, OC itself use openICL which is a modal agnostic framework and during inference we usually treat different parts of a VLM as a unified one and use it just like how we use other huggingface/API model, right?

- in the aspect of multimodal benchmarks, I have tried most of the built-in benchmarks in vlmevalkit and I found most of the evaluation is about to compare a model's response with a given groundtruth or use other LLMs (e.g. gpt4) to evaluate the result. In my point of view, OC itself defines a dataset's load and evaluation logic in its [`Dataset`](https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/base.py#L9) and [`Evaluator`](https://github.com/open-compass/opencompass/blob/main/opencompass/openicl/icl_evaluator/icl_base_evaluator.py#L39) class, everything implemented in vlmevalkit seems still quite easy in vlmevalkit and I had implemented it for the similar usecase in my personal private branch.

To concluded, what confused me is
- are there anything that required in VLM benchmarks's evaluation is complicated in opencompass so we have to implement it under vlmevalkit as a new software? a concrete example would be helpful for understanding :)

- what is the merit of using vlmevalkit over opencompass? except for the built-in VLM benchmarks (as it is easy to migrate as all required feature is already included in the scope of opencopmass)

> Hi, thanks for sharing your insights and sorry for the late response. I see your point that vlmevalkit is essential in the aspect of model architecture and handling of multi-modal benchmarks. As I also a former user of opencompass, may I ask some follow-up questions:嗨，感谢您分享您的见解，抱歉回复迟了。我明白您的观点，即 vlmevalkit 在模型架构和多模态基准测试处理方面至关重要。由于我也是 opencompass 的前用户，我可以问一些后续问题吗：
> 
> * in the aspect of model architecture, although we would have to handle more encoders for different modality, minor effort should be required and it does not seems a big issue to implement it in opencompass. in my point of view, OC itself use openICL which is a modal agnostic framework and during inference we usually treat different parts of a VLM as a unified one and use it just like how we use other huggingface/API model, right?在模型架构方面，虽然我们必须为不同的模态处理更多的编码器，但只需要付出很少的努力，在 opencompass 中实现它似乎不是什么大问题。在我看来，OC 本身使用 openICL，它是一个与模态无关的框架，在推理过程中，我们通常将 VLM 的不同部分视为一个统一的部分，并像使用其他 huggingface/API 模型一样使用它，对吗？
> * in the aspect of multimodal benchmarks, I have tried most of the built-in benchmarks in vlmevalkit and I found most of the evaluation is about to compare a model's response with a given groundtruth or use other LLMs (e.g. gpt4) to evaluate the result. In my point of view, OC itself defines a dataset's load and evaluation logic in its [`Dataset`](https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/base.py#L9) and [`Evaluator`](https://github.com/open-compass/opencompass/blob/main/opencompass/openicl/icl_evaluator/icl_base_evaluator.py#L39) class, everything implemented in vlmevalkit seems still quite easy in vlmevalkit and I had implemented it for the similar usecase in my personal private branch.在多模态基准测试方面，我尝试了 vlmevalkit 中的大多数内置基准测试，我发现大多数评估都是将模型的响应与给定的地面实况进行比较，或者使用其他LLMs（例如 gpt4）来评估结果。在我看来，OC 本身在其[`Dataset`](https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/base.py#L9)和[`Evaluator`](https://github.com/open-compass/opencompass/blob/main/opencompass/openicl/icl_evaluator/icl_base_evaluator.py#L39)类中定义了数据集的加载和评估逻辑，在 vlmevalkit 中实现的一切似乎仍然相当容易，我已经在我的个人私有分支中为类似的用例实现了它。
> 
> To concluded, what confused me is总而言之，令我困惑的是
> 
> * are there anything that required in VLM benchmarks's evaluation is complicated in opencompass so we have to implement it under vlmevalkit as a new software? a concrete example would be helpful for understanding :)VLM 基准测试评估中需要的东西在 opencompass 中是否很复杂，所以我们必须在 vlmevalkit 下将其作为新软件实现？一个具体的例子将有助于理解 :)
> * what is the merit of using vlmevalkit over opencompass? except for the built-in VLM benchmarks (as it is easy to migrate as all required feature is already included in the scope of opencopmass)与 opencompass 相比，使用 vlmevalkit 有什么好处？除了内置的 VLM 基准测试（因为所有必需的功能都已包含在 opencopmass 的范围内，因此很容易迁移）



It's more like some legacy issues ......
Actually, the evaluation of VLMs was used to be a feature of OpenCompass (in some early versions, such as https://github.com/open-compass/opencompass/tree/0.1.8). In late 2023, the current maintainer was in charge of multi-modality evaluation and noticed some problems about the implementation in OpenCompass: 
1. Some models can not run properly, and the accuracy of some models can not be reproduced; 
2. Some complicated features in OpenCompass are not necessary for VLM evaluation; 
3. Integrating VLM evaluation in OpenCompass introduces some additional requirements which are unnecessary for LLM users. 
Then, to make things easy, we all agree to support VLM evaluation in a separate repository. 

By the way, there is some practices that enable you to adopt OpenCompass for VLM evaluation, since the evaluation interface is relatively fixed, it can be integrated into OpenCompass by implementing some model / dataset wrappers. I knew some guys who have achieved this. However, that is not the priority of our work. 


thanks for your prompt response and sharing on the development anecdotes of this software!  

based on your description, looks like the scope of these 2 software is like `opencompass` is dedicated for evaluating text-based LLMs and `vlmevalkit` is dedicated for evaluating visual LLM. I understand and respect to you team's decision to split the scope of large model evaluation into 2 independent repos to narrow down the dependency space.

I see you also mentioned there are three issues when employ opencompass to evaluate VLM benchmarks. particularly, I'm curious about the 2nd one i.e. what complicated feature in OpenCompass is necessary for text-based LLM while is not necessary for VLM? I curious about it as it introduces many duplicated work by having a new evaluation framework dedicated for VLMs. It would be great if you could share the details and I believe it would be valuable to the entire community!

Also, I would be appreciate it if you could share some lesson learned and ideation during the design and development of a LLM evaluation framework :)