zhiminy/EvalEng huggingface dataset:
Repository: open-compass/VLMEvalKit
Issue: Detail of the scores for MathVista
Root cause: Documentation Deficiency

Thank you for your useful library.

When I compute the scores for MathVista, I eventually obtain
` 0  Overall                      1000  687  584  68.7     58.4`
 Which one is the final score?
What does prefetch_hit mean?

Thank you for your help

I meet the same problem.

Hi, @HugoLaurencon @linhaojia13 ,
In our output file, `prefetch` means the prediction can be matched with the ground-truth answer (open-ended questions) or one of the choices (multi-choice questions, MCQ) with rule-based matching, it doesn't necessarily mean the prediction is correct for MCQ questions; while `hit` means the prediction is correct (the judgement is rely on GPT when `prefetch` fails). 
Basically, in the metric csv file, only `acc` matters and indicates the benchmark accuracy. We are sorry for the confusion and will try to improve the output format.

Lol, I also had to google this xD Hi Hugo :)
Repository: open-compass/VLMEvalKit
Issue: 如何更改数据下载的缓存目录
Root cause: Documentation Deficiency


在 .env 设置 `LMUData=` 文件夹路径

> 在 .env 设置 `LMUData=` 文件夹路径

.env在哪儿呀，没找到

> > 在 .env 设置 `LMUData=` 文件夹路径
> 
> .env在哪儿呀，没找到

自己创建就行了～
Repository: open-compass/VLMEvalKit
Issue: InternVL2-76B/ VILA Series
Root cause: Others

Hey @kennymckormick @junming-yang 

Sharing offline eval results to help updated the HF leaderboard
[InternVL2-76B.zip](https://github.com/user-attachments/files/16482630/InternVL2-76B.zip)
[vila_3b.zip](https://github.com/user-attachments/files/16482631/vila_3b.zip)
[vila_8b.zip](https://github.com/user-attachments/files/16482632/vila_8b.zip)
[vila_13b.zip](https://github.com/user-attachments/files/16482633/vila_13b.zip)
[vila_40b.zip](https://github.com/user-attachments/files/16482634/vila_40b.zip)



Thanks you so much. We are working on validating those results.

Hi, @amitbcp ，
All provided results has been validated and merged into the OpenVLM leaderboard.
Repository: open-compass/VLMEvalKit
Issue: Inconsistent max_new_tokens
Root cause: Architectural Constraint

Across different LMMs the max new token is different .
I believe we should have a consistent MAX_NEW_TOKENS across the project, set to 512 or 1024

If it makes sense, I can create a PR to modify all of them

Hi, @amitbcp , is there any specific cases you are talking about? I think for most VLMs we adopt a MAX_NEW_TOKENS >= 512. 

@kennymckormick:

For example : 
1. in MiniCPM we have defined max length which we haven't done for models https://github.com/open-compass/VLMEvalKit/blob/22991ca6109c5d4e65bc4a1a9273234d23c3e13f/vlmeval/vlm/minicpm_v.py#L186 

2. For Phi3 its 500 https://github.com/open-compass/VLMEvalKit/blob/22991ca6109c5d4e65bc4a1a9273234d23c3e13f/vlmeval/vlm/phi3_vision.py#L36

3. For Qwen also we adjust the tokens https://github.com/open-compass/VLMEvalKit/blob/22991ca6109c5d4e65bc4a1a9273234d23c3e13f/vlmeval/vlm/qwen_vl.py#L38

4. For BunnyLlama https://github.com/open-compass/VLMEvalKit/blob/22991ca6109c5d4e65bc4a1a9273234d23c3e13f/vlmeval/vlm/bunnyllama3.py#L131


5. For CogVLM we only use 2048 : https://github.com/open-compass/VLMEvalKit/blob/22991ca6109c5d4e65bc4a1a9273234d23c3e13f/vlmeval/vlm/cogvlm.py#L24


and more.


So should we set a consistent length for all models to have them perform equally on similar bases ?
Repository: open-compass/VLMEvalKit
Issue: 如何手动下载数据，并且在代码中配置好路径
Root cause: Documentation Deficiency

如何手动下载数据，并且在代码中配置好路径

手动下载：查看对应 vlmeval.dataset 中，对应 dataset 名的 `DATASET_URL` 链接，通过wget进行下载；
配置路径：在 .env 文件中设置 `LMUData=/path/to/your/dataset`。
Repository: open-compass/VLMEvalKit
Issue: Any pypi package for this toolkit?
Root cause: Documentation Deficiency


You can see pypi Package dependencies list in https://github.com/open-compass/VLMEvalKit/blob/main/requirements.txt

> You can see pypi Package dependencies list in https://github.com/open-compass/VLMEvalKit/blob/main/requirements.txt

I mean, is this tool installable?

You can install it by this
```
git clone https://github.com/open-compass/VLMEvalKit.git
cd VLMEvalKit
pip install -e .
```
and use it by this
```
from vlmeval import *
model_name = 'llava_v1.5_13b'  # for example
model = supported_VLM[model_name]() if isinstance(model_name, str) else model_name
model.generate('hello') # only text
model.generate(['./assets/apple.jpg', 'Please describe this image.'])  # text + image
```
Furthermore, we more recommend using the VLMEvalKit repository for one-click evaluation of multimodal models and benchmarks.

> You can install it by this
> 
> ```
> git clone https://github.com/open-compass/VLMEvalKit.git
> cd VLMEvalKit
> pip install -e .
> ```
> 
> and use it by this
> 
> ```
> from vlmeval import *
> model_name = 'llava_v1.5_13b'  # for example
> model = supported_VLM[model_name]() if isinstance(model_name, str) else model_name
> model.generate('hello') # only text
> model.generate(['./assets/apple.jpg', 'Please describe this image.'])  # text + image
> ```
> 
> Furthermore, we more recommend using the VLMEvalKit repository for one-click evaluation of multimodal models and benchmarks.

Ok, I initially thought it had a Python package in PYPI like https://pypi.org/project/opencompass.
Repository: open-compass/VLMEvalKit
Issue: 超过1GB的TSV数据集无法将image字段中的图像编码保存下来
Root cause: Validation Gap

在tsv转换成local tsv文件中，image字段会被丢弃，而没有进行保存

也就是无法将tsv中的图像编码保存为本地的图片

Hello，请问有具体的问题吗？
我们的设计是：对于大于 1G 的 TSV 文件，在初始化数据集时进行图片保存，并生成 _local.tsv (只需包含保存后的图像路径即可)。这样应该是可以正常 work，并减小评测时内存占用。
Repository: open-compass/VLMEvalKit
Issue: 7B模型在4*A100 80GB上发生OOM
Root cause: Resource Mishandling

当我针对`InternLM-XComposer2_5-7B`模型进行评估时，出现了OOM，配置是4*A100 80GB。我观察到执行过程送入query进入模型的速度非常的快，发生OOM的原因是否和一次性送入的batch有关？刚开始的时候，每块GPU显存在20~70GiB上下疯狂跳动，而在第12轮左右便发生了OOM。

但是同样的4*A100 80GB的硬件条件下，评估 `InternVL2-40B`竟然没有任何问题，显存也很稳定的在45GB左右。不过执行的速度很慢，似乎在`InternVL2-40B`中每次就送入一个batch。

这很奇怪，如果是batch问题我应该如何修改它？我尝试修改过`${VLMEvalKit}/vlmeval/vlm/xcomposer/xcomposer2d5.py`的代码，但是并不奏效，我也没发现batch是在哪里实现的。

如果不是batch问题，我应该如何让7B的模型能够完成评估？

**我的执行命令是：**
```bash
torchrun --nproc-per-node=4 run.py --data MathVision MathVision_MINI --model XComposer2d5 --verbose
```
**报错信息：**
![image](https://github.com/user-attachments/assets/8f728cf6-a45c-44ff-88e4-db78eb8d7eaf)

此外，我76B的模型仍然也会有OOM的情况。我尝试先将 76B 模型利用 LMDepoly 部署成 openai 接口，再接入到 VLMEvalKit 评测框架中，但是发现失败了。报错是：`2024-08-13 16:48:22,299 - ChatAPI - ERROR - HTTPSConnectionPool(host='openaipublic.blob.core.windows.net', port=443): Max retries exceeded with url: /encodings/cl100k_base.tiktoken (Caused by NameResolutionError("<urllib3.connection.HTTPSConnection object at 0x2ae04fd4b400>: Failed to resolve 'openaipublic.blob.core.windows.net' ([Errno -2] Name or service not known)"))`




我也是类似的报错，你最后解决了吗 @Rocky77JHxu 

> 我也是类似的报错，你最后解决了吗 @Rocky77JHxu

我最终采用了ms-swift框架来确保能够先完成任务。

> > 我也是类似的报错，你最后解决了吗 @Rocky77JHxu
> 
> 我最终采用了ms-swift框架来确保能够先完成任务。

噢噢感谢！但是我用ms-swift也有类似的错误，用的命令是 `swift eval --model_type llava1_5-7b-instruct --eval_dataset POPE`，连接openai超时了。但我本地已经下载好模型和数据集了，不知道您有设置其他的东西让它不要连接openai吗，而是直接用本地的模型？

> > > 我也是类似的报错，你最后解决了吗 @Rocky77JHxu
> > 
> > 
> > 我最终采用了ms-swift框架来确保能够先完成任务。
> 
> 噢噢感谢！但是我用ms-swift也有类似的错误，用的命令是 `swift eval --model_type llava1_5-7b-instruct --eval_dataset POPE`，连接openai超时了。但我本地已经下载好模型和数据集了，不知道您有设置其他的东西让它不要连接openai吗，而是直接用本地的模型？

在.env文件中设置本地模型部署服务的`base_url`和`key`

Hi, @Rocky77JHxu @Harry-zzh , 
I have tried to evaluate XComposer2.5 on the two mentioned benchmarks and it works well. 

Evaluation Screen Shot:
![image](https://github.com/user-attachments/assets/dcc34cfd-63d9-4315-bc10-bfe7385665d9)

GPU memory usage: 
![image](https://github.com/user-attachments/assets/9fffcac5-ddd8-4639-9634-ab99ceacc476)

Meanwhile, I noticed that during evaluation, sometimes the GPU memory usage is very close to 80GB (70GB+). That is due to XComposer2.5 adopts high-resolution setting for evaluation: it features a 16k context window and accept 36 image patches at most. Thus you may encounter the OOM error if you have some another processes also using part of the GPU memory. You can try to reduce `hd_num` (Here: https://github.com/open-compass/VLMEvalKit/blob/b94a4242b5cdb06275376c91e70994c63f250510/vlmeval/vlm/xcomposer/xcomposer2d5.py#L131) if you want to save some GPU memory.
Repository: open-compass/VLMEvalKit
Issue: 关于幻觉benchmark的支持
Root cause: Unimplemented Feature Gap

请问有计划支持更多的幻觉benchmark吗，这也是一个重要的评估维度

Hi, @shufangxun , 
我们内部近期暂未有对应的开发计划，但我们欢迎来自社区的 PR 并承诺会及时 review 与 merge
Repository: open-compass/VLMEvalKit
Issue: What is LLM-based answer extraction?

I cannot find any specification about this concept?

Some datasets (like MathVista) require extracted options from prediction to calculate accuracy with ground truth, but part of the models fails to output options, or the option and the reason for the choice will be output together, which is not conducive to the extraction of the choice. For these cases, we will extract options from the prediction based on LLM (like ChatGPT) after prefetching the option. This is "LLM-based answer extraction".

Ref: https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/utils/mathvista.py#L104

> Some datasets (like MathVista) require extracted options from prediction to calculate accuracy with ground truth, but part of the models fails to output options, or the option and the reason for the choice will be output together, which is not conducive to the extraction of the choice. For these cases, we will extract options from the prediction based on LLM (like ChatGPT) after prefetching the option. This is "LLM-based answer extraction".
> 
> Ref: https://github.com/open-compass/VLMEvalKit/blob/main/vlmeval/dataset/utils/mathvista.py#L104

Thanks for explanation!
Repository: open-compass/VLMEvalKit
Issue: Dataset url access denied
Root cause: External Dependency Breakage

Hi there, it looks like the dataset server is down. Can someone take a look? Thank you!

<img width="843" alt="image" src="https://github.com/user-attachments/assets/fbb0b31a-f18c-49d6-817b-da663a138a86">


Sorry, we have located the problem and is now fixing it. 

The problem has been fixed now, please have a retry.

Thank you @kennymckormick a lot for the prompt fix!
Repository: open-compass/VLMEvalKit
Issue: TypeError in parallel API calling
Root cause: Resource Mishandling


![企业微信截图_17042675828124](https://github.com/open-compass/VLMEvalKit/assets/118104525/9b6168ed-27fd-4978-a954-7ea2a6325854)

I met this error when I called openai API  parallelly (nproc=4). I am not sure if this an error from the side of VLMEvalKit developpers...Could someone give me some tips for fixing it?


Hi, @moiSusu . This bug is fixed in #40 , feel free to reopen this issue if you still have troubles with the latest code.
Repository: open-compass/VLMEvalKit
Issue: 配置数据集的vlmeval/utils/dataset_config.py文件不存在
Root cause: Documentation Deficiency

设置在 VLMEvalKit 中支持的数据集名称（在 vlmeval/utils/dataset_config.py 中定义），这个文件没有在对应路径找到

Hi @scl-01 
由于先前对代码进行重构，这一文件目前已经不存在
你可以在Readme中查看当前支持的数据集，并在 vlmeval/dataset 下搜索该数据集具体的名字

@FangXinyu-0913 怎么搜索具体名字呢，是以数据集名字命名的py文件吗

我们现在把数据集分成了几个大类，放在几个不同的文件内（比如image_mcq代表多模态视觉领域下的多项选择题），你可以根据你想测评的数据集的类型进到对应文件内，查看里面类的DATASET_URL的key名字，即为具体名字

OK，看到了，感谢
Repository: open-compass/VLMEvalKit
Issue: There is no answer field in DocVQA_TEST
Root cause: Documentation Deficiency

There is no answer field in DocVQA_TEST. How can I get the answer field?

DocVQA is a challenge raised by researchers, and you can find the details at https://rrc.cvc.uab.es/?ch=17&com=introduction. The answer to DocVQA_TEST is not public because this challenge remains.
You can submit your results on this page to get your final results.
Repository: open-compass/VLMEvalKit
Issue: ChatAPI - ERROR - HTTPSConnectionPool(host='api.openai.com', port=443): Max retries exceeded with url: /v1/chat/completions (Caused by ProxyError('Unable to connect to proxy', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f481233f640>: Failed to establish a new connection: [Errno 111] Connection refused')))
Root cause: Environment Incompatibility

How can I solve this problem:
ChatAPI - ERROR - HTTPSConnectionPool(host='api.openai.com', port=443): Max retries exceeded with url: /v1/chat/completions (Caused by ProxyError('Unable to connect to proxy', NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f481233f640>: Failed to establish a new connection: [Errno 111] Connection refused')))

Hi, @yansuoyuli , 
Looks that is some problems related to your network environment. Make sure you can successfully run the official openai demo first.

Dear Duan,
I can run the demo successfully.



---Original---
From: "Haodong ***@***.***&gt;
Date: Sun, Aug 25, 2024 21:59 PM
To: ***@***.***&gt;;
Cc: ***@***.******@***.***&gt;;
Subject: Re: [open-compass/VLMEvalKit] ChatAPI - ERROR -HTTPSConnectionPool(host='api.openai.com', port=443): Max retries exceededwith url: /v1/chat/completions (Caused by ProxyError('Unable to connect toproxy', NewConnectionError('<urllib3.connection.HTTPSConnection object at0x7f481233f640&gt;: Failed to establish a new connection: [Errno 111] Connectionrefused'))) (Issue #406)




 
Hi, @yansuoyuli ,
 Looks that is some problems related to your network environment. Make sure you can successfully run the official openai demo first.
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***&gt;

hello, I meet the same question, did you solve it?
Repository: open-compass/VLMEvalKit
Issue: create new benchmark
Root cause: Documentation Deficiency

您好，我想请问一下，如何把数据集变成这个框架下的.tsv文件呀？

你这个问题问的不是很清楚，你可以根据数据集的类型例如MCQ，VQA，yes or no类型的问题去下载该框架下对应的tsv文件，在'''vlmeval/dataset’‘’路径下的脚本中记录着tsv的下载链接，参考相关的文件创建你自己的数据集的tsv文件以及相关处理接入代码

Hi, @yansuoyuli ,
可以参照我们的开发指南：https://github.com/open-compass/VLMEvalKit/blob/main/docs/zh-CN/advanced_guides/Development.md
Repository: open-compass/VLMEvalKit
Issue: cogvlm2-llama3-chat-19B设置device_map="auto"报错
Root cause: Configuration Error

如题，推理时单卡放不下整个模型，但是为vlmeval/vlm/cogvlm.py中的model = AutoModelForCausalLM.from_pretrained添加device_map="auto"报错
```
model = AutoModelForCausalLM.from_pretrained(
            model_path,
            torch_dtype=torch.bfloat16,
            trust_remote_code=True,
            device_map="auto").eval()
```
<img width="959" alt="image" src="https://github.com/user-attachments/assets/3e4b587b-a6fd-4800-bb7c-d6d23dd6c115">

提示不在同一个device RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:5 and cuda:6!


I fix this by changing the code with the following setting:
```
def get_memory():
    total_memory = torch.cuda.get_device_properties(0).total_memory
    total_mem = total_memory / 1024 / 1024 / 1024
    return total_mem


def build_device_map(model, default_map=None, no_split=None, alpha=0.97, beta=0.9):
    total_num_gpus = torch.cuda.device_count()
    rank, world_size = get_rank_and_world_size()
    if world_size == total_num_gpus:
        return model.cuda()

    num_gpus = total_num_gpus // world_size
    memory_map = {}
    per_gpu_mem = get_memory() * alpha
    memory_map.update({rank: f'{beta * per_gpu_mem:.2f}GiB'})
    for gpu_id in range(1, num_gpus):
        memory_map.update({rank + gpu_id * world_size: f'{per_gpu_mem:.2f}GiB'})
    if hasattr(model, '_no_split_modules'):
        no_split_module = model._no_split_modules
    else:
        no_split_module = []
    if no_split is not None:
        no_split_module = list(set((no_split_module + no_split)))
    device_map = infer_auto_device_map(
        model,
        max_memory=memory_map,
        no_split_module_classes=no_split_module
    )
    if default_map is not None:
        for i in default_map:
            device_map[i] = rank
    for value in device_map.values():
        assert value != 'disk', 'Please check and make sure to have enough memory to load model.'

    model = dispatch_model(
        model,
        device_map=device_map).eval()
    return model, device_map


from accelerate import init_empty_weights

        with init_empty_weights():
            model = AutoModelForCausalLM.from_pretrained(
                model_path,
                torch_dtype=torch.bfloat16,
                low_cpu_mem_usage=True,
                trust_remote_code=True,
            )
        default_map = ['model.vision']
        model, _ = build_device_map(model, default_map, ["EVA2CLIPModel"])
```
Repository: open-compass/VLMEvalKit
Issue: How to evaluate my own model
Root cause: Documentation Deficiency

Hi, I am wondering how to evaluate a new model that developed by myself using VLMEvalKit? The README file does mention that I only need to create a function called inner_function() (maybe.. can't remeber the exact name), but does not provide any instruction about how to proceed. Can anybody help? Thanks.

The simplest way is to first define a model base class to initialize the model, tokenizer, and chat template of your model. Then define a ```def generate_inner(self, message, dataset=None):``` function to output the answer. Note that this function is for a single question.

> The simplest way is to first define a model base class to initialize the model, tokenizer, and chat template of your model. Then define a `def generate_inner(self, message, dataset=None):` function to output the answer. Note that this function is for a single question.

Thanks a lot for the timely response. Your explanation makes sense for me, but i am still wondering if there is any document in the git folder that teaches how to implement the generate_inner() function and what the inputs and ouputs are expected to be. 


For example, if you are using a model based on the `llava-next` architecture, then you need to ensure that your input's message meets form:
```
        conversation = [
            {
                'role': 'user',
                'content': content, 
            }
        ]
```

The input is passed into the model to generate the answer roughly in the following way:
```
        prompt = self.processor.apply_chat_template(conversation, add_generation_prompt=True)
        inputs = self.processor(prompt, images, return_tensors='pt').to('cuda', torch.float16)
        output = self.model.generate(**inputs, **self.kwargs)
        answer = self.processor.decode(output[0], skip_special_token=True)
        answer = self.output_process(answer)
```
Everything depends on your model architecture. You can refer to the `vlmeval/vlm/mantis.py` I created, which also includes how to build prompts, remove identifier tokens, and add <image> based on the numbers of pictures. 

by the way, the `images` generally is a list containing the images in PIL decoded form, and also is RGB form.
Repository: open-compass/VLMEvalKit
Issue: 上传新的benchmark
Root cause: Unimplemented Feature Gap

我将新的benchmark：MM-NIAH的tsv格式数据存储到了https://huggingface.co/datasets/petter12321/MM-NIAH-VLMEvalKit/tree/main，其中MM_NIAH_VAL.tsv文件是完整的，而MM_NIAH_TEST.tsv文件被拆分成了part-aa~e五个文件，请将这五个文件组合成MM_NIAH_TEST.tsv文件而后将这两个tsv文件上传到服务器中，谢谢！

Working.
Repository: open-compass/VLMEvalKit
Issue: The Dataset Names
Root cause: Documentation Deficiency

Hi,

Does anybody know where to find the dataset names that can be passed to the run.py? In the README, I am told to find the names in defined in vlmeval/utils/dataset_config.py, however, I couldn't find this configuration file in the given directory.

Thanks.

I have found the names in a table at the upper part of the README file.

https://github.com/open-compass/VLMEvalKit/issues/402
Repository: open-compass/VLMEvalKit
Issue: openai.InternalServerError: Bad Gateway
Root cause: Environment Incompatibility

I've set up a judge with 
```
lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333 --model-name internlm2-chat-1_8b
```
And now when I try with evaluation
```
python run.py --data MMMU_DEV_VAL --model InternVL2-2B --verbose
```

It gives me `openai.InternalServerError: Bad Gateway` like the following,
```
Traceback (most recent call last):
  File "run.py", line 18, in <module>
    model_name = client.models.list().data[0].id
  File "/workdir/my_conda_env/lib/python3.8/site-packages/openai/resources/models.py", line 80, in list
    return self._get_api_list(
  File "/workdir/my_conda_env/lib/python3.8/site-packages/openai/_base_client.py", line 1310, in get_api_list
    return self._request_api_list(model, page, opts)
  File "/workdir/my_conda_env/lib/python3.8/site-packages/openai/_base_client.py", line 1161, in _request_api_list
    return self.request(page, options, stream=False)
  File "/workdir/my_conda_env/lib/python3.8/site-packages/openai/_base_client.py", line 937, in request
    return self._request(
  File "/workdir/my_conda_env/lib/python3.8/site-packages/openai/_base_client.py", line 1027, in _request
    return self._retry_request(
  File "/workdir/my_conda_env/lib/python3.8/site-packages/openai/_base_client.py", line 1076, in _retry_request
    return self._request(
  File "/workdir/my_conda_env/lib/python3.8/site-packages/openai/_base_client.py", line 1027, in _request
    return self._retry_request(
  File "/workdir/my_conda_env/lib/python3.8/site-packages/openai/_base_client.py", line 1076, in _retry_request
    return self._request(
  File "/workdir/my_conda_env/lib/python3.8/site-packages/openai/_base_client.py", line 1042, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.InternalServerError: Bad Gateway
```

I've tried with `curl` and the judge is all good to respond. What could be the problem here for the bad gateway?

By they way, if I use local LLM judge, do I need to set OPENAI_API_KEY like this?

```
lmdeploy serve api_server internlm/internlm2-chat-1_8b --server-port 23333 --model-name internlm2-chat-1_8b --api-keys sk-123456
```



It is probably due the firewall on the port. I switched to another one and explicitly specified server name with 0.0.0.0 and api-keys with suggested ones. And it now calls the local judge server on each output.
Repository: open-compass/VLMEvalKit
Issue: Cannot reproduce llava v1.5 7b SEEDBench_IMG results
Root cause: Configuration Error

When i use default setting to initiate llava v1.5 7b model evaluation on SEEDBench_IMG dataset , I got results like this :
![屏幕截图 2024-01-04 211703](https://github.com/open-compass/VLMEvalKit/assets/47048568/cea21a88-da42-4f79-bac2-6ab70f5eb38c)

I checked intermediate results , but the model seems to generate options correctly :
![屏幕截图 2024-01-04 212152](https://github.com/open-compass/VLMEvalKit/assets/47048568/195b9900-92fa-43d2-8476-0ab354257488)

The default generation config should be 
- do_sample=True
- temperature=0.2
- max_new_tokens=512
- top_p=None
- num_beams=1

And the officially reported results should be 
![屏幕截图 2024-01-04 212602](https://github.com/open-compass/VLMEvalKit/assets/47048568/fef77989-4b92-40b8-8aa8-3145bb1e7f33)

It's really weird . I don't know why there is a huge gap here. Hope to get help. Thank you in advance!
Repository: open-compass/VLMEvalKit
Issue: 修改结果路径
Root cause: Documentation Deficiency

你好！请问怎么修改结果路径，假如我的模型是llava-v1.5-7b，它会默认保存在/VLMEvalKit的目录下，llava-v1.5-7b的评测结果会保存在/VLMEvalKit/llava-v1.5-7b目录下，现在我想要这种效果/VLMEvalKit/my_test_result/llava-v1.5-7b，请问在哪里修改？

请问如何实现的？

> 请问如何实现的？

# Work Dir
    parser.add_argument('--work-dir', type=str, default='./outputs', help='select the output directory')

使用这个参数即可