zhiminy/EvalEng huggingface dataset:
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Degraded performance of Qwen2-VL on scienceqa_img
Root cause: Others

I observed severe performance degradation on scienceqa_img task. Does anyone have an idea why? Thanks in advance.

Script:
```
python3 -m accelerate.commands.launch \
    --num_processes=1 \
    -m lmms_eval \
    --model qwen2_vl \
    --model_args pretrained=Qwen/Qwen2-VL-7B-Instruct \
    --tasks scienceqa_img \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix ${run_name} \
    --output_path ./${log_dir}/${run_name}  \
    --verbosity=DEBUG
```

Reported result:
`|scienceqa_img |Yaml   |none  |     0|exact_match            |↑  |   0.1423|±  |0.0078|
`

lmms-eval version:
```
pip show lmms-eval
Name: lmms_eval
Version: 0.3.0
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: For Qwen3-VL-8B/4B
Root cause: Documentation Deficiency

When will you support Qwen3-VL-8B/4B for evaluation

It's supported already, check the examples for sglang and vllm about qwen3 vl

@kcz358 is it possible to have a chat file for qwen3_vl as well?

Hi, I think it is merged in #883
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Using VLLM to test Qwen3vl, video frame mismatch occurs
Root cause: Validation Gap

I set max_frame_nums to 16. Normally, when decoding a video, a maximum of 16 frames are selected. However, I get an error: ValueError: nframes should be in interval [2, 9], but got 16.

Shouldn't it automatically fit all frames if the number is less than 16? Why is this happening?

My script is as follows:


MODEL="Qwen/Qwen3-VL-8B-Instruct"
TENSOR_PARALLEL_SIZE=4
GPU_MEMORY_UTILIZATION=0.85
BATCH_SIZE=32
TASKS="mvbench"
OUTPUT_PATH="./results"
LOG_SUFFIX="qwen3vl_8b_vllm"

echo "Model: $MODEL"
echo "Tensor Parallel Size: $TENSOR_PARALLEL_SIZE"
echo "Tasks: $TASKS"
echo "Batch Size: $BATCH_SIZE"
echo "Output Path: $OUTPUT_PATH"
echo "=========================================="

python -m lmms_eval \ 
--model vllm \ 
--model_args model=${MODEL},tensor_parallel_size=${TENSOR_PARALLEL_SIZE},gpu_memory_utilization=${GPU_MEMORY_UTILIZATION},max_pixels=200704,max_frame_num=16 \ 
--tasks ${TASKS} \ 
--batch_size ${BATCH_SIZE} \ 
--log_samples \ 
--log_samples_suffix ${LOG_SUFFIX} \ 
--use_cache ./cache/sqlite_cache.db \ 
--output_path ${OUTPUT_PATH}

echo ""
echo "=========================================="
echo "Evaluation Complete!"
echo "Results saved to: $OUTPUT_PATH"
echo "=========================================="

You might need to set fps=1 for a dynamic video sampling. Otherwise, the qwen_vl_utils backend logic might have an error

> You might need to set fps=1 for a dynamic video sampling. Otherwise, the qwen_vl_utils backend logic might have an error您可能需要设置 fps=1 才能进行动态视频采样。否则，qwen_vl_utils 后端逻辑可能会出错。

Thanks for your reply. I noticed that only the VLLM chat model has the FPS parameter, and even after adding it, I'm still getting this error. ValueError: At most 0 image(s) may be provided in one prompt.

After adding this parameter, `limit_mm_per_prompt='{"image":16}'` worked successfully. Is this the correct modification?

same problem,  is there any solution?

> same problem, is there any solution?同样的问题，有什么解决办法吗？

 adding this parameter, limit_mm_per_prompt='{"image":16}' worked successfully.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: post prompt doenst work
Root cause: Configuration Error

I changed post_prompt in all` .yaml` in /lmms-eval/lmms_eval/tasks/vqav2. But the output `.jsonl` file still shows` \nAnswer the question using a single word or phrase.` which means it doesnt work if I change the post-prompt in the` .yaml` file. It works if I build my own dataset.

.sh file:

```
export CUDA_VISIBLE_DEVICES=7

MODEL_PATH="/workspace/models/Qwen/Qwen2___5-VL-7B-Instruct"
OUTPUT_DIR="./vqa_qwen2.5_results"
LOG_FILE="${OUTPUT_DIR}/vqa_qwen2.5_7b.log"

python -m lmms_eval \
    --model qwen2_5_vl \
    --model_args pretrained="${MODEL_PATH}" \
    --tasks vqav2_val \
    --batch_size 64 \
    --output_path "${OUTPUT_DIR}" \
    --log_samples \
    --verbosity INFO \
    --limit 10 \
    --include_path /workspace/data/lmms-eval/lmms_eval/tasks/vqav2 \
    2>&1 | tee -a "${LOG_FILE}"
```


vqav2_val.yaml:

```
task: "vqav2_val"
include: _default_template_vqav2_yaml
test_split: validation
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: true
process_results: !function utils.vqav2_process_results_val
lmms_eval_specific_kwargs:
  default:
    pre_prompt: ""
    post_prompt: "Answer the question using a single word or phrase. for Unanswerable use: unanswerable/unknown/not possible/uncertain; for Don’t Know use: don’t know/unknown/cannot confirm; for False Questions use: none/bad question/cannot confirm/unanswerable; for Not Sure or low-quality evidence use: ambiguous/not sure/depend/not clear/it is difficult/hard to determine/i can’t; only use these words, keep replies short, no guessing, sorry is allowed."

  plm:
    pre_prompt: ""
    post_prompt: ""

```

output_jsonl file (sample):
`
{"doc_id": 0, "target": "answer", "filtered_resps": ["down"], "doc_hash": "74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", "exact_match": 1, "input": "Where is he looking?\nAnswer the question using a single word or phrase."}`




I fixed the issue because I built a new evaluation dataset based on VQAv2.
However, the task field in the YAML file didn’t change — it still used the original task name (idk_vqav2), even though the file name was updated.

As a result, the evaluation always pointed to the new handmade dataset’s YAML file at
/lmms-eval/lmms_eval/tasks/idk_vqav2/idk_vqav2_val.yaml,
since the task name (task: idk_vqav2) determines which configuration to load.


```
task: idk_vqav2  # originally vqav2
include: _default_template_idk_vqav2_yaml
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: No "llava_llama_3" template
Root cause: Unimplemented Feature Gap

Hello!
I am evaluating the llava-next-llama-3-8b model using lmms-eval, meeting this bug:
```
  File "lmms-eval/lmms_eval/models/llava.py", line 358, in generate_until
    conv = copy.deepcopy(conv_templates[self.conv_template])
KeyError: 'llava_llama_3'
```

It seems that there does not exist a conversational template called llava_llama_3. Can you help with that? Thank you!

Hi, may I ask which repo you used to install llava? If you want to use the llava-next-llama-3-8b, you might need to use the newest [`LLaVA-NeXT`](https://github.com/LLaVA-VL/LLaVA-NeXT) repo to install llava.

Got it. Thank you! I used the original llava. I will have a try
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: llava_1.5 配不好环境  env aways Error
Root cause: Documentation Deficiency

我根据流程搭环境，在conda里搭环境好几次，都失败
1.conda create -n xxxx python=3.10 
2. clone LLaVA1.5
3. pip install -e .[train]
4. clone lmms-eval
5. pip install -e .
6. python3 -m accelerate.commands.launch     --num_processes=1     -m lmms_eval     --model llava     --model_args pretrained="~/llava-v1.5-7b,device_map=cuda:1"     --tasks mme     --batch_size 1     --log_samples     --log_samples_suffix llava_v1.5_mme     --output_path ./logs/


然后就报错缺库

可不可以出一个官方的安装教程，感谢

解决了吗？

Would recommend to llava_hf instead of original llava for an easier env setup. Otherwise, you might need to follow the official instruction from llava for installation.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Question about evaluation results for llava-v1.6-mistral-7b
Root cause: Validation Gap

Hello. Recently I used the lmms-eval repo to evaluate the [checkpoint](https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b). However, the evalution results are different from here [link](https://github.com/haotian-liu/LLaVA/blob/main/docs/MODEL_ZOO.md). 

<img width="841" height="255" alt="Image" src="https://github.com/user-attachments/assets/992b9ace-bb56-4b78-bcf9-1ba8ebc77f9d" />

<img width="861" height="256" alt="Image" src="https://github.com/user-attachments/assets/70548ea2-3811-414d-80bf-516ebfc7d8a6" />

<img width="871" height="214" alt="Image" src="https://github.com/user-attachments/assets/f6984971-0be3-4f07-99f6-44a94fba493e" />
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Requirements for Llava v1.5-7b
Root cause: Documentation Deficiency

python                       3.12.3
torch                         2.9.0a0+50eac811a6.nv25.9
cuda                          13.0
transformers                  4.37.2
llava                         1.1.3                     /mnt/volumes/.../LLaVA-1.2.0
lmms_eval                     0.5.0                     /mnt/volumes/.../lmms-eval-main

key refs:
[保姆级llava-v1.5-7b部署教程](https://blog.csdn.net/kikiLQQ/article/details/135613642) 这个很重要，下载模型的过程建议看一遍（llava的ViT要单独下载）

bash：
```
# run once
# # install lmms_eval without building dependencies
cd lmms_eval;
pip install --no-deps -U -e .

# install LLaVA without building dependencies
cd LLaVA
pip install --no-deps -U -e .

# install all the requirements that require for reproduce llava results
pip install -r llava_repr_requirements.txt

pip install transformers==4.37.2 tokenizers -U

export CUDA_VISIBLE_DEVICES=0 #,1,2,3,4,5,6
export OPENAI_API_KEY="not used"
export HF_HOME="/lpai/volumes/.../huggingface"
export HF_HUB_OFFLINE=

export HF_TOKEN="...seems not used"
export HF_HUB_ENABLE_HF_TRANSFER="1"
export REKA_API_KEY="not used"
# Other possible environment variables include 
# ANTHROPIC_API_KEY,DASHSCOPE_API_KEY etc.

export DATASET_NAME=mme

# Run and exactly reproduce llava_v1.5 results!
# mme as an example

accelerate launch --num_processes=1 -m \
    lmms_eval \
    --model llava   \
    --model_args pretrained="/lpai/volumes/.../model/llava-v1.5-7b-from-lpai/llava-v1.5-7b,device_map=auto"   \
    --tasks $DATASET_NAME  \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix reproduce \
    --output_path ./logs/


```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: qwen3vl evaluate refcoco and IOU are very low
Root cause: Validation Gap



```bash
MODEL="./models/Qwen3VL"
export HF_HOME="path"
export HF_DATASETS_OFFLINE=1
export HF_HUB_OFFLINE=1
export CUDA_VISIBLE_DEVICES=1,2,3,6
TENSOR_PARALLEL_SIZE=4  # Number of GPUs for tensor parallelism
DATA_PARALLEL_SIZE=1     # Number of GPUs for data parallelism
GPU_MEMORY_UTILIZATION=0.85  # Fraction of GPU memory to use (0.0 - 1.0)
BATCH_SIZE=1                # Batch size for evaluation
TASKS="refcoco_bbox_testB"
OUTPUT_PATH="./lmms-eval/logs/qwen3vl_vllm"
LOG_SAMPLES=true
LOG_SUFFIX="qwen3vl_vllm"
export NCCL_BLOCKING_WAIT=1
export NCCL_TIMEOUT=18000000
CMD="uv run python -m lmms_eval \
    --model vllm \
    --model_args model=${MODEL},tensor_parallel_size=${TENSOR_PARALLEL_SIZE},data_parallel_size=${DATA_PARALLEL_SIZE},gpu_memory_utilization=${GPU_MEMORY_UTILIZATION},max_model_len=16384 \
    --tasks ${TASKS} \
    --batch_size ${BATCH_SIZE} \
    --output_path ${OUTPUT_PATH}\
    --verbosity=DEBUG"
```

Issue 98 has been reviewed, and rec_utils.py and the prompt words have been modified.
can you help to solving this problem?

Thank you for your contributions!

Use this I get right result in rec_utils.py

```
# qwen3vl
def refcoco_bbox_rec_doc_to_text(doc):
    assert isinstance(doc["answer"], str), "Answer must be a string"
    return (
       "Bounding box coordinates are specified in the format (top-left x, top-left y, bottom-right x, bottom-right y). All values are floating point numbers bounded between 0 and 1000. Please provide the bounding box coordinate of the region this sentence describes: " + doc["answer"]
    )

# qwen3vl
def parse_float_sequence_within(input_str, scale_ratio=1000):
    
    # Define the regex pattern to find the first instance of four floats within square brackets
    pattern = r"\[\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\s*\]"
    match = re.search(pattern, input_str)
    
    if not match:
        # If no match found, check for other patterns (parentheses and pairs)
        pattern = r"\(\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\s*\)"
        match = re.search(pattern, input_str)
        
    if not match:
        # Check for pairs of (num, num) format
        pattern = r"\(\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\),\s*\(\s*(-?\d+(?:\.\d+)?),\s*(-?\d+(?:\.\d+)?)\)"
        match = re.search(pattern, input_str)
    
    # If no pattern match was found, return the null sequence
    if not match:
        return [0, 0, 0, 0]

    # If a match is found, convert the captured groups into a list of floats
    values = [float(match.group(i)) for i in range(1, 5)]
    
    # Scale values if any of them are greater than 1
    if any(v > 1 for v in values):
        values = [v / scale_ratio for v in values]
    
    return values
```
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: vllm eval qwen2.5-vl error
Root cause: Validation Gap

hello, @kcz358 ,

When I set the `model_type` to `vllm`, the following error occurs and `qwen2_5_vl` has no error:
```bash
  File "lmms-eval/lmms_eval/models/chat/vllm.py", line 53, in make_one_request
    raw_messages = doc_to_messages(self.task_dict[task][split][doc_id])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lmms-eval/lmms_eval/api/task.py", line 1677, in doc_to_messages
    return auto_doc_to_messages(doc)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lmms-eval/lmms_eval/api/task.py", line 1660, in auto_doc_to_messages
    visuals = self.doc_to_visual(doc)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "lmms_eval/api/task.py", line 1384, in doc_to_visual
    else self.config.doc_to_visual(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lmms-eval/lmms_eval/tasks/gqa/utils.py", line 15, in gqa_doc_to_visual
    image = GQA_ID2IMAGE[doc["imageId"]]
            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'n54424'
```

my run scripts as follows:

```bash
model_args: model=qwen2.5_7b_vl,tensor_parallel_size=1,data_parallel_size=8,gpu_memory_utilization=0.78                      
tasks: mmmu_val,chartqa,mmbench_en_dev,pope,docvqa_val,ai2d,gqa,mme,realworldqa,textvqa_val,mmvet,mathvista,ocrbench   
CMD="accelerate launch --num_processes=${NUM_GPUS} --main_process_port=10392 -m lmms_eval \
        --model ${model_type} \
        --model_args ${model_args} \
        --tasks ${tasks} \
        --batch_size ${BATCH_SIZE} \
        --log_samples \
        --log_samples_suffix all_${task_type} \
        --output_path ${output_path} ${extra_kwargs}"

```

I can't reproduce this error when I test gqa with llava ov 1d5. May I ask if you still encounter this error if you are testing the gqa soley

If only GQA is tested, this error will still be reported.

```bash
Using test mode
model: Qwen/Qwen2.5-VL-7B-Instruct
model_type: vllm
extra_kwargs: 
log_file: logs_eval/eval_vllm/Qwen2.5-VL-7B-Instruct_base_images.log
output_path: logs_eval/eval_vllm/res/Qwen2.5-VL-7B-Instruct_base_images
model_args: model=Qwen/Qwen2.5-VL-7B-Instruct,tensor_parallel_size=1,data_parallel_size=8,gpu_memory_utilization=0.78
tasks: gqa
accelerate launch --num_processes=8 --main_process_port=10392 -m lmms_eval --model vllm --model_args model=Qwen/Qwen2.5-VL-7B-Instruct,tensor_parallel_size=1,data_parallel_size=8,gpu_memory_utilization=0.78 --tasks gqa --batch_size 64 --log_samples --log_samples_suffix all_images --output_path logs_eval/eval_vllm/res/Qwen2.5-VL-7B-Instruct_base_images
```

```bash
  File "lmms-eval/lmms_eval/api/task.py", line 1660, in auto_doc_to_messages
    visuals = self.doc_to_visual(doc)
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "lmms-eval/lmms_eval/api/task.py", line 1384, in doc_to_visual
    else self.config.doc_to_visual(
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "lmms-eval/lmms_eval/tasks/gqa/utils.py", line 15, in gqa_doc_to_visual
    image = GQA_ID2IMAGE[doc["imageId"]]
            ~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'n382416'
```

Hi, I think I have fixed it in the latest main.

Great job!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: 执行textvqa任务时卡住
Root cause: Resource Mishandling

```
CUDA_VISIBLE_DEVICES=4 accelerate launch --mixed_precision=bf16 --num_processes=1 --main_process_port=12346 -m lmms_eval --model qwen3_vl --model_args pretrained=/home/models/models--Qwen--Qwen3-VL-4B-Instruct,attn_implementation=flash_attention_2,interleave_visuals=False --tasks textvqa --batch_size 32 --output_path /home/output_lmms/Qwen3-VL-4B-Instruct --log_samples --verbosity DEBUG
The following values were not passed to `accelerate launch` and had defaults used instead:
        `--num_machines` was set to a value of `1`
        `--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2025-11-12 02:01:57 | INFO     | __main__:cli_evaluate:311 - Verbosity set to DEBUG
2025-11-12 02:01:57 | DEBUG    | lmms_eval.tasks:_get_task_and_group:484 - File _default_template.yaml in /data2/zhangdeming/VLMZip/lmms-eval/lmms_eval/tasks/video-tt could not be loaded as a task or group
2025-11-12 02:01:58 | DEBUG    | lmms_eval.tasks:_get_task_and_group:458 - `group` and `group_alias` keys in tasks' configs will no longer be used in the next release of lmms-eval. `tag` will be used to allow to call a collection of tasks just like `group`. `group` will be removed in order to not cause confusion with the new ConfigurableGroup which will be the offical way to create groups with addition of group-wide configuations.
2025-11-12 02:01:58 | DEBUG    | lmms_eval.tasks:_get_task_and_group:484 - File illusionvqa.yaml in /data2/zhangdeming/VLMZip/lmms-eval/lmms_eval/tasks/illusionvqa could not be loaded as a task or group
2025-11-12 02:01:59 | INFO     | __main__:cli_evaluate_single:400 - Evaluation tracker args: {'output_path': '/data2/zhangdeming/VLMZip/output_lmms/Qwen3-VL-4B-Instruct'}
2025-11-12 02:01:59 | INFO     | __main__:cli_evaluate_single:480 - Selected Tasks: ['textvqa']
2025-11-12 02:01:59 | INFO     | lmms_eval.evaluator:simple_evaluate:161 - Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00,  1.08s/it]
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 10389.66it/s]
Resolving data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 11908.87it/s]
```


其他任务（"docvqa_val,chartqa,ocrbench,scienceqa_img,ai2d,mmmu_val,mme,pope"）都没有问题，就添加textvqa会卡在这里
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Prompt for MathVista
Root cause: Documentation Deficiency

I noticed that there are several kinds of prompts for MathVista. Which one is used for the reported number of LLaVA-NeXT? Thank you.

![image](https://github.com/EvolvingLMMs-Lab/lmms-eval/assets/55196500/854d65f2-6d8f-411b-b3e8-3c22584f04bd)


We use the `default` for llava-next.

Got it, Thank you!
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: OCRBench v2
Root cause: Validation Gap

I got an error when I eval the OCRBench V2:
[rank1]:   lmms_eval/lmms_eval/tasks/ocrbench_v2/utils.py", line 107, in ocrbench_v2_process_results                      
[rank1]:     if doc["eval"] == "multiple choice":                                                                                                                
[rank1]:        ~~~^^^^^^^^                                                                                                                                      
[rank1]: KeyError: 'eval'
How can I resolve this? please

lmms-eval下载的OCRBenchV2和原始的数据集有差异，重新下载替换一下就好了
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: exceed the api limitation in charxiv
Root cause: Architectural Constraint

Hi, thanks for your great work on lmms-eval!

I am running CharXiv reasoning evaluation on Qwen2.5-VL, and I frequently hit the OpenAI API rate limit:
`
Error code: 429 - Rate limit reached for gpt-4o (TPM: Limit 30000, Used 29527, Requested 484)`

To reduce the API usage, I tried to manually modify the code in:

`lmms_eval/tasks/charxiv/reasoning_utils.py`

specifically changing:
`model="gpt-4o-2024-05-13"`
to
`model="gpt-4o-mini"`

However, during evaluation it still calls gpt-4o, not gpt-4o-mini.
It seems that the model name is hard-coded somewhere else in the CharXiv pipeline.


Where is the correct place to configure the GPT model used for CharXiv reasoning and grading?
I want to switch to gpt-4o-mini (or a different local judge) to avoid the TPM rate limit.

I think probably because their original api have different parameters for descriptive and reasoning. So I copied from theirs and change to config here for the descriptive.

https://github.com/EvolvingLMMs-Lab/lmms-eval/blob/4993e6e2669523e9e79976bf25ea1810cb4902f1/lmms_eval/tasks/charxiv/utils.py#L101-L104
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: The LLM Judge function takes too long, resulting in low GPU utilization.
Root cause: Documentation Deficiency

The GPU is used normally during the model inference stage, but the LLM judge is time-consuming when calculating the score, resulting in low overall GPU utilization. Are there any methods or instructions to use parallel processing when calling the LLM judge API, or to separate the inference and scoring processes?
Thank you!

The reason is probably because you are sending requests one-by-one. I would recommend you to change the call llm as judge that in `process_result` use threading. Also one thing to notice is that if you put the function in `process_result` when launching the lmms-eval on different gpu cards, by default it is parallel with `num_gpu` workers
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: how to use “few shot”

is there an example of how to use "few shot"

how to set this two args:

(1)fewshot_split (str, optional) — Split in the dataset to draw few-shot exemplars from. assert that this not None if num_fewshot > 0. This function is not well tested so far
(2)num_fewshot (int, optional, defaults to 0) — Number of few-shot examples before the input. This function is not well tested so far

This is still work in progress. We developed it and succesfully test on `textvqa`. But this is not confirmed fully correct. You can see there's a PR. You may checkout it if you really need.
Repository: EvolvingLMMs-Lab/lmms-eval
Issue: Misaligned question assignment in CharXiv descriptive evaluation
Root cause: Algorithmic Error

Hi, thanks for maintaining this excellent evaluation framework! I've been using it for research and noticed an issue with the CharXiv descriptive task.

**Problem:**
In `charxiv_descriptive_process_docs` of `lmms_eval/tasks/charxiv/utils.py`, the line `q_number = indice % 4 + 1` doesn't correctly assign questions after `dataset.repeat(4)`. 

Say if I have a dataset of 1000 samples, after `repeat(4)`, the dataset structure is:
- Indices 0-999: Copy 1
- Indices 1000-1999: Copy 2
- Indices 2000-2999: Copy 3
- Indices 3000-3999: Copy 4

However, the modulo operation gives the same result for all copies of the same example:
- Index 0: 0 % 4 = 0, q_number = 1
- Index 1000: 1000 % 4 = 0, q_number = 1 (should be 2)
- Index 2000: 2000 % 4 = 0, q_number = 1 (should be 3)
- Index 3000: 3000 % 4 = 0, q_number = 1 (should be 4)

This means all four copies use descriptive_q1, so only one of the four question types is ever evaluated.
I wanted to check if this is the intended behavior or if I'm misunderstanding the logic here?

**Proposed fix (if this is indeed a problem):**
Perhaps replace `q_number = indice % 4 + 1` with `q_number = indice // original_size + 1` where original_size is 1000 in this example case.

This correctly maps:
- Indices 0-999: q_number = 1
- Indices 1000-1999: q_number = 2
- Indices 2000-2999: q_number = 3
- Indices 3000-3999: q_number = 4

Would you please confirm the logic here? Thanks very much!