Repository: open-compass/VLMEvalKit
Issue: RuntimeError when running evaluation on video datasets
Root cause: External Dependency Breakage

I am currently trying to evaluate multiple qwen series models, some are baseline and some are trained.

- Qwen2-VL-2B (lora FT, full FT) 
- Qwen2-VL-2B-Instruct (baseline, lora FT, and full FT)
- Qwen2.5-VL-3B-Instruct (baseline, lora FT, and full FT)

It works when I tried to evaluate with image benchmark datasets, but I encountered errors on multiple video benchmark datasets.

I have tried running evaluation on these datasets with these fps settings:
- MMBench_Video_1fps_nopack
- Video-MME_0.5fps
- MVBench_MP4_1fps
- LongVideoBench_0.5fps
- MLVU_1fps

So far, only evaluation on **MVBench dataset** succeded. On other video benchmark datasets, I encountered this error:
`RuntimeError: Triton Error [CUDA]: invalid argument`

Can anyone help me resolve this issue or provide me some insights on what may be causing it? Thanks in advance.
For context, I am running the code on a H100 server with 1 80G GPU card

Full error log:
[2025-06-30 07:31:31] ERROR - RUN - run.py: main - 503: Model Qwen2.5-VL-3B-Instruct-permian_v0 x Dataset MLVU_1fps combination failed: Triton Error [CUDA]: invalid argument, skipping this combination.
Traceback (most recent call last):
  File "/notebooks/beatrice/vlm_benchmark_code/run.py", line 353, in main
    model = infer_data_job_video(
  File "/notebooks/beatrice/vlm_benchmark_code/vlmeval/inference_video.py", line 184, in infer_data_job_video
    model = infer_data(
  File "/notebooks/beatrice/vlm_benchmark_code/vlmeval/inference_video.py", line 148, in infer_data
    response = model.generate(message=struct, dataset=dataset_name)
  File "/notebooks/beatrice/vlm_benchmark_code/vlmeval/vlm/base.py", line 116, in generate
    return self.generate_inner(message, dataset)
  File "/notebooks/beatrice/vlm_benchmark_code/vlmeval/vlm/qwen2_vl/model.py", line 621, in generate_inner
    return self.generate_inner_transformers(message, dataset=dataset)
  File "/notebooks/beatrice/vlm_benchmark_code/vlmeval/vlm/qwen2_vl/model.py", line 471, in generate_inner_transformers
    generated_ids = self.model.generate(
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 2597, in generate
    result = self._sample(
  File "/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py", line 3557, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py", line 969, in wrapper
    output = func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1908, in forward
    outputs = self.model(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1678, in forward
    video_embeds = self.get_video_features(pixel_values_videos, video_grid_thw)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1600, in get_video_features
    video_embeds = self.visual(pixel_values_videos, grid_thw=video_grid_thw)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 530, in forward
    hidden_states = blk(hidden_states, cu_seqlens=cu_seqlens_now, position_embeddings=position_embeddings)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 341, in forward
    hidden_states = hidden_states + self.attn(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1762, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 189, in forward
    q, k = apply_rotary_pos_emb_flashatt(q.unsqueeze(0), k.unsqueeze(0), cos, sin)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 156, in apply_rotary_pos_emb_flashatt
    q_embed = apply_rotary_emb(q.float(), cos.float(), sin.float()).type_as(q)
  File "/usr/local/lib/python3.10/dist-packages/flash_attn/layers/rotary.py", line 121, in apply_rotary_emb
    return ApplyRotaryEmb.apply(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 575, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/usr/local/lib/python3.10/dist-packages/flash_attn/layers/rotary.py", line 51, in forward
    out = apply_rotary(
  File "/usr/local/lib/python3.10/dist-packages/flash_attn/ops/triton/rotary.py", line 159, in apply_rotary
    torch.library.wrap_triton(rotary_kernel)[grid](
  File "/usr/local/lib/python3.10/dist-packages/torch/_higher_order_ops/triton_kernel_wrap.py", line 1812, in __call__
    return tracing_triton_hopifier_singleton.call_triton_kernel(
  File "/usr/local/lib/python3.10/dist-packages/torch/_higher_order_ops/triton_kernel_wrap.py", line 1670, in call_triton_kernel
    return self.call_HOP(variable, grids, combined_args_raw, tx)
  File "/usr/local/lib/python3.10/dist-packages/torch/_higher_order_ops/triton_kernel_wrap.py", line 1766, in call_HOP
    return triton_kernel_wrapper_mutation(
  File "/usr/local/lib/python3.10/dist-packages/torch/_higher_order_ops/triton_kernel_wrap.py", line 783, in __call__
    return super().__call__(
  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 471, in __call__
    return wrapper()
  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 467, in wrapper
    return self.dispatch(
  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 455, in dispatch
    return kernel(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_higher_order_ops/triton_kernel_wrap.py", line 886, in triton_kernel_wrapper_mutation_dense
    kernel[grid_fn](*args, **kwargs, **constant_args)
  File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 347, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/triton/runtime/jit.py", line 591, in run
    kernel.run(grid_0, grid_1, grid_2, stream, kernel.function, kernel.packed_metadata,
  File "/usr/local/lib/python3.10/dist-packages/triton/backends/nvidia/driver.py", line 529, in __call__
    self.launch(gridX, gridY, gridZ, stream, function, self.launch_cooperative_grid, global_scratch, *args)
RuntimeError: Triton Error [CUDA]: invalid argument