Repository: modelscope/evalscope
Issue: 评估Gemma3系列模型时出现RuntimeError: CUDA error: device-side assert triggered错误
Root cause: Validation Gap

## 问题描述 / Issue Description

进行Gemma3系列模型评估时，一旦指定max_new_tokens或者max_tokens，便会出现RuntimeError: CUDA error: device-side assert triggered错误

## 使用的工具 / Tools Used
-  原生框架


## 执行的代码或指令 / Code or Commands Executed

```python
#!/bin/bash

evalscope eval \
  --model /home/dataset-assist-0/zgy/LLaMA-Factory/output/gemma3_lora_sft \
  --model-args "device_map=cuda" \
  --generation-config "max_tokens=10000" \
  --datasets mmlu_pro \
  --dataset-args '{"mmlu_pro": {"filters": {"remove_until": "</think>"}}}' \
  --limit 10
```

## 错误日志 / Error Log


```
2025-03-23 21:06:39,105 - evalscope - INFO - Args: Task config is provided with CommandLine type.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████| 5/5 [00:14<00:00,  2.83s/it]
2025-03-23 21:06:55,301 - evalscope - WARNING - Got local model dir: /home/dataset-assist-0/zgy/LLaMA-Factory/output/gemma3_lora_sft
2025-03-23 21:06:55,303 - evalscope - INFO - Updating generation config ...
2025-03-23 21:06:55,355 - evalscope - INFO - Dump task config to ./outputs/20250323_210639/configs/task_config_d677a6.yaml
2025-03-23 21:06:55,360 - evalscope - INFO - {
    "model": "/home/dataset-assist-0/zgy/LLaMA-Factory/output/gemma3_lora_sft",
    "model_id": "gemma3_lora_sft",
    "model_args": {
        "device_map": "cuda"
    },
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "mmlu_pro"
    ],
    "dataset_args": {
        "mmlu_pro": {
            "name": "mmlu_pro",
            "dataset_id": "modelscope/MMLU-Pro",
            "model_adapter": "generation",
            "output_types": [
                "multiple_choice_logits",
                "generation"
            ],
            "subset_list": [
                "computer science",
                "math",
                "chemistry",
                "engineering",
                "law",
                "biology",
                "health",
                "physics",
                "business",
                "philosophy",
                "economics",
                "other",
                "psychology",
                "history"
            ],
            "metric_list": [
                "AverageAccuracy"
            ],
            "few_shot_num": 5,
            "few_shot_random": false,
            "train_split": "validation",
            "eval_split": "test",
            "prompt_template": "The following are multiple choice questions (with answers) about {subset_name}. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n{query}",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "MMLU-Pro",
            "filters": {
                "remove_until": "</think>"
            },
            "extra_params": {}
        }
    },
    "dataset_dir": "/root/.cache/modelscope/hub/datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_tokens": 10000
    },
    "eval_type": "checkpoint",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": 10,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs/20250323_210639",
    "outputs": null,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": null,
    "api_key": "EMPTY",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 8,
    "judge_model_args": {}
}
2025-03-23 21:06:55,360 - evalscope - INFO - **** Start evaluating on dataset modelscope/MMLU-Pro ****
2025-03-23 21:06:55,360 - evalscope - INFO - Loading dataset from hub: modelscope/MMLU-Pro
2025-03-23 21:06:55,570 - evalscope - INFO - Loading dataset: dataset_name: modelscope/MMLU-Pro > subsets: ['default']
2025-03-23 21:06:55,570 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 21:06:56,973 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 21:06:56,973 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 21:06:56,973 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 21:07:07,707 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 21:07:08,308 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 21:07:08,309 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 21:07:08,309 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 21:07:19,737 - evalscope - INFO - Use default settings: > few_shot_num: 5, > few_shot_split: validation, > target_eval_split: test
Predicting(computer science):   0%|                                                                 | 0/10 [00:00<?, ?it/s]Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
/pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.
Predicting(computer science):   0%|                                                                 | 0/10 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/opt/conda/envs/landai/bin/evalscope", line 8, in <module>
    sys.exit(run_cmd())
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/cli/cli.py", line 25, in run_cmd
    cmd.execute()
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/cli/start_eval.py", line 32, in execute
    run_task(self.args)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    return evaluate_model(task_cfg, outputs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/run.py", line 117, in evaluate_model
    res_dict = evaluator.eval()
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 408, in eval
    answers_list: list = self.get_answers(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 184, in get_answers
    answer_ds: List[dict] = self._get_answer(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 105, in _get_answer
    answer_ds: List[dict] = self.model_adapter.predict(inputs=input_prompts, infer_cfg=infer_cfg)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/models/chat_adapter.py", line 144, in predict
    responses = self._model_generate(queries, system_prompts, infer_cfg)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/models/chat_adapter.py", line 112, in _model_generate
    output_ids = self.model.generate(**inputs, generation_config=self.generation_config)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/generation/utils.py", line 3332, in _sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  -  Ubuntu

- Python版本 / Python Version:
  -  3.10



gemma3是多模态模型，目前框架的chat_adapter对于多模态模型的支持不是很完善，建议使用模型推理框架（vllm等）拉起模型服务来进行评测

已经解决