Repository: modelscope/evalscope
Issue: 评测Gemma3系列模型时出现RuntimeError: p.attn_bias_ptr is not correctly aligned错误
Root cause: External Dependency Breakage

## 问题描述 / Issue Description

使用MMLU-PRO评测gemma3-sft模型时，出现RuntimeError: p.attn_bias_ptr is not correctly aligned错误 / Please briefly describe the issue you encountered.

## 使用的工具 / Tools Used
- ✅ Native / 原生框架
- [ ] Opencompass backend
- [ ] VLMEvalKit backend
- [ ] RAGEval backend
- [ ] Perf / 模型推理压测工具
- [ ] Arena  /竞技场模式


## 执行的代码或指令 / Code or Commands Executed

请提供您执行的主要代码或指令。 / Please provide the main code or commands you executed. 例如 / For example:

```python
from evalscope.run import run_task
from evalscope.config import TaskConfig

task_cfg = TaskConfig(
    model='/home/dataset-assist-0/zgy/LLaMA-Factory/output/gemma3_lora_sft',
    datasets=['mmlu_pro'],
    dataset_args={"mmlu_pro": {"filters": {"remove_until": "</think>"}}},  # 数据集参数
    limit=5,
)

run_task(task_cfg=task_cfg)
```

```bash
python evaluation/evl.py
```

## 错误日志 / Error Log

请粘贴完整的错误日志或控制台输出。 / Please paste the full error log or console output. 例如 / For example:

```
2025-03-23 14:14:43,798 - evalscope - INFO - Args: Task config is provided with TaskConfig type.
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:14<00:00,  2.83s/it]
2025-03-23 14:15:00,163 - evalscope - WARNING - Got local model dir: /home/dataset-assist-0/zgy/LLaMA-Factory/output/gemma3_lora_sft
2025-03-23 14:15:00,164 - evalscope - INFO - Updating generation config ...
2025-03-23 14:15:00,227 - evalscope - INFO - Dump task config to ./outputs/20250323_141443/configs/task_config_613b19.yaml
2025-03-23 14:15:00,232 - evalscope - INFO - {
    "model": "/home/dataset-assist-0/zgy/LLaMA-Factory/output/gemma3_lora_sft",
    "model_id": "gemma3_lora_sft",
    "model_args": {
        "revision": "master",
        "precision": "torch.float16"
    },
    "template_type": null,
    "chat_template": null,
    "datasets": [
        "mmlu_pro"
    ],
    "dataset_args": {
        "mmlu_pro": {
            "name": "mmlu_pro",
            "dataset_id": "modelscope/MMLU-Pro",
            "model_adapter": "generation",
            "output_types": [
                "multiple_choice_logits",
                "generation"
            ],
            "subset_list": [
                "computer science",
                "math",
                "chemistry",
                "engineering",
                "law",
                "biology",
                "health",
                "physics",
                "business",
                "philosophy",
                "economics",
                "other",
                "psychology",
                "history"
            ],
            "metric_list": [
                "AverageAccuracy"
            ],
            "few_shot_num": 5,
            "few_shot_random": false,
            "train_split": "validation",
            "eval_split": "test",
            "prompt_template": "The following are multiple choice questions (with answers) about {subset_name}. Think step by step and then finish your answer with \"the answer is (X)\" where X is the correct letter choice.\n{query}",
            "system_prompt": null,
            "query_template": null,
            "pretty_name": "MMLU-Pro",
            "filters": {
                "remove_until": "</think>"
            },
            "extra_params": {}
        }
    },
    "dataset_dir": "/root/.cache/modelscope/hub/datasets",
    "dataset_hub": "modelscope",
    "generation_config": {
        "max_length": 2048,
        "max_new_tokens": 512,
        "do_sample": false,
        "top_k": 50,
        "top_p": 1.0,
        "temperature": 1.0
    },
    "eval_type": "checkpoint",
    "eval_backend": "Native",
    "eval_config": null,
    "stage": "all",
    "limit": 5,
    "eval_batch_size": 1,
    "mem_cache": false,
    "use_cache": null,
    "work_dir": "./outputs/20250323_141443",
    "outputs": null,
    "debug": false,
    "dry_run": false,
    "seed": 42,
    "api_url": null,
    "api_key": "EMPTY",
    "timeout": null,
    "stream": false,
    "judge_strategy": "auto",
    "judge_worker_num": 8,
    "judge_model_args": {}
}
2025-03-23 14:15:00,234 - evalscope - INFO - **** Start evaluating on dataset modelscope/MMLU-Pro ****
2025-03-23 14:15:00,234 - evalscope - INFO - Loading dataset from hub: modelscope/MMLU-Pro
2025-03-23 14:15:00,451 - evalscope - INFO - Loading dataset: dataset_name: modelscope/MMLU-Pro > subsets: ['default']
2025-03-23 14:15:00,451 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 14:15:01,795 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 14:15:01,795 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 14:15:01,795 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 14:15:10,449 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 14:15:11,180 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 14:15:11,181 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 14:15:11,181 - modelscope - WARNING - Use trust_remote_code=True. Will invoke codes from modelscope/MMLU-Pro. Please make sure that you can trust the external codes.
2025-03-23 14:15:17,308 - evalscope - INFO - Use default settings: > few_shot_num: 5, > few_shot_split: validation, > target_eval_split: test
Predicting(computer science):   0%|                                                                                                                            | 0/5 [00:00<?, ?it/s]Asking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.
Predicting(computer science): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [02:51<00:00, 34.29s/it]
2025-03-23 14:18:09,001 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_computer science.jsonl.
Reviewing(computer science): 100%|██████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 561.08it/s]
Predicting(math): 100%|██████████████████████████████████████████████████████████████████████| 5/5 [02:51<00:00, 34.28s/it]
2025-03-23 14:21:00,408 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_math.jsonl.
Reviewing(math): 100%|█████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1080.84it/s]
Predicting(chemistry): 100%|█████████████████████████████████████████████████████████████████| 5/5 [02:48<00:00, 33.80s/it]
2025-03-23 14:23:49,405 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_chemistry.jsonl.
Reviewing(chemistry): 100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1068.18it/s]
Predicting(engineering): 100%|███████████████████████████████████████████████████████████████| 5/5 [02:49<00:00, 33.97s/it]
2025-03-23 14:26:39,258 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_engineering.jsonl.
Reviewing(engineering): 100%|██████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1052.26it/s]
Predicting(law): 100%|███████████████████████████████████████████████████████████████████████| 5/5 [02:49<00:00, 33.93s/it]
2025-03-23 14:29:28,901 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_law.jsonl.
Reviewing(law): 100%|███████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 977.01it/s]
Predicting(biology): 100%|███████████████████████████████████████████████████████████████████| 5/5 [02:48<00:00, 33.80s/it]
2025-03-23 14:32:17,904 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_biology.jsonl.
Reviewing(biology): 100%|██████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1079.73it/s]
Predicting(health): 100%|████████████████████████████████████████████████████████████████████| 5/5 [02:49<00:00, 33.95s/it]
2025-03-23 14:35:07,677 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_health.jsonl.
Reviewing(health): 100%|████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 629.76it/s]
Predicting(physics): 100%|███████████████████████████████████████████████████████████████████| 5/5 [02:53<00:00, 34.69s/it]
2025-03-23 14:38:01,145 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_physics.jsonl.
Reviewing(physics): 100%|██████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1024.20it/s]
Predicting(business): 100%|██████████████████████████████████████████████████████████████████| 5/5 [02:53<00:00, 34.61s/it]
2025-03-23 14:40:54,194 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_business.jsonl.
Reviewing(business): 100%|█████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1101.62it/s]
Predicting(philosophy): 100%|████████████████████████████████████████████████████████████████| 5/5 [02:53<00:00, 34.63s/it]
2025-03-23 14:43:47,374 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_philosophy.jsonl.
Reviewing(philosophy): 100%|███████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1047.79it/s]
Predicting(economics): 100%|█████████████████████████████████████████████████████████████████| 5/5 [02:53<00:00, 34.78s/it]
2025-03-23 14:46:41,287 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_economics.jsonl.
Reviewing(economics): 100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1053.00it/s]
Predicting(other): 100%|█████████████████████████████████████████████████████████████████████| 5/5 [02:53<00:00, 34.64s/it]
2025-03-23 14:49:34,472 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_other.jsonl.
Reviewing(other): 100%|████████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 1166.38it/s]
Predicting(psychology): 100%|████████████████████████████████████████████████████████████████| 5/5 [02:53<00:00, 34.62s/it]
2025-03-23 14:52:27,581 - evalscope - INFO - Dump predictions to ./outputs/20250323_141443/predictions/gemma3_lora_sft/mmlu_pro_psychology.jsonl.
Reviewing(psychology): 100%|████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 572.35it/s]
Predicting(history):  40%|██████████████████████████▊                                        | 2/5 [01:10<01:45, 35.08s/it]
Traceback (most recent call last):
  File "/home/dataset-assist-0/zgy/LLaMA-Factory/evaluation/evl.py", line 11, in <module>
    run_task(task_cfg=task_cfg)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/run.py", line 31, in run_task
    return run_single_task(task_cfg, run_time)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/run.py", line 44, in run_single_task
    return evaluate_model(task_cfg, outputs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/run.py", line 117, in evaluate_model
    res_dict = evaluator.eval()
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 408, in eval
    answers_list: list = self.get_answers(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 184, in get_answers
    answer_ds: List[dict] = self._get_answer(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/evaluator/evaluator.py", line 105, in _get_answer
    answer_ds: List[dict] = self.model_adapter.predict(inputs=input_prompts, infer_cfg=infer_cfg)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/models/chat_adapter.py", line 144, in predict
    responses = self._model_generate(queries, system_prompts, infer_cfg)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/evalscope/models/chat_adapter.py", line 112, in _model_generate
    output_ids = self.model.generate(**inputs, generation_config=self.generation_config)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/generation/utils.py", line 2326, in generate
    result = self._sample(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/generation/utils.py", line 3289, in _sample
    outputs = model_forward(**model_inputs, return_dict=True)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func
    return func(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 978, in forward
    outputs = self.model(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 756, in forward
    layer_outputs = decoder_layer(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 445, in forward
    hidden_states, self_attn_weights = self.self_attn(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 367, in forward
    attn_output, attn_weights = attention_interface(
  File "/opt/conda/envs/landai/lib/python3.10/site-packages/transformers/integrations/sdpa_attention.py", line 54, in sdpa_attention_forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(
RuntimeError: p.attn_bias_ptr is not correctly aligned
```

## 运行环境 / Runtime Environment

- 操作系统 / Operating System:
  - [ ] Windows
  - [ ] macOS
  - ✅ Ubuntu

- Python版本 / Python Version:
  - [ ] 3.11
  - ✅ 3.10
  - [ ] 3.9



参考https://github.com/google-deepmind/gemma/issues/169
已经修正此错误。
需要在[evalscope](https://github.com/modelscope/evalscope/tree/main)/[evalscope](https://github.com/modelscope/evalscope/tree/main/evalscope)/[models](https://github.com/modelscope/evalscope/tree/main/evalscope/models)/chat_adapter.py中修改，添加以下内容：
inputs = tokenizer(
prompts,
return_tensors="pt",
padding="longest",
pad_to_multiple_of=8, # <-- ensures seq_len % 8 == 0
).to(model.device)

期待后续团队支持gemma3评测的兼容，这样改了之后模型依旧是没有输出，但就只对本问题而言，已经解决