Repository: EleutherAI/lm-evaluation-harness
Issue: Fail to Run on Gemma3
Root cause: External Dependency Breakage

Code:
`accelerate launch --num_processes 8 -m lm_eval --model hf --model_args pretrained=google/gemma-3-27b-it,trust_remote_code=True,dtype=bfloat16,parallelize=True --tasks leaderboard_bbh --num_fewshot 3 --batch_size auto:1`

Error:
```
[rank6]: Traceback (most recent call last):
[rank6]:   File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/runpy.py", line 196, in _run_module_as_main
[rank6]:     return _run_code(code, main_globals, None,
[rank6]:   File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/runpy.py", line 86, in _run_code
[rank6]:     exec(code, run_globals)
[rank6]:   File "/home/ec2-user/SageMaker/Zhichao/UFT_SFT_Compare_Gemma27B/lm-evaluation-harness/lm-evaluation-harness/lm_eval/__main__.py", line 512, in <module>
[rank6]:     cli_evaluate()
[rank6]:   File "/home/ec2-user/SageMaker/Zhichao/UFT_SFT_Compare_Gemma27B/lm-evaluation-harness/lm-evaluation-harness/lm_eval/__main__.py", line 432, in cli_evaluate
[rank6]:     results = evaluator.simple_evaluate(
[rank6]:   File "/home/ec2-user/SageMaker/Zhichao/UFT_SFT_Compare_Gemma27B/lm-evaluation-harness/lm-evaluation-harness/lm_eval/utils.py", line 439, in _wrapper
[rank6]:     return fn(*args, **kwargs)
[rank6]:   File "/home/ec2-user/SageMaker/Zhichao/UFT_SFT_Compare_Gemma27B/lm-evaluation-harness/lm-evaluation-harness/lm_eval/evaluator.py", line 221, in simple_evaluate
[rank6]:     lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
[rank6]:   File "/home/ec2-user/SageMaker/Zhichao/UFT_SFT_Compare_Gemma27B/lm-evaluation-harness/lm-evaluation-harness/lm_eval/api/model.py", line 151, in create_from_arg_string
[rank6]:     return cls(**args, **args2)
[rank6]:   File "/home/ec2-user/SageMaker/Zhichao/UFT_SFT_Compare_Gemma27B/lm-evaluation-harness/lm-evaluation-harness/lm_eval/models/huggingface.py", line 192, in __init__
[rank6]:     self._create_model(
[rank6]:   File "/home/ec2-user/SageMaker/Zhichao/UFT_SFT_Compare_Gemma27B/lm-evaluation-harness/lm-evaluation-harness/lm_eval/models/huggingface.py", line 588, in _create_model
[rank6]:     self._model = self.AUTO_MODEL_CLASS.from_pretrained(
[rank6]:   File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
[rank6]:     return model_class.from_pretrained(
[rank6]:   File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 273, in _wrapper
[rank6]:     return func(*args, **kwargs)
[rank6]:   File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4313, in from_pretrained
[rank6]:     model = cls(config, *model_args, **model_kwargs)
[rank6]:   File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 889, in __init__
[rank6]:     self.model = Gemma3TextModel(config)
[rank6]:   File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 622, in __init__
[rank6]:     self.vocab_size = config.vocab_size
[rank6]:   File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.10/site-packages/transformers/configuration_utils.py", line 214, in __getattribute__
[rank6]:     return super().__getattribute__(key)
[rank6]: AttributeError: 'Gemma3Config' object has no attribute 'vocab_size'
```

Reason: the Gemma3 needs to be loaded through Gemma3's config. Will you update to accept it?
```
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from PIL import Image
import requests
import torch

model_id = "google/gemma-3-27b-it"

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto"
).eval()

processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device, dtype=torch.bfloat16)

input_len = inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**inputs, max_new_tokens=100, do_sample=False)
    generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)
```

Hi, which transformers version are you using? Using source repo transformers version (`v4.52.0.dev0`) helps solve this issue. If it does not help, try using [this branch](https://github.com/huggingface/transformers/tree/v4.49.0-Gemma-3-release) in particular which is supposed to fix this bug. Hope this helps!

On related note, I have been struggling to run generation-related benchmarks on Gemma-3-27B model myself - I get accuracy very close to 0 on any benchmark of this type and saved model responses are empty. Can anyone verify if Gemma3 works for this set of tasks?

@ayukh Thank you for your help. 

I install this branch: [https://github.com/huggingface/transformers/tree/main](https://github.com/huggingface/transformers/tree/main)
Now, the problem is solved, and the obtained values are non-zero. Maybe you can try to update with this version of transformer.