Repository: bigcode-project/bigcode-evaluation-harness
Issue: Cannot run eval with local model directory
Root cause: Environment Incompatibility

Hi. Thank you for your hard work!

I am trying to run bigcode/starcoder model on a server. I downloaded the huggingface repo with git and transferred the folder to the server. 

To be extra sure I wasn't passing the path wrong, I modified main.py:
``` python
    parser.add_argument(
        "--model",
        default="/home/ubuntu/.cache/huggingface/hub/models--bigcode--starcoder/snapshots/8a57e3930912e5d22ddc4d5f46b4b99f169afbe9",
        help="Model to evaluate, provide a repo name in Hugging Face hub or a local path",
    )
```

Once I run `python main.py` I get:
``` bash
─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/ubuntu/star-coder/bigcode-evaluation-harness/main.py:230 in <module>                       │
│                                                                                                  │
│   227                                                                                            │
│   228                                                                                            │
│   229 if __name__ == "__main__":                                                                 │
│ ❱ 230 │   main()                                                                                 │
│   231                                                                                            │
│                                                                                                  │
│ /home/ubuntu/star-coder/bigcode-evaluation-harness/main.py:176 in main                           │
│                                                                                                  │
│   173 │   │   │   │   f"Non valid precision {args.precision}, choose from: fp16, fp32, bf16"     │
│   174 │   │   │   )                                                                              │
│   175 │   │   print(f"Loading tokenizer and model (in {args.precision})")                        │
│ ❱ 176 │   │   model = AutoModelForCausalLM.from_pretrained(                                      │
│   177 │   │   │   args.model,                                                                    │
│   178 │   │   │   revision=args.revision,                                                        │
│   179 │   │   │   torch_dtype=dict_precisions[args.precision],                                   │
│                                                                                                  │
│ /home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/auto/auto_factory.py:434 in  │
│ from_pretrained                                                                                  │
│                                                                                                  │
│   431 │   │   ]                                                                                  │
│   432 │   │   hub_kwargs = {name: kwargs.pop(name) for name in hub_kwargs_names if name in kwa   │
│   433 │   │   if not isinstance(config, PretrainedConfig):                                       │
│ ❱ 434 │   │   │   config, kwargs = AutoConfig.from_pretrained(                                   │
│   435 │   │   │   │   pretrained_model_name_or_path,                                             │
│   436 │   │   │   │   return_unused_kwargs=True,                                                 │
│   437 │   │   │   │   trust_remote_code=trust_remote_code,                                       │
│                                                                                                  │
│ /home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py:8 │
│ 29 in from_pretrained                                                                            │
│                                                                                                  │
│   826 │   │   │   )                                                                              │
│   827 │   │   │   return config_class.from_pretrained(pretrained_model_name_or_path, **kwargs)   │
│   828 │   │   elif "model_type" in config_dict:                                                  │
│ ❱ 829 │   │   │   config_class = CONFIG_MAPPING[config_dict["model_type"]]                       │
│   830 │   │   │   return config_class.from_dict(config_dict, **unused_kwargs)                    │
│   831 │   │   else:                                                                              │
│   832 │   │   │   # Fallback: use pattern matching on the string.                                │
│                                                                                                  │
│ /home/ubuntu/.local/lib/python3.8/site-packages/transformers/models/auto/configuration_auto.py:5 │
│ 36 in __getitem__                                                                                │
│                                                                                                  │
│   533 │   │   if key in self._extra_content:                                                     │
│   534 │   │   │   return self._extra_content[key]                                                │
│   535 │   │   if key not in self._mapping:                                                       │
│ ❱ 536 │   │   │   raise KeyError(key)                                                            │
│   537 │   │   value = self._mapping[key]                                                         │
│   538 │   │   module_name = model_type_to_module_name(key)                                       │
│   539 │   │   if module_name not in self._modules:                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'gpt_bigcode'
```

Any idea on how to fix this issue? Had trouble authenticating my huggingface that is why I am trying to load a local model.

Can confirm that I am using transformers==4.28.1

Can you try uninstalling and re-installing `transformers` again the error indicates not finding the GPTBigCode architecture.

Since it's an issue with loading the model, to debug you can try running this and make sure the model loads
```python
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(YOUR_PATH)
```

Did you fix the issue?

closing as issue is inactive