Repository: EleutherAI/lm-evaluation-harness
Issue: Issue when Evaluating ChatGLM
Root cause: Validation Gap

I encountered a ValueError while trying to evaluate the chatglm model in the repository. I used the following command:

```
python main.py --model hf-causal --model_args pretrained=/mypath/chatglm-6b,trust_remote_code=True,max_length=1024 --tasks hellaswag --device cuda:0
```

The specific error message I received is as follows:
```
ValueError: Unrecognized configuration class <class 'transformers_modules.chatglm-6b.configuration_chatglm.ChatGLMConfig'> for this kind of AutoModel: AutoModelForCausalLM. Model type should be one of the following: [List of allowed model types].
```

What I do is clone the repository, install the necessary dependencies.

Python version: 3.9
Installed Transformers Version:  4.30.2

Hi! If you are encountering this on the `big-refactor` branch, this bug is fixed in #686 -- until that PR is shortly merged, please use its branch to evaluate ChatGLM.

If you are encountering this on the `master` branch, I can investigate why this error occurs, I am unsure why it would happen with the `hf-causal` model.

Hi! I used to work on `master` branch and I've switched to branch `big-refactor`.  I used the following command:
```
python main.py     --model hf     --model_args pretrained=/mypath/chatglm-6b,trust_remote_code=True    --tasks hellaswag     --device cuda:0
```
The error message I received is:
```
  File "/path/.cache/huggingface/modules/transformers_modules/chatglm-6b/modeling_chatglm.py", line 474, in forward
    context_layer, present, attention_probs = attention_fn(
  File "/path/llm2/.cache/huggingface/modules/transformers_modules/chatglm-6b/modeling_chatglm.py", line 306, in attention_fn
    attention_scores.masked_fill_(attention_mask, -10000.0)
RuntimeError: expected mask dtype to be Bool but got Long
```

I also test the `ChatGLM2` model, the error message I get is:
```
File "/path/lm-evaluation-harness-big-refactor/lm_eval/models/huggingface.py", line 228, in __init__
    self.tokenizer.pad_token_id = self.tokenizer.eos_token_id
AttributeError: can't set attribute
```



> Hi! If you are encountering this on the `big-refactor` branch, this bug is fixed in #686 -- until that PR is shortly merged, please use its branch to evaluate ChatGLM.
> 
> If you are encountering this on the `master` branch, I can investigate why this error occurs, I am unsure why it would happen with the `hf-causal` model.

I encountered the same problem on the master branch. My command is:
`
python main.py --model hf-causal --model_args pretrained=/mypath/chatglm-6b,trust_remote_code=True --tasks "arc_challenge" --num_fewshot 25 --device cuda
`

>  raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.chatglm-6b.configuration_chatglm.ChatGLMConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, CodeGenConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, ElectraConfig, ErnieConfig, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, LlamaConfig, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MvpConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, TransfoXLConfig, TrOCRConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig.

> 

same issue

same issue.

@jimmyzhang610 your new error reads as if the verison of transformers isn't up-to-date. What happens when you install the latest version of transformers?

> @jimmyzhang610 your new error reads as if the verison of transformers isn't up-to-date. What happens when you install the latest version of transformers?

I think the version of transformers should not matter. From the config.json file of the chatglm model, it can be observed that it is an AutoModelForSeq2SeqLM and does not support AutoModelForCausalLM.

`
{
  "_name_or_path": "THUDM/chatglm-6b",
  "architectures": [
    "ChatGLMModel"
  ],
  "auto_map": {
    "AutoConfig": "configuration_chatglm.ChatGLMConfig",
    "AutoModel": "modeling_chatglm.ChatGLMForConditionalGeneration",
    "AutoModelForSeq2SeqLM": "modeling_chatglm.ChatGLMForConditionalGeneration"
  },
  "bos_token_id": 130004,
  "eos_token_id": 130005,
`

Oh okay. In that case use `hf-seq2seq` on `master`