Repository: EleutherAI/lm-evaluation-harness
Issue: Evaluate encoder-decoder-models
Root cause: Unimplemented Feature Gap

The command to evaluate a encoder-decoder model, which isn't loaded as AutoModel (see https://github.com/huggingface/transformers/issues/28721):
```
!lm_eval --model hf \
    --model_args pretrained=Bachstelze/instructionRoberta-base \
    --tasks hellaswag \
    --device cuda:0 \
    --batch_size 2
```
This command throws the following `NotImplementedError: Cannot copy out of meta tensor; no data!`
```
2024-05-14 10:35:09.169205: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-05-14 10:35:09.169261: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-05-14 10:35:09.170625: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-05-14 10:35:10.347152: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Downloading builder script: 100% 5.67k/5.67k [00:00<00:00, 20.1MB/s]
2024-05-14:10:35:15,965 INFO     [__main__.py:254] Verbosity set to INFO
2024-05-14:10:35:22,625 INFO     [__main__.py:341] Selected Tasks: ['hellaswag']
2024-05-14:10:35:22,631 INFO     [evaluator.py:141] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234
2024-05-14:10:35:22,631 INFO     [evaluator.py:178] Initializing hf model, with arguments: {'pretrained': 'Bachstelze/instructionRoberta-base'}
2024-05-14:10:35:22,714 INFO     [huggingface.py:165] Using device 'cuda:0'
/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
config.json: 100% 4.79k/4.79k [00:00<00:00, 18.9MB/s]
model.safetensors: 100% 615M/615M [00:03<00:00, 162MB/s]
The following encoder weights were not tied to the decoder ['roberta/pooler']
The following encoder weights were not tied to the decoder ['roberta/pooler']
The following encoder weights were not tied to the decoder ['roberta/pooler']
The following encoder weights were not tied to the decoder ['roberta/pooler']
Some weights of EncoderDecoderModel were not initialized from the model checkpoint at Bachstelze/instructionRoberta-base and are newly initialized: ['decoder.lm_head.decoder.bias', 'decoder.lm_head.decoder.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
The following encoder weights were not tied to the decoder ['roberta/pooler']
generation_config.json: 100% 132/132 [00:00<00:00, 877kB/s]
Traceback (most recent call last):
  File "/usr/local/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/content/lm-evaluation-harness/lm_eval/__main__.py", line 347, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/content/lm-evaluation-harness/lm_eval/utils.py", line 321, in _wrapper
    return fn(*args, **kwargs)
  File "/content/lm-evaluation-harness/lm_eval/evaluator.py", line 181, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/content/lm-evaluation-harness/lm_eval/api/model.py", line 134, in create_from_arg_string
    return cls(**args, **args2)
  File "/content/lm-evaluation-harness/lm_eval/models/huggingface.py", line 204, in __init__
    self._create_model(
  File "/content/lm-evaluation-harness/lm_eval/models/huggingface.py", line 547, in _create_model
    self._model = self.AUTO_MODEL_CLASS.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py", line 390, in from_pretrained
    return super().from_pretrained(pretrained_model_name_or_path, *model_args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3735, in from_pretrained
    dispatch_model(model, **device_map_kwargs)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/big_modeling.py", line 488, in dispatch_model
    model.to(device)
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2692, in to
    return super().to(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1152, in to
    return self._apply(convert)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 802, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 802, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 802, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 825, in _apply
    param_applied = fn(param)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1150, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!
```
setup:
- `transformers` version: 4.40.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.3 (gpu)
- Jax version: 0.4.26
- JaxLib version: 0.4.26
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Hi! 

To evaluate your model you could extend HFLM so that its _create_model() method loads an EncoderDecoderModel of the sort you want. It looks like as is, the model is being attempted to be loaded as a decoder-only model, and doesn't have an LM head that it can load.

We are unlikely to support encoder-only models or BertForGeneration-type models natively. We do however support T5 models at the moment.