Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate huge models like LLAMA-65b?
Root cause: Documentation Deficiency

I tried this on 8 A100 GPUs (40GB) and it OOMed. Any pointer on how to run inference with very big models?
```
python main.py --model hf-causal-experimental --model_args pretrained=huggyllama/llama-65b,use_accelerate=True --tasks wikitext --batch_size 1 --no_cache
```
[accelerate](https://huggingface.co/docs/accelerate/usage_guides/big_modeling) currently supports inference with large models but I'm not sure if this could be used under the hood.

Best,
Rui


I tried to run an inference job with large models (llama-30b). 
`python main.py --model hf-causal-experimental --model_args pretrained=huggyllama/llama-30b,use_accelerate=True,max_memory_per_gpu=40gb --tasks wikitext --batch_size 1 --no_cache --write_out --output_path sf_results/llama-30b/wikitext --device auto`

But I ran into this error. 
```
Running loglikelihood_rolling requests
  0%|                                                    | 0/62 [00:00<?, ?it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (8987 > 2048). Running this sequence through the model will result in indexing errors
python-BaseException
  0%|                                                    | 0/62 [02:08<?, ?it/s]
Traceback (most recent call last):
  File "/root/.pycharm_helpers/pydev/pydevd.py", line 1496, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/root/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "main.py", line 89, in <module>
    main()
  File "main.py", line 57, in main
    results = evaluator.simple_evaluate(
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/utils.py", line 242, in _wrapper
    return fn(*args, **kwargs)
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/evaluator.py", line 91, in simple_evaluate
    results = evaluate(
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/utils.py", line 242, in _wrapper
    return fn(*args, **kwargs)
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/evaluator.py", line 285, in evaluate
    resps = getattr(lm, reqtype)([req.args for req in reqs])
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/base.py", line 239, in loglikelihood_rolling
    string_nll = self._loglikelihood_tokens(
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/base.py", line 323, in _loglikelihood_tokens
    inp = torch.tensor(
RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: disk
```

I think it's because of the offloading to disk, so I set `offload_folder=None` and another error is raised.
```
python-BaseException
Traceback (most recent call last):
  File "/root/.pycharm_helpers/pydev/pydevd.py", line 1496, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/root/.pycharm_helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "main.py", line 89, in <module>
    main()
  File "main.py", line 57, in main
    results = evaluator.simple_evaluate(
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/utils.py", line 242, in _wrapper
    return fn(*args, **kwargs)
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/evaluator.py", line 69, in simple_evaluate
    lm = lm_eval.models.get_model(model).create_from_arg_string(
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/base.py", line 115, in create_from_arg_string
    return cls(**args, **args2)
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/models/huggingface.py", line 205, in __init__
    self.model = self._create_auto_model(
  File "/export/home/project/llm/lm-evaluation-harness/lm_eval/models/huggingface.py", line 261, in _create_auto_model
    model = self.AUTO_MODEL_CLASS.from_pretrained(
  File "/export/home/project/llm/transformers/src/transformers/models/auto/auto_factory.py", line 484, in from_pretrained
    return model_class.from_pretrained(
  File "/export/home/project/llm/transformers/src/transformers/modeling_utils.py", line 2937, in from_pretrained
    dispatch_model(model, **kwargs)
  File "/export/home/project/llm/accelerate/src/accelerate/big_modeling.py", line 366, in dispatch_model
    weights_map = OffloadedWeightsLoader(
  File "/export/home/project/llm/accelerate/src/accelerate/utils/offload.py", line 154, in __init__
    raise ValueError("Need either a `state_dict` or a `save_folder` containing offloaded weights.")
ValueError: Need either a `state_dict` or a `save_folder` containing offloaded weights.
```

FYI, this worked for me (use --device auto):
```
python main.py --model hf-causal-experimental --model_args pretrained=huggyllama/llama-65b,use_accelerate=True --tasks wikitext --batch_size 1 --no_cache --write_out --output_path sf_results/llama-65b/wikitext.json --device auto
```

For large model you should be using `auto` as your device. You can also pass a device map if you’re a super-user, but there’s almost no reason to do that if you just want to evaluate a LLM.