Repository: EleutherAI/lm-evaluation-harness
Issue: How to evaluate custom pretrained LLMs using multiple GPUs and huggingface accelerate?
Root cause: Documentation Deficiency

I am trying to evaluate the OPT model using the framework in the following way:

```python3
model = OPTForCausalLM.from_pretrained(model_name)
model = accelerator.prepare(model)

... # doing something with the model to prepare for evaluation

results = evaluator.simple_evaluate(
    model=model,
    no_cache=True,
    limit=limit,
    tasks=tasks,
    device=self.device,
)
```

However, I did not manage to run it on multiple GPUs, the code ends up running the evaluation only on one GPU. I cannot use the framework via `./lm_eval` script, I need to prepare the model before running the evaluation. Could you tell me what I might be missing here?

Hi @JungleTryne , 

there are two things I see here:
1. Are you using the `master` branch or v0.3.0 ? I see you're passing `no_cache=True` , which is from those versions. If so, that was prior to supporting data-parallel evaluation.
2. Are you wanting to split the model across GPUs, or to run data-parallel evaluation? if the former, you can pass a `device_map` to the `from_pretrained()` call. If the latter, you'd need to launch your script on multiple processes in parallel using `accelerate launch`, and prepare your model then. However, there might be some incompatibilities that arise when passing in an already-initialized model while using `accelerate launch`, which I'd be happy to assist with (we've typically assumed people needing this will instead subclass HFLM for this--is there any reason you are unable to shift the custom model preparation you need into `HFLM.__init__()`?


Hi @haileyschoelkopf ,

indeed, I was using an old version, now I switched to the master branch, thank you for the advice!

I want to split the model across multiple GPUs because it doesn't fit in one. Before evaluating the model I want to initialize it, work with it, and only then launch the benchmark, as I showed in my initial question. But now with the new version of lm-eval, I cannot simply do this:

```python
model = OPTForCausalLM.from_pretrained(model_name, ..., device_map="auto")
model = accelerator.prepare(model)

evaluator.simple_evaluate(
    model=model
    ...
)
```

because it gives me this error:

```
Traceback (most recent call last):
   ...
    _ = evaluator.simple_evaluate(
  File "/mnt/sdc/home/jungletryne/transformers/external/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/sdc/home/jungletryne/transformers/external/lm-evaluation-harness/lm_eval/evaluator.py", line 121, in simple_evaluate
    assert isinstance(model, lm_eval.api.model.LM)
AssertionError
``` 

How can I convert this model to lm_eval.api.model.LM, if I have an already pretrained one? And won't I lose the ability to perform inference on multiple GPUs? Because I checked the code of HFLM and I saw this

```python3
if not isinstance(pretrained, str):
    ...
    assert not parallelize, "`parallelize=True` is not compatible with passing pre-initialized model to `pretrained`"
```


regarding the assertionerror you can do this by first passing in

`lm = HFLM(pretrained=model)` and then passing that object to simple_evaluate!

> if not isinstance(pretrained, str):
    ...
    assert not parallelize, "`parallelize=True` is not compatible with passing pre-initialized model to `pretrained`"

regarding this part of HFLM: yes, we don’t allow one to pass `parallelize=True` in this case because when you send a pre-initialized model into HFLM, we won’t move that model onto a new device.

However, if you have already parallelized your model using `device_map`, and *then* pass that model into HFLM for initialization—this should be respected and allow for you to run across multiple GPUs.

Let me know if this makes sense or helps!

Yes, it helps! Thank you very much :)

Hi @haileyschoelkopf . I am trying to do `data-parallel` by passing an already initialized model. Can you help me with setting this up? How should I go about subclassing the `HFLM`?