Repository: EleutherAI/lm-evaluation-harness
Issue: compatible of parallelize=True and batch_size=auto
Root cause: Validation Gap

I got the following error when evaluate a 70b model on 4 gpus, with accelerate, parallelize, and batch_size=auto.

python=3.10
lm-eval Version: 0.4.0

```
accelerate launch --use_fsdp --no_python  lm_eval --model hf \
    --model_args pretrained=meta-llama/Llama-2-70b-hf,parallelize=True,dtype=auto \
    --tasks hellaswag --num_fewshot 10 \
    --batch_size auto
2023-12-30:17:48:00,526 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-12-30:17:48:33,142 INFO     [__main__.py:156] Verbosity set to INFO
2023-12-30:17:48:49,778 INFO     [__main__.py:229] Selected Tasks: ['hellaswag']
Downloading shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [00:04<00:00,  3.54it/s]
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 15/15 [02:22<00:00,  9.48s/it]
2023-12-30:17:51:41,611 WARNING  [evaluator.py:141] Overwriting default num_fewshot of hellaswag from None to 10
2023-12-30:17:51:41,615 INFO     [task.py:344] Building contexts for task on rank 0...
2023-12-30:17:53:59,991 INFO     [evaluator.py:314] Running loglikelihood requests
  0%|                                                                                                                                                                                       | 0/40168 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/lustre/home/haonan.li/miniconda3/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/__main__.py", line 231, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/evaluator.py", line 150, in simple_evaluate
    results = evaluate(
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/utils.py", line 415, in _wrapper
    return fn(*args, **kwargs)
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/evaluator.py", line 325, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/models/huggingface.py", line 759, in loglikelihood
    return self._loglikelihood_tokens(new_reqs)
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/models/huggingface.py", line 877, in _loglikelihood_tokens
    for chunk in chunks:
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/utils.py", line 822, in get_batched
    yield from batch
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/utils.py", line 918, in get_chunks
    if len(arr) == (fn(i, iter) if fn else n):
  File "/lustre/home/haonan.li/mygit/lm-evaluation-harness/lm_eval/models/huggingface.py", line 818, in _batch_scheduler
    sched = pos // int(len(n_reordered_requests) / self.batch_schedule)
TypeError: object of type 'generator' has no len()
  0%|                                                                                                                                                                                       | 0/40168 [00:01<?, ?it/s]
[2023-12-30 17:57:02,128] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: 1) local_rank: 0 (pid: 3615208) of binary: lm_eval
Traceback (most recent call last):
  File "/lustre/home/haonan.li/miniconda3/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/lustre/home/haonan.li/miniconda3/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
    args.func(args)
  File "/lustre/home/haonan.li/miniconda3/lib/python3.10/site-packages/accelerate/commands/launch.py", line 966, in launch_command
    multi_gpu_launcher(args)
  File "/lustre/home/haonan.li/miniconda3/lib/python3.10/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
    distrib_run.run(args)
  File "/lustre/home/haonan.li/miniconda3/lib/python3.10/site-packages/torch/distributed/run.py", line 797, in run
    elastic_launch(
  File "/lustre/home/haonan.li/miniconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/lustre/home/haonan.li/miniconda3/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
lm_eval FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-12-30_17:57:02
```

Hi, thanks for reporting this. 

Note that launching using `accelerate launch` is not supposed to be compatible with `parallelize=True`. Could you remove one of these and try again?

Rather than using FSDP through Accelerate, which can be slow or buggy during generation, we recommend using vLLM for heavy + multi-GPU inference loads if possible for your usecase--this should speed things up significantly compared to `parallelize=True` which only fully utilizes one GPU at once.

Thanks for your prompt response and suggestion.

Here is my observation:

There is no bug when I use `accelerate launch` and `parallelize=True`, with `batch_size=16`.
There is no bug when I use `parallelize=True`, with `batch_size=16`.
There is bug when I use `accelerate launch` and `parallelize=True`, with `batch_size=auto`.

vllm is really good and easy to use. But I met some bug there. 
So, do you have any suggestion on how to accelerate the evaluation without vllm?
If the answer is no, I will close this issue, and open another issue to discuss vllm.
Thanks.

Opening an issue about any problems you've had using vllm would be very helpful. 

What about when you use `parallelize=True, with batch_size=auto` and no accelerate launcher? I think there may be a bug here w.r.t. auto batching but a relatively simple one to fix. 

To your question, even without accelerate, there is the same bug when I set `parallelize=True`, with `batch_size=auto`.