Repository: EleutherAI/lm-evaluation-harness
Issue: [Refactor] Invalid DDP on multi-nodes single-card environment
Root cause: Architectural Constraint

Hi, masters, recently I endeavored to run through mmlu in the big-refactor branch.  But I got some unexpected situations.
My environment is multi nodes and each node only includes one GPU card. Are there some omissions in such an environment? 
## Unexpected situations
Here I list some errors I met. (All tests were based on llama2-13b on mmlu dataset)
In the master branch, The overall inference time is about one hour.
<img width="387" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/28424280/d647cb62-4bf2-40f8-bba8-055aa66e6166">
But in big-refactor:
it may take more time than the master's. (Please ignore the total number here because the dataset I tested myself overlaps with the latest commit, dividing  2 is the actual time)
[3 nodes ]
<img width="231" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/28424280/b29ba1a5-4526-4683-8763-e20463b94874">
[5 nodes ]
<img width="399" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/28424280/58703584-6f1c-4af4-a620-f31ea55d6fbb">
Through further investigation, I found that [huggingface.py#L104](https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/lm_eval/models/huggingface.py#L104) can only return 1 in my environment, and it will not enter the [huggingface.py#L301](https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/lm_eval/models/huggingface.py#L301)  conditional statement.  Therefore in [task.py#L364](https://github.com/EleutherAI/lm-evaluation-harness/blob/big-refactor/lm_eval/api/task.py#L364) I got the parameter rank=0, world_size=1 (But the ideal value is rank=(0,..n), world_size=n )).
So among all my nodes, each node executed the full mmlu data set.
After I modified 'task.py#L364' like below. (and some other modifications). Seems each node can obtained correctly segment.
<img width="601" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/28424280/3bbde885-b4d3-4b7a-a665-cabb9b75828c">.
## Questions
1. I want to check if have you ever considered a multi-nodes single-card environment before. If not I can push a PR for this,  If have already considered it, could you let me know how to set the config to startup evaluation in my environment? (but I don't have other machines like single-node multi-card or multi-nodes multi-cards. So it may be a bit hard to test the compatibility for different environments.)
2. This may only be compatible with single-node multi-card situations:
<img width="871" alt="image" src="https://github.com/EleutherAI/lm-evaluation-harness/assets/28424280/e987de02-08e4-4f66-9bc5-5072ff08db51">
Will you consider adding more explanations of possible situations in the readme?
Finally, I test with the command(Can it serve as a supplement to readme?): 
```accelerate launch --num_machines $WORLD_SIZE --num_processes $WORLD_SIZE --main_process_ip $MASTER_ADDR --machine_rank $RANK --main_process_port $MASTER_PORT --multi_gpu -m lm_eval --model hf --model_args pretrained=$MODEL_PATH --tasks mmlu --batch_size auto:10 --num_fewshot 5```

Hi! We have not considered a multi-node environment historically and have assumed that either one writes a custom model integration with your distributed library (see https://github.com/bigscience-workshop/architecture-objective/blob/main/t5x/eval_harness.py , https://github.com/EleutherAI/gpt-neox/blob/main/eval_tasks/eval_adapter.py . ) We'll clarify this assumption in the README.

I think that for the multi-node case, it would be better to integrate data parallelism alongside faster inference libraries that also allow one to spin up APIs that can be queried locally, such as vLLM or TGI, in terms of speeding evaluation up.