Repository: stanford-crfm/helm
Issue: Using llama2 with Multi GPU
Root cause: Unimplemented Feature Gap

Hello,
I am interested in using llama2 with multi-GPU. I understand that current support for this configuration within Helm is unavailable( #1683 ), and I was wondering if I could get some guidance or advice on how to proceed.

### What I did
- Upgraded transformers to version 4.31.0
- modify huggingface_client.py
  - Added device_map="auto" to AutoModelForCausalLM.from_pretrained
  - Using model_config from "meta-llama/Llama-2-70b-hf"
  - [my huggingface_client.py](https://gist.github.com/takitsuba/48db8e83cf166e9ef0b0685f21f99f7e)
- num-threads is 1


I am currently facing the following issues:

### When using two GPUs
- Some entries were successful, while others failed. I encountered Out Of Memory (OOM) error.
  - success:   {description: "mmlu:subject=abstract_algebra,model=meta-llama/Llama-2-70b-hf", priority: 1, groups: ["mmlu"]}
  - failed: {description: "mmlu:subject=high_school_us_history,model=meta-llama/Llama-2-70b-hf", priority: 1, groups: ["mmlu"]}
- Error message
```
HuggingFace error: CUDA out of memory. Tried to allocate 1.44 GiB (GPU 0; 79.35 GiB total capacity; 76.50 GiB already allocated; 134.44 MiB free; 77.93 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
        Request failed. Retrying (attempt #2) in 63 seconds... (See above for error details)
...
```

### When using four GPUs
- HuggingFace error: CUDA error: device-side assert triggered
- Error message
```
(...)
../aten/src/ATen/native/cuda/IndexKernel.cu:91: operator(): block: [377,0,0], thread: [95,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
        HuggingFace error: CUDA error: device-side assert triggered
        Request failed. Retrying (attempt #2) in 4 seconds... (See above for error details)
 (...)
```
- When using two GPUs (A100), the memory usage is 80GB x 2 = 160GB. However, even when increasing to four GPUs, the memory usage is about 75GB x 4 = 300GB.


Any advice or suggestions would be greatly appreciated.
I would also appreciate it if you could let me know in case a substantial change is needed and the issue seems difficult to address.
Thank you for your time.

Sorry, it seems that the slow communication speed between GPUs (without NVLink) was the cause of this issue.
I will try again after improving the speed. I apologize for the inconvenience.

Hi @takitsuba, in general, I don't think HELM has been tested with multi-GPU yet, so you would be the first.

I think the general approach would be to get `meta-llama/Llama-2-70b-hf` working in pure Hugging Face without HELM first, and then trying to replicate the setup and configuration as closely as possible in HELM. Are you already able to get the model working on multi-GPU on just Hugging Face? If so, could you share some details on your setup?

Sorry for the late reply!
In my environment, I become to be able to run HELM with multiple GPUs by changing `huggingface_client.py`.
The above problems were caused by the late communication speed between GPUs.

So I will close this issue. (Sorry for the trouble)