Repository: EleutherAI/lm-evaluation-harness
Issue: Bus Error with Multiple GPUs on GSM8K Task
Root cause: Environment Incompatibility

I hope this message finds you well. 

I have successfully executed the evaluation on a single GPU without any hitches. However, when attempting to scale the process to utilize multiple GPUs, I am consistently faced with a bus error. The specific error message I receive is 

"Caught signal 7 (Bus error: nonexistent physical address)"

This issue arises only when I exceed the use of one GPU. The error seems to suggest a memory addressing problem that occurs during multi-GPU execution. It's important to note that this behavior is reproducible and occurs every time I attempt to use more than one GPU for the task.

Here are the details of my several failure setups:

setup 1:
accelerate launch --num_processes 2 --num_machines 1 --multi_gpu --gpu_ids 0,1 --no_python lm-eval \
        --model hf \
        --model_args pretrained='/models/model/Mistral-7B-v0.1/',dtype=bfloat16 \
        --tasks gsm8k \
        --num_fewshot 5 \
        --batch_size 2 \
        --write_out  \
        --log_samples \
        --show_config

setup 2:
accelerate launch --num_processes 2 --num_machines 1 --multi_gpu --gpu_ids 0,1  --main_process_port 6000  --no_python lm-eval \
        --model hf \
        --model_args pretrained='/models/model/Mistral-7B-v0.1/',dtype=bfloat16 \
        --tasks gsm8k \
        --num_fewshot 5 \
        --batch_size 2 \
        --write_out  \
        --log_samples \
        --show_config

setup 3:
accelerate launch --num_processes 2 --num_machines 1 --multi_gpu --gpu_ids 0,1 --main_process_ip 127.0.0.1  --main_process_port 6000  --no_python lm-eval \
        --model hf \
        --model_args pretrained='/models/model/Mistral-7B-v0.1/',dtype=bfloat16 \
        --tasks gsm8k \
        --num_fewshot 5 \
        --batch_size 2 \
        --write_out  \
        --log_samples \
        --show_config

==== backtrace (tid:   4366) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x000000000008def3 ncclGroupEnd()  ???:0
 2 0x000000000007cf07 ncclGroupEnd()  ???:0
 3 0x000000000007f24b ncclGroupEnd()  ???:0
 4 0x0000000000094b43 pthread_condattr_setpshared()  ???:0
 5 0x0000000000126a00 __xmknodat()  ???:0
=================================
==== backtrace (tid:   4367) ====
 0 0x0000000000042520 __sigaction()  ???:0
 1 0x000000000008def3 ncclGroupEnd()  ???:0
 2 0x000000000007cf07 ncclGroupEnd()  ???:0
 3 0x000000000007f24b ncclGroupEnd()  ???:0
 4 0x0000000000094b43 pthread_condattr_setpshared()  ???:0
 5 0x0000000000126a00 __xmknodat()  ???:0
=================================

[2023-12-13 18:47:19,575] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -7) local_rank: 0 (pid: 4336) of binary: lm-eval
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1008, in launch_command
    multi_gpu_launcher(args)
  File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 666, in multi_gpu_launcher
    distrib_run.run(args)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/run.py", line 788, in run
    elastic_launch(
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/launcher/api.py", line 264, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

Hi! Is there any more of the stacktrace that can be posted?

It looks like this may have to do with the size of shared mem : https://github.com/huggingface/transformers-bloom-inference/issues/16 https://github.com/THUDM/GLM-130B/issues/115

> Hi! Is there any more of the stacktrace that can be posted?
> 
> It looks like this may have to do with the size of shared mem : [huggingface/transformers-bloom-inference#16](https://github.com/huggingface/transformers-bloom-inference/issues/16) [THUDM/GLM-130B#115](https://github.com/THUDM/GLM-130B/issues/115)

Hello,

Thank you for your suggestion. After increasing the shared memory size to 48GB, the GSM8K tests are now running smoothly on 8 GPUs.

Best regards,