Repository: EleutherAI/lm-evaluation-harness
Issue: How to test with multiple Gpus
Root cause: Validation Gap

How to test with multiple Gpus, here is my command but only one gpu will be called:
python main.py \
    --model hf-causal \
    --model_args pretrained=output_models/gpt2-large-news1\
    --tasks hellaswag \
    --num_fewshot 10 \
    --device cuda:0,1,2,3,4,5

Hi! Please check out our [refactored branch](https://github.com/EleutherAI/lm-evaluation-harness/tree/big-refactor) which has support for this functionality. It will be coming to main soon.

But when I tried that branch, I got the following error: TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until. This may seem like a problem with the code itself

This is the complete code error：
[2023-10-19 03:51:25,767] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
The following values were not passed to `accelerate launch` and had defaults used instead:
        `--num_processes` was set to a value of `6`
                More than one GPU was found, enabling multi-GPU training.
                If this was unintended please pass in `--num_processes=1`.
        `--num_machines` was set to a value of `1`
        `--mixed_precision` was set to a value of `'no'`
        `--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2023-10-19:03:51:29,086 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:03:51:29,150 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:03:51:29,158 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:03:51:29,158 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:03:51:29,192 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:03:51:29,194 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:03:51:29,659 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmpp3ds3s36
2023-10-19:03:51:29,659 INFO     [instantiator.py:76] Writing /tmp/tmpp3ds3s36/_remote_module_non_scriptable.py
2023-10-19:03:51:29,742 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmpsn1ftij1
2023-10-19:03:51:29,742 INFO     [instantiator.py:76] Writing /tmp/tmpsn1ftij1/_remote_module_non_scriptable.py
2023-10-19:03:51:29,761 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmpag9ssuxq
2023-10-19:03:51:29,761 INFO     [instantiator.py:76] Writing /tmp/tmpag9ssuxq/_remote_module_non_scriptable.py
2023-10-19:03:51:29,762 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmp5tddo5eb
2023-10-19:03:51:29,763 INFO     [instantiator.py:76] Writing /tmp/tmp5tddo5eb/_remote_module_non_scriptable.py
2023-10-19:03:51:29,790 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmpdytfbnf1
2023-10-19:03:51:29,790 INFO     [instantiator.py:76] Writing /tmp/tmpdytfbnf1/_remote_module_non_scriptable.py
2023-10-19:03:51:29,794 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmp12cdenca
2023-10-19:03:51:29,795 INFO     [instantiator.py:76] Writing /tmp/tmp12cdenca/_remote_module_non_scriptable.py
[2023-10-19 03:51:30,193] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 03:51:30,282] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 03:51:30,301] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 03:51:30,307] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 03:51:30,330] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 03:51:30,334] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
2023-10-19:03:51:36,026 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,037 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,217 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,228 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,256 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,267 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,358 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,369 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,465 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,476 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,490 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:36,501 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:03:51:38,737 INFO     [__main__.py:177] Selected Tasks: ['arc_challenge']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:03:51:38,780 INFO     [__main__.py:177] Selected Tasks: ['arc_challenge']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:03:51:38,850 INFO     [__main__.py:177] Selected Tasks: ['arc_challenge']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:03:51:38,867 INFO     [__main__.py:177] Selected Tasks: ['arc_challenge']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:03:51:38,868 INFO     [__main__.py:177] Selected Tasks: ['arc_challenge']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:03:51:38,876 INFO     [__main__.py:177] Selected Tasks: ['arc_challenge']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 244167) of binary: /data1/sunhaoyu/miniconda3/envs/lmflow/bin/python
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
    args.func(args)
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/commands/launch.py", line 977, in launch_command
    multi_gpu_launcher(args)
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
    distrib_run.run(args)
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
lm_eval FAILED
------------------------------------------------------------
Failures:
[1]:
  time      : 2023-10-19_03:51:42
  host      : localhost.localdomain
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 244168)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2023-10-19_03:51:42
  host      : localhost.localdomain
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 244169)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2023-10-19_03:51:42
  host      : localhost.localdomain
  rank      : 3 (local_rank: 3)
  exitcode  : 1 (pid: 244170)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2023-10-19_03:51:42
  host      : localhost.localdomain
  rank      : 4 (local_rank: 4)
  exitcode  : 1 (pid: 244171)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[5]:
  time      : 2023-10-19_03:51:42
  host      : localhost.localdomain
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 244172)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-10-19_03:51:42
  host      : localhost.localdomain
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 244167)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================

You should configure `accelerate` before using it