Repository: EleutherAI/lm-evaluation-harness
Issue: big-refactor branch, multi-GPU error：TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
Root cause: Validation Gap

When I tested with the big-refactor branch, I got the following error, which seemed to be a problem with the code itself. I can myself use the accelerate library for multi-GPU fine-tuning. 

This is the command I used：
accelerate launch -m lm_eval \
    --model hf \
    --model_args pretrained=gpt2-large \
    --tasks arc_easy 

The error is as follows：TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until

I will show the whole error in comment.

[2023-10-19 04:38:45,435] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
The following values were not passed to `accelerate launch` and had defaults used instead:
        `--num_processes` was set to a value of `6`
                More than one GPU was found, enabling multi-GPU training.
                If this was unintended please pass in `--num_processes=1`.
        `--num_machines` was set to a value of `1`
        `--mixed_precision` was set to a value of `'no'`
        `--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2023-10-19:04:38:51,356 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:04:38:51,433 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:04:38:51,472 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:04:38:51,575 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:04:38:51,638 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:04:38:51,774 INFO     [utils.py:148] Note: NumExpr detected 64 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-10-19:04:38:52,354 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmpkdob55yi
2023-10-19:04:38:52,354 INFO     [instantiator.py:76] Writing /tmp/tmpkdob55yi/_remote_module_non_scriptable.py
2023-10-19:04:38:52,372 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmpydiw43yw
2023-10-19:04:38:52,372 INFO     [instantiator.py:76] Writing /tmp/tmpydiw43yw/_remote_module_non_scriptable.py
2023-10-19:04:38:52,512 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmppq145p1v
2023-10-19:04:38:52,533 INFO     [instantiator.py:76] Writing /tmp/tmppq145p1v/_remote_module_non_scriptable.py
2023-10-19:04:38:52,537 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmp_y_ww9j8
2023-10-19:04:38:52,552 INFO     [instantiator.py:76] Writing /tmp/tmp_y_ww9j8/_remote_module_non_scriptable.py
2023-10-19:04:38:52,822 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmpdf61ijed
2023-10-19:04:38:52,822 INFO     [instantiator.py:76] Writing /tmp/tmpdf61ijed/_remote_module_non_scriptable.py
2023-10-19:04:38:52,949 INFO     [instantiator.py:21] Created a temporary directory at /tmp/tmp52833wgi
2023-10-19:04:38:52,958 INFO     [instantiator.py:76] Writing /tmp/tmp52833wgi/_remote_module_non_scriptable.py
[2023-10-19 04:38:53,214] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 04:38:53,364] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 04:38:53,378] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 04:38:53,415] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 04:38:53,737] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-10-19 04:38:53,783] [INFO] [real_accelerator.py:133:get_accelerator] Setting ds_accelerator to cuda (auto detect)
2023-10-19:04:39:03,802 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:03,818 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,131 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,151 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,548 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,581 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,716 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,724 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,728 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,736 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,760 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/t0_eval.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:04,772 WARNING  [__init__.py:144] /data1/sunhaoyu/lm-evaluation1/lm_eval/tasks/benchmarks/flan/flan_cot.yaml: No module named 'promptsource'. Config will not be added to registry.
2023-10-19:04:39:05,754 INFO     [__main__.py:177] Selected Tasks: ['arc_easy']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:04:39:06,515 INFO     [__main__.py:177] Selected Tasks: ['arc_easy']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:04:39:06,808 INFO     [__main__.py:177] Selected Tasks: ['arc_easy']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:04:39:06,849 INFO     [__main__.py:177] Selected Tasks: ['arc_easy']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:04:39:06,863 INFO     [__main__.py:177] Selected Tasks: ['arc_easy']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
2023-10-19:04:39:06,870 INFO     [__main__.py:177] Selected Tasks: ['arc_easy']
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 227, in <module>
    cli_evaluate()
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/__main__.py", line 179, in cli_evaluate
    results = evaluator.simple_evaluate(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/utils.py", line 343, in _wrapper
    return fn(*args, **kwargs)
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/evaluator.py", line 95, in simple_evaluate
    lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
  File "/data1/sunhaoyu/lm-evaluation1/lm_eval/api/model.py", line 140, in create_from_arg_string
    return cls(**args, **args2)
TypeError: Can't instantiate abstract class HFLM with abstract method greedy_until
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 262962 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 262959) of binary: /data1/sunhaoyu/miniconda3/envs/lmflow/bin/python
Traceback (most recent call last):
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/commands/accelerate_cli.py", line 45, in main
    args.func(args)
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/commands/launch.py", line 977, in launch_command
    multi_gpu_launcher(args)
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/accelerate/commands/launch.py", line 646, in multi_gpu_launcher
    distrib_run.run(args)
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/data1/sunhaoyu/miniconda3/envs/lmflow/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(

torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 

lm_eval FAILED

Failures:
[1]:
  time      : 2023-10-19_04:39:08
  host      : localhost.localdomain
  rank      : 1 (local_rank: 1)
  exitcode  : 1 (pid: 262960)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
  time      : 2023-10-19_04:39:08
  host      : localhost.localdomain
  rank      : 2 (local_rank: 2)
  exitcode  : 1 (pid: 262961)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[3]:
  time      : 2023-10-19_04:39:08
  host      : localhost.localdomain
  rank      : 4 (local_rank: 4)
  exitcode  : 1 (pid: 262963)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[4]:
  time      : 2023-10-19_04:39:08
  host      : localhost.localdomain
  rank      : 5 (local_rank: 5)
  exitcode  : 1 (pid: 262964)
  error_file: <N/A>

traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Root Cause (first observed failure):
[0]:
  time      : 2023-10-19_04:39:08
  host      : localhost.localdomain
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 262959)
  error_file: <N/A>

traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html

Hi! This is fixed in #929 , so sorry about the bug.