Repository: EleutherAI/lm-evaluation-harness
Issue: [big-refactor] Error running with DDP
Root cause: Validation Gap

Run with 8xA100:
```bash
accelerate launch main.py \
    --model hf \
    --model_args pretrained=/path/to/model \
    --tasks super-glue-lm-eval-v1 \
    --batch_size 1
```
Error:
```
2023-06-30 17:05:51.289493: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-30 17:05:51.349179: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-30 17:05:52.075865: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
The following values were not passed to `accelerate launch` and had defaults used instead:
	`--num_processes` was set to a value of `8`
		More than one GPU was found, enabling multi-GPU training.
		If this was unintended please pass in `--num_processes=1`.
	`--num_machines` was set to a value of `1`
	`--mixed_precision` was set to a value of `'no'`
	`--dynamo_backend` was set to a value of `'no'`
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
2023-06-30:17:05:57,701 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-06-30:17:05:57,701 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-06-30:17:05:57,702 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-06-30:17:05:57,702 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-06-30:17:05:57,737 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-06-30:17:05:57,737 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-06-30:17:05:57,738 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-06-30:17:05:57,739 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-06-30:17:05:57,899 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-06-30:17:05:57,900 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-06-30:17:05:57,904 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-06-30:17:05:57,904 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-06-30:17:05:57,937 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-06-30:17:05:57,938 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-06-30:17:05:57,948 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-06-30:17:05:57,949 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-06-30 17:05:59.166840: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-30 17:05:59.204572: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-30 17:05:59.205053: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-30 17:05:59.258666: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-30 17:05:59.358467: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-30 17:05:59.367915: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-30 17:05:59.421066: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-30 17:05:59.449562: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-30:17:06:00,697 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'delimiter'
2023-06-30:17:06:00,722 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'delimiter'
2023-06-30:17:06:00,727 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'delimiter'
2023-06-30:17:06:00,818 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,820 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,831 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/copa/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,833 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,835 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,840 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,842 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,843 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'delimiter'
2023-06-30:17:06:00,844 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/copa/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,848 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/record/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,851 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 style' conflicts with existing registered task!
2023-06-30:17:06:00,852 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/copa/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,853 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-01.yaml
                                 Config will not be added to registry
                                 Error: task named 'MNLI crowdsource' conflicts with existing registered task!
2023-06-30:17:06:00,856 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,860 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/record/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,862 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 style' conflicts with existing registered task!
2023-06-30:17:06:00,865 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-01.yaml
                                 Config will not be added to registry
                                 Error: task named 'MNLI crowdsource' conflicts with existing registered task!
2023-06-30:17:06:00,867 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,867 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/record/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,867 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,869 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'delimiter'
2023-06-30:17:06:00,870 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 style' conflicts with existing registered task!
2023-06-30:17:06:00,871 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc.fixed/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 Style' conflicts with existing registered task!
2023-06-30:17:06:00,872 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-01.yaml
                                 Config will not be added to registry
                                 Error: task named 'MNLI crowdsource' conflicts with existing registered task!
2023-06-30:17:06:00,874 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,875 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'delimiter'
2023-06-30:17:06:00,876 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,879 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc.fixed/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 Style' conflicts with existing registered task!
2023-06-30:17:06:00,884 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,887 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc.fixed/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 Style' conflicts with existing registered task!
2023-06-30:17:06:00,951 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,953 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,962 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/copa/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,978 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/record/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,980 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 style' conflicts with existing registered task!
2023-06-30:17:06:00,981 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,982 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-01.yaml
                                 Config will not be added to registry
                                 Error: task named 'MNLI crowdsource' conflicts with existing registered task!
2023-06-30:17:06:00,983 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,985 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,985 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:00,987 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,992 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/copa/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,994 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,996 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/copa/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:00,997 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc.fixed/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 Style' conflicts with existing registered task!
2023-06-30:17:06:01,008 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/record/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,010 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 style' conflicts with existing registered task!
2023-06-30:17:06:01,011 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/record/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,012 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-01.yaml
                                 Config will not be added to registry
                                 Error: task named 'MNLI crowdsource' conflicts with existing registered task!
2023-06-30:17:06:01,014 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 style' conflicts with existing registered task!
2023-06-30:17:06:01,015 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:01,016 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-01.yaml
                                 Config will not be added to registry
                                 Error: task named 'MNLI crowdsource' conflicts with existing registered task!
2023-06-30:17:06:01,018 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:01,028 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,032 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc.fixed/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 Style' conflicts with existing registered task!
2023-06-30:17:06:01,035 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,038 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc.fixed/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 Style' conflicts with existing registered task!
2023-06-30:17:06:01,052 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'delimiter'
2023-06-30:17:06:01,080 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'delimiter'
2023-06-30:17:06:01,176 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:01,178 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,187 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/copa/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,199 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:01,201 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,204 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/record/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,206 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 style' conflicts with existing registered task!
2023-06-30:17:06:01,209 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-01.yaml
                                 Config will not be added to registry
                                 Error: task named 'MNLI crowdsource' conflicts with existing registered task!
2023-06-30:17:06:01,210 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/copa/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,211 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:01,221 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,223 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc.fixed/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 Style' conflicts with existing registered task!
2023-06-30:17:06:01,226 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/record/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,228 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 style' conflicts with existing registered task!
2023-06-30:17:06:01,231 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-01.yaml
                                 Config will not be added to registry
                                 Error: task named 'MNLI crowdsource' conflicts with existing registered task!
2023-06-30:17:06:01,233 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:06:01,243 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:06:01,246 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc.fixed/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 Style' conflicts with existing registered task!
2023-06-30:17:06:01,597 INFO     [main.py:78] Selected Tasks: ['super-glue-lm-eval-v1']
2023-06-30:17:06:01,597 INFO     [main.py:78] Selected Tasks: ['super-glue-lm-eval-v1']
2023-06-30:17:06:01,599 INFO     [main.py:78] Selected Tasks: ['super-glue-lm-eval-v1']
2023-06-30:17:06:01,666 INFO     [main.py:78] Selected Tasks: ['super-glue-lm-eval-v1']
2023-06-30:17:06:01,666 INFO     [main.py:78] Selected Tasks: ['super-glue-lm-eval-v1']
2023-06-30:17:06:01,677 INFO     [main.py:78] Selected Tasks: ['super-glue-lm-eval-v1']
2023-06-30:17:06:01,748 INFO     [huggingface.py:102] Passed device 'cuda', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
2023-06-30:17:06:01,756 INFO     [huggingface.py:102] Passed device 'cuda', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
2023-06-30:17:06:01,763 INFO     [huggingface.py:102] Passed device 'cuda', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
2023-06-30:17:06:01,830 INFO     [huggingface.py:102] Passed device 'cuda', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
2023-06-30:17:06:01,840 INFO     [huggingface.py:102] Passed device 'cuda', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
2023-06-30:17:06:01,851 INFO     [huggingface.py:102] Passed device 'cuda', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
2023-06-30:17:06:01,954 INFO     [main.py:78] Selected Tasks: ['super-glue-lm-eval-v1']
2023-06-30:17:06:01,954 INFO     [main.py:78] Selected Tasks: ['super-glue-lm-eval-v1']
2023-06-30:17:06:02,063 INFO     [huggingface.py:102] Passed device 'cuda', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
2023-06-30:17:06:02,068 INFO     [huggingface.py:102] Passed device 'cuda', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:36<00:00, 48.47s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:37<00:00, 48.68s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:37<00:00, 48.70s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:37<00:00, 48.51s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:37<00:00, 48.56s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:37<00:00, 48.83s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:37<00:00, 48.64s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [01:38<00:00, 49.32s/it]
2023-06-30:17:08:38,006 INFO     [distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 0
2023-06-30:17:08:38,019 INFO     [distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 2
2023-06-30:17:08:38,032 INFO     [distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 4
2023-06-30:17:08:38,046 INFO     [distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 1
2023-06-30:17:08:38,059 INFO     [distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 5
2023-06-30:17:08:38,074 INFO     [distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 7
2023-06-30:17:08:38,091 INFO     [distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 6
2023-06-30:17:08:38,100 INFO     [distributed_c10d.py:442] Added key: store_based_barrier_key:1 to store for rank: 3
2023-06-30:17:08:38,100 INFO     [distributed_c10d.py:476] Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-30:17:08:38,101 INFO     [distributed_c10d.py:476] Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-30:17:08:38,103 INFO     [distributed_c10d.py:476] Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-30:17:08:38,104 INFO     [distributed_c10d.py:476] Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-30:17:08:38,106 INFO     [distributed_c10d.py:476] Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-30:17:08:38,107 INFO     [distributed_c10d.py:476] Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-30:17:08:38,110 INFO     [distributed_c10d.py:476] Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-30:17:08:38,110 INFO     [distributed_c10d.py:476] Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-06-30:17:08:49,453 INFO     [huggingface.py:197] Using 8 devices with data parallelism
2023-06-30:17:08:49,453 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:49,453 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:49,453 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:49,453 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:49,453 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:49,453 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:49,453 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:49,454 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:53,743 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 868.45it/s]
2023-06-30:17:08:53,764 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 892.79it/s]
2023-06-30:17:08:53,866 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 856.56it/s]
2023-06-30:17:08:53,887 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
  0%|                                                                                                                                                            | 0/3 [00:00<?, ?it/s]2023-06-30:17:08:53,890 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 825.33it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 860.43it/s]
2023-06-30:17:08:54,070 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:54,089 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:54,161 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 783.20it/s]
2023-06-30:17:08:54,188 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 876.31it/s]
2023-06-30:17:08:54,197 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:54,220 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:54,225 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:54,244 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 838.69it/s]
2023-06-30:17:08:54,499 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:54,535 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:54,573 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:08:58,209 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
  0%|                                                                                                                                                            | 0/3 [00:00<?, ?it/s]2023-06-30:17:08:58,211 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1044.14it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1061.67it/s]
2023-06-30:17:08:58,289 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 812.74it/s]
2023-06-30:17:08:58,538 WARNING  [task.py:565] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-06-30:17:08:58,538 WARNING  [task.py:577] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:58,538 WARNING  [task.py:577] metric f1 is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:58,541 WARNING  [task.py:565] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-06-30:17:08:58,542 WARNING  [task.py:577] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:58,542 WARNING  [task.py:577] metric f1 is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:58,626 WARNING  [task.py:565] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-06-30:17:08:58,626 WARNING  [task.py:577] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:58,626 WARNING  [task.py:577] metric f1 is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:58,641 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1030.04it/s]
2023-06-30:17:08:58,717 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 929.45it/s]
2023-06-30:17:08:58,742 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 903.49it/s]
2023-06-30:17:08:58,746 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 871.39it/s]
2023-06-30:17:08:58,972 WARNING  [task.py:565] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-06-30:17:08:58,973 WARNING  [task.py:577] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:58,973 WARNING  [task.py:577] metric f1 is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:59,050 WARNING  [task.py:565] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-06-30:17:08:59,050 WARNING  [task.py:577] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:59,050 WARNING  [task.py:577] metric f1 is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:59,072 WARNING  [task.py:565] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-06-30:17:08:59,072 WARNING  [task.py:577] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:59,072 WARNING  [task.py:577] metric f1 is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:59,082 WARNING  [task.py:565] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-06-30:17:08:59,082 WARNING  [task.py:577] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:59,082 WARNING  [task.py:577] metric f1 is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:59,142 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 974.51it/s]
2023-06-30:17:08:59,475 WARNING  [task.py:565] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-06-30:17:08:59,475 WARNING  [task.py:577] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:08:59,475 WARNING  [task.py:577] metric f1 is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:09:01,199 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/cb/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
  0%|                                                                                                                                                            | 0/3 [00:00<?, ?it/s]2023-06-30:17:09:01,200 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/cb/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1108.43it/s]
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1013.77it/s]
2023-06-30:17:09:01,575 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/cb/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1126.39it/s]
2023-06-30:17:09:01,587 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/cb/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1057.39it/s]
2023-06-30:17:09:01,671 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/cb/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1131.15it/s]
2023-06-30:17:09:01,703 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/cb/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 910.42it/s]
2023-06-30:17:09:02,488 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/cb/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 965.17it/s]
2023-06-30:17:09:02,894 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/cb/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 984.89it/s]
2023-06-30:17:09:04,047 INFO     [evaluator.py:229] Running loglikelihood requests
2023-06-30:17:09:04,047 INFO     [evaluator.py:229] Running loglikelihood requests
2023-06-30:17:09:04,047 INFO     [evaluator.py:229] Running loglikelihood requests
2023-06-30:17:09:04,047 INFO     [evaluator.py:229] Running loglikelihood requests
2023-06-30:17:09:04,047 INFO     [evaluator.py:229] Running loglikelihood requests
2023-06-30:17:09:04,047 INFO     [evaluator.py:229] Running loglikelihood requests
2023-06-30:17:09:04,047 INFO     [evaluator.py:229] Running loglikelihood requests
2023-06-30:17:09:04,047 INFO     [evaluator.py:229] Running loglikelihood requests
 93%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████          | 781/839 [02:22<00:06,  8.40it/s]2023-06-30:17:11:27,055 INFO     [evaluator.py:229] Running greedy_until requests
 95%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▍      | 801/839 [02:24<00:03, 10.20it/s]2023-06-30:17:11:28,981 INFO     [evaluator.py:229] Running greedy_until requests
 96%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊     | 809/839 [02:25<00:02, 10.25it/s]2023-06-30:17:11:29,767 INFO     [evaluator.py:229] Running greedy_until requests
 98%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▎  | 823/839 [02:26<00:01, 10.30it/s]2023-06-30:17:11:31,286 INFO     [evaluator.py:229] Running greedy_until requests
 98%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌  | 825/839 [02:26<00:01, 10.26it/s]2023-06-30:17:11:31,418 INFO     [evaluator.py:229] Running greedy_until requests
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 839/839 [02:28<00:00,  5.66it/s]
2023-06-30:17:11:32,658 INFO     [evaluator.py:229] Running greedy_until requests
2023-06-30:17:11:32,821 INFO     [evaluator.py:229] Running greedy_until requests
  0%|                                                                                                                                                          | 0/409 [00:00<?, ?it/s]2023-06-30:17:11:33,209 INFO     [evaluator.py:229] Running greedy_until requests
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 409/409 [01:15<00:00,  5.45it/s]
Traceback (most recent call last):
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/main.py", line 131, in <module>
    main()
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/main.py", line 80, in main
    results = evaluator.simple_evaluate(
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/utils.py", line 318, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/evaluator.py", line 117, in simple_evaluate
    results = evaluate(
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/utils.py", line 318, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/evaluator.py", line 279, in evaluate
    target = task.doc_to_target(doc)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/api/task.py", line 713, in doc_to_target
    return utils.apply_template(doc_to_target, doc)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/utils.py", line 444, in apply_template
    return rtemplate.render(**doc)
  File "/usr/local/lib/miniconda3/envs/cloud-ai-lab/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/miniconda3/envs/cloud-ai-lab/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: list object has no element Undefined
```

Same with parallelize=True:
```
2023-06-30:17:17:38,301 INFO     [utils.py:145] Note: detected 128 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
2023-06-30:17:17:38,302 INFO     [utils.py:148] Note: NumExpr detected 128 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2023-06-30:17:17:38,302 INFO     [utils.py:160] NumExpr defaulting to 8 threads.
2023-06-30 17:17:38.909485: I tensorflow/core/util/port.cc:110] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-06-30 17:17:38.968750: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-06-30 17:17:39.708921: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2023-06-30:17:17:41,216 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/gsm8k/gsm8k.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'delimiter'
2023-06-30:17:17:41,334 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:17:41,336 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/cb/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:17:41,346 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/copa/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:17:41,364 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/record/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:17:41,366 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 style' conflicts with existing registered task!
2023-06-30:17:17:41,369 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-01.yaml
                                 Config will not be added to registry
                                 Error: task named 'MNLI crowdsource' conflicts with existing registered task!
2023-06-30:17:17:41,371 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/rte/promptsource-02.yaml
                                 Config will not be added to registry
                                 Error: task named 'based on the previous passage' conflicts with existing registered task!
2023-06-30:17:17:41,382 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc/t5-prompt.yaml
                                 Config will not be added to registry
                                 Error: TaskConfig.__init__() got an unexpected keyword argument 'reference'
2023-06-30:17:17:41,385 WARNING  [__init__.py:49] Failed to load config in
                                 /mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/tasks/super_glue/wsc.fixed/promptsource-00.yaml
                                 Config will not be added to registry
                                 Error: task named 'GPT-3 Style' conflicts with existing registered task!
2023-06-30:17:17:42,008 INFO     [main.py:78] Selected Tasks: ['super-glue-lm-eval-v1']
2023-06-30:17:17:42,104 INFO     [huggingface.py:102] Passed device 'cuda', but using `accelerate launch` or `parallelize=True`. This will be overridden when placing model.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:39<00:00, 19.65s/it]
2023-06-30:17:18:40,124 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:18:45,011 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 916.39it/s]
2023-06-30:17:18:45,341 WARNING  [task.py:536] Metric exact_match not found, Searching from https://huggingface.co/evaluate-metric
2023-06-30:17:18:49,935 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/boolq/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1015.57it/s]
2023-06-30:17:18:50,270 WARNING  [task.py:565] metric acc is defined, but aggregation is not. using default aggregation=mean
2023-06-30:17:18:50,270 WARNING  [task.py:577] metric acc is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:18:50,270 WARNING  [task.py:577] metric f1 is defined, but higher_is_better is not. using default higher_is_better=True
2023-06-30:17:18:53,289 WARNING  [builder.py:835] Found cached dataset super_glue (/root/.cache/huggingface/datasets/super_glue/cb/1.0.3/bb9675f958ebfee0d5d6dc5476fafe38c79123727a7258d515c450873dbdbbed)
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 1107.65it/s]
2023-06-30:17:19:00,311 INFO     [evaluator.py:229] Running loglikelihood requests
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6708/6708 [12:55<00:00,  8.65it/s]
2023-06-30:17:31:58,639 INFO     [evaluator.py:229] Running greedy_until requests
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3270/3270 [09:50<00:00,  5.54it/s]
Traceback (most recent call last):
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/main.py", line 131, in <module>
    main()
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/main.py", line 80, in main
    results = evaluator.simple_evaluate(
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/utils.py", line 318, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/evaluator.py", line 117, in simple_evaluate
    results = evaluate(
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/utils.py", line 318, in _wrapper
    return fn(*args, **kwargs)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/evaluator.py", line 279, in evaluate
    target = task.doc_to_target(doc)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/api/task.py", line 713, in doc_to_target
    return utils.apply_template(doc_to_target, doc)
  File "/mnt/home/linjunpeng/llm/test/lm-evaluation-harness/lm_eval/utils.py", line 444, in apply_template
    return rtemplate.render(**doc)
  File "/usr/local/lib/miniconda3/envs/cloud-ai-lab/lib/python3.10/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/miniconda3/envs/cloud-ai-lab/lib/python3.10/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: list object has no element Undefined
```

@lintangsutawika would you possibly have time to look into this? These tasks are the ones that are supposed to match v0.3.0 of the library in performance/implementation right?

Took a look, it was an issue in the yaml config. Fixed with #645

But `super-glue-lm-eval-v1` is still not complete. @SingL3 you are welcome to help if you are interested.