Repository: EleutherAI/lm-evaluation-harness
Issue: vllm mode has return "[A"
Root cause: Validation Gap

Hi. harness developers. I am using vllm mode for evaluation leaderboard benchmarks. 

This is my setting for evaluation in jupyter notebook

```

vllm_settings= ',dtype=auto,gpu_memory_utilization=0.8'

args = argparse.Namespace(
    model="vllm",  # 
    model_args="pretrained=/home/jovyan/data-vol-1/models/" 
                + 'LGAI-EXAONE__EXAONE-3.0-7.8B-Instruct'
                + vllm_settings, #
    tasks="leaderboard",  # 
    num_fewshot=None,
    batch_size="auto",
    max_batch_size=None,
    device="cuda",  #
    output_path=f'result/LGAI-EXAONE__EXAONE-3.0-7.8B-Instruct.json',
    limit=None,
    use_cache=None,
    cache_requests=None,
    check_integrity=False,
    write_out=False,
    log_samples=False,
    system_instruction=None,
    apply_chat_template=False,
    fewshot_as_multiturn=False,
    show_config=False,
    include_path=None,
    gen_kwargs=None,
    verbosity="INFO",
    wandb_args="",
    hf_hub_log_args="",
    predict_only=False,
    seed=[0, 1234, 1234, 1234],
    trust_remote_code=True,
    parallelize=True
)

# 
cli_evaluate(args)
```


Below is the log for


2024-10-21:10:42:03,900 WARNING  [task.py:332] [Task: leaderboard_musr_team_allocation] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-10-21:10:42:03,900 WARNING  [task.py:332] [Task: leaderboard_musr_team_allocation] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-10-21:10:42:03,901 WARNING  [task.py:332] [Task: leaderboard_musr_team_allocation] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-10-21:10:42:03,919 WARNING  [task.py:332] [Task: leaderboard_musr_object_placements] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-10-21:10:42:03,919 WARNING  [task.py:332] [Task: leaderboard_musr_object_placements] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-10-21:10:42:03,937 WARNING  [task.py:332] [Task: leaderboard_musr_murder_mysteries] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-10-21:10:42:03,938 WARNING  [task.py:332] [Task: leaderboard_musr_murder_mysteries] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-10-21:10:42:03,959 WARNING  [task.py:332] [Task: leaderboard_ifeval] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-10-21:10:42:03,959 WARNING  [task.py:332] [Task: leaderboard_ifeval] has_training_docs and has_validation_docs are False, using test_docs as fewshot_docs but this is not recommended.
2024-10-21:10:42:04,883 WARNING  [model.py:422] model.chat_template was called with the chat_template set to False or None. Therefore no chat template will be applied. Make sure this is an intended behavior.
2024-10-21:10:42:04,893 INFO     [task.py:422] Building contexts for leaderboard_mmlu_pro on rank 0...
100%|██████████| 12032/12032 [00:02<00:00, 4647.58it/s]
2024-10-21:10:42:08,200 INFO     [task.py:422] Building contexts for leaderboard_bbh_boolean_expressions on rank 0...
100%|██████████| 250/250 [00:00<00:00, 528.44it/s]
2024-10-21:10:42:08,679 INFO     [task.py:422] Building contexts for leaderboard_bbh_causal_judgement on rank 0...
100%|██████████| 187/187 [00:00<00:00, 538.46it/s]
2024-10-21:10:42:09,031 INFO     [task.py:422] Building contexts for leaderboard_bbh_date_understanding on rank 0...
100%|██████████| 250/250 [00:00<00:00, 535.24it/s]
2024-10-21:10:42:09,504 INFO     [task.py:422] Building contexts for leaderboard_bbh_disambiguation_qa on rank 0...
100%|██████████| 250/250 [00:00<00:00, 535.03it/s]
2024-10-21:10:42:09,977 INFO     [task.py:422] Building contexts for leaderboard_bbh_formal_fallacies on rank 0...
100%|██████████| 250/250 [00:00<00:00, 534.61it/s]
2024-10-21:10:42:10,451 INFO     [task.py:422] Building contexts for leaderboard_bbh_geometric_shapes on rank 0...
100%|██████████| 250/250 [00:00<00:00, 537.51it/s]
2024-10-21:10:42:10,923 INFO     [task.py:422] Building contexts for leaderboard_bbh_hyperbaton on rank 0...
100%|██████████| 250/250 [00:00<00:00, 543.99it/s]
2024-10-21:10:42:11,388 INFO     [task.py:422] Building contexts for leaderboard_bbh_logical_deduction_five_objects on rank 0...
100%|██████████| 250/250 [00:00<00:00, 532.50it/s]
2024-10-21:10:42:11,864 INFO     [task.py:422] Building contexts for leaderboard_bbh_logical_deduction_seven_objects on rank 0...
100%|██████████| 250/250 [00:00<00:00, 531.48it/s]
2024-10-21:10:42:12,341 INFO     [task.py:422] Building contexts for leaderboard_bbh_logical_deduction_three_objects on rank 0...
100%|██████████| 250/250 [00:00<00:00, 531.48it/s]
2024-10-21:10:42:12,817 INFO     [task.py:422] Building contexts for leaderboard_bbh_movie_recommendation on rank 0...
100%|██████████| 250/250 [00:00<00:00, 537.73it/s]
2024-10-21:10:42:13,288 INFO     [task.py:422] Building contexts for leaderboard_bbh_navigate on rank 0...
100%|██████████| 250/250 [00:00<00:00, 546.45it/s]
2024-10-21:10:42:13,751 INFO     [task.py:422] Building contexts for leaderboard_bbh_object_counting on rank 0...
100%|██████████| 250/250 [00:00<00:00, 523.34it/s]
2024-10-21:10:42:14,235 INFO     [task.py:422] Building contexts for leaderboard_bbh_penguins_in_a_table on rank 0...
100%|██████████| 146/146 [00:00<00:00, 544.57it/s]
2024-10-21:10:42:14,507 INFO     [task.py:422] Building contexts for leaderboard_bbh_reasoning_about_colored_objects on rank 0...
100%|██████████| 250/250 [00:00<00:00, 534.53it/s]
2024-10-21:10:42:14,981 INFO     [task.py:422] Building contexts for leaderboard_bbh_ruin_names on rank 0...
100%|██████████| 250/250 [00:00<00:00, 538.14it/s]
2024-10-21:10:42:15,451 INFO     [task.py:422] Building contexts for leaderboard_bbh_salient_translation_error_detection on rank 0...
100%|██████████| 250/250 [00:00<00:00, 533.15it/s]
2024-10-21:10:42:15,927 INFO     [task.py:422] Building contexts for leaderboard_bbh_snarks on rank 0...
100%|██████████| 178/178 [00:00<00:00, 539.38it/s]
2024-10-21:10:42:16,261 INFO     [task.py:422] Building contexts for leaderboard_bbh_sports_understanding on rank 0...
100%|██████████| 250/250 [00:00<00:00, 542.76it/s]
2024-10-21:10:42:16,727 INFO     [task.py:422] Building contexts for leaderboard_bbh_temporal_sequences on rank 0...
100%|██████████| 250/250 [00:00<00:00, 544.27it/s]
2024-10-21:10:42:17,193 INFO     [task.py:422] Building contexts for leaderboard_bbh_tracking_shuffled_objects_five_objects on rank 0...
100%|██████████| 250/250 [00:00<00:00, 540.94it/s]
2024-10-21:10:42:17,661 INFO     [task.py:422] Building contexts for leaderboard_bbh_tracking_shuffled_objects_seven_objects on rank 0...
100%|██████████| 250/250 [00:00<00:00, 536.05it/s]
2024-10-21:10:42:18,134 INFO     [task.py:422] Building contexts for leaderboard_bbh_tracking_shuffled_objects_three_objects on rank 0...
100%|██████████| 250/250 [00:00<00:00, 542.32it/s]
2024-10-21:10:42:18,601 INFO     [task.py:422] Building contexts for leaderboard_bbh_web_of_lies on rank 0...
100%|██████████| 250/250 [00:00<00:00, 547.28it/s]
2024-10-21:10:42:19,064 INFO     [task.py:422] Building contexts for leaderboard_gpqa_diamond on rank 0...
100%|██████████| 198/198 [00:00<00:00, 1859.83it/s]
2024-10-21:10:42:19,265 INFO     [task.py:422] Building contexts for leaderboard_gpqa_extended on rank 0...
100%|██████████| 546/546 [00:00<00:00, 1865.76it/s]
2024-10-21:10:42:19,803 INFO     [task.py:422] Building contexts for leaderboard_gpqa_main on rank 0...
100%|██████████| 448/448 [00:00<00:00, 1867.20it/s]
2024-10-21:10:42:20,249 INFO     [task.py:422] Building contexts for leaderboard_math_algebra_hard on rank 0...
100%|██████████| 307/307 [00:00<00:00, 530.82it/s]
2024-10-21:10:42:20,846 INFO     [task.py:422] Building contexts for leaderboard_math_counting_and_prob_hard on rank 0...
100%|██████████| 123/123 [00:00<00:00, 533.15it/s]
2024-10-21:10:42:21,090 INFO     [task.py:422] Building contexts for leaderboard_math_geometry_hard on rank 0...
100%|██████████| 132/132 [00:00<00:00, 529.71it/s]
2024-10-21:10:42:21,352 INFO     [task.py:422] Building contexts for leaderboard_math_intermediate_algebra_hard on rank 0...
100%|██████████| 280/280 [00:00<00:00, 537.31it/s]
2024-10-21:10:42:21,891 INFO     [task.py:422] Building contexts for leaderboard_math_num_theory_hard on rank 0...
100%|██████████| 154/154 [00:00<00:00, 536.92it/s]
2024-10-21:10:42:22,192 INFO     [task.py:422] Building contexts for leaderboard_math_prealgebra_hard on rank 0...
100%|██████████| 193/193 [00:00<00:00, 537.16it/s]
2024-10-21:10:42:22,566 INFO     [task.py:422] Building contexts for leaderboard_math_precalculus_hard on rank 0...
100%|██████████| 135/135 [00:00<00:00, 538.78it/s]
2024-10-21:10:42:22,829 INFO     [task.py:422] Building contexts for leaderboard_ifeval on rank 0...
100%|██████████| 541/541 [00:00<00:00, 146432.53it/s]
2024-10-21:10:42:22,886 INFO     [task.py:422] Building contexts for leaderboard_musr_murder_mysteries on rank 0...
100%|██████████| 250/250 [00:00<00:00, 3876.81it/s]
2024-10-21:10:42:22,962 INFO     [task.py:422] Building contexts for leaderboard_musr_object_placements on rank 0...
100%|██████████| 256/256 [00:00<00:00, 3773.71it/s]
2024-10-21:10:42:23,041 INFO     [task.py:422] Building contexts for leaderboard_musr_team_allocation on rank 0...
100%|██████████| 250/250 [00:00<00:00, 3798.49it/s]
2024-10-21:10:42:23,118 INFO     [evaluator.py:489] Running loglikelihood requests

[A
[A
[A
[A
[A
[A
[A
[A
[A
[A
[A
[A
...

-> This is repeated. But, this issues is not happen in hf mode. 

Hi! For loglikehood tasks, we are not generating anything but calculating the conditional logliklehood of a given answer string conditioned on some context (but vllm requires at least one output token, but we discard that). I guess you can check if the results are similar to HF, to confirm if the results are correct. (also optionally you can use a chat template for your instruct model with `apply_chat_template=True`)