Repository: EleutherAI/lm-evaluation-harness
Issue: `AnthropicChat` fails when "until" is not provided explicitly in "generation_kwargs" or contains whitespace-only options
Root cause: Validation Gap

While adding a GLUE-style benchmark for the Belarusian language, I noticed unexpected behavior. Below are details and example based on the `nq_open` task. I can suggest a fix and implement it, but first decided to align on that by reporting a bug. 

## Description
The `AnthropicChat` is failing when the "until" parameter is not explicitly provided in the "generation_kwargs" or contains whitespace-only option. This leads to an API request failure due to an invalid stop sequence value.

1. "generation_kwargs" should be provided in the config.
2. By default, "until" is set to "\n\n" ([code link](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/api/task.py#L112)).
3. When the payload for the Anthropic API is created ([code link](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/anthropic_llms.py#L324)), "gen_kwargs" already has the "until" key in the dict. This prevents the fallback option ["\n\nHuman:"] from being used.
4. The Anthropic API requires a non-whitespace sequences for "stop_sequences", causing the request to fail when at least one is present.

## Steps to reproduce
1. Code version is currently the latest version of the code from main (commit `9b052fdc`)
2. Use the [nq_open](https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/nq_open/nq_open.yaml) configuration with (contains whitespace-only option) or without the "until" section (set to "\n\n"):
```
task: nq_open
dataset_path: nq_open
output_type: generate_until
training_split: train
validation_split: validation
description: "Answer these questions:\n\n"
doc_to_text: "Q: {{question}}?\nA:"
doc_to_target: "{{answer}}" # TODO: should be multi-target
fewshot_delimiter: "\n"
generation_kwargs:
  do_sample: false
  temperature: 0.0
filter_list:
  - name: remove_whitespace
    filter:
      - function: remove_whitespace
      - function: take_first
target_delimiter: " "
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true
    ignore_case: true
    ignore_punctuation: true
    regexes_to_ignore:
    - "\\b(?:The |the |An |A |The |a |an )"
metadata:
  version: 4.0
``` 

3. Run the evaluation using this configuration.
```
ANTHROPIC_API_KEY="{your_api_key}" lm_eval \
  --model="anthropic-chat-completions" \
  --model_args="model=claude-3-5-sonnet-20240620" \
  --tasks="nq_open" \
  --apply_chat_template
```
4. The evaluation fails with the following error message:
```
API request failed with error message: {"type":"error","error":{"type":"invalid_request_error","message":"stop_sequences: each stop sequence must contain non-whitespace"}}. Retrying...
```

## Workaround
The bug can be easily fixed by excluding all whitespace-only "until" options and explicitly adding non-whitespace ones to override the default "\n\n" value.


## Traceback
```
2024-10-20:17:04:34,449 INFO     [__main__.py:279] Verbosity set to INFO
2024-10-20:17:04:37,942 INFO     [__main__.py:376] Selected Tasks: ['nq_open']
2024-10-20:17:04:37,945 INFO     [evaluator.py:164] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
2024-10-20:17:04:37,945 INFO     [evaluator.py:201] Initializing anthropic-chat-completions model, with arguments: {'model': 'claude-3-5-sonnet-20240620'}
2024-10-20:17:04:37,945 INFO     [api_models.py:111] Using max length 2048 - 1
2024-10-20:17:04:37,945 INFO     [api_models.py:114] Concurrent requests are disabled. To enable concurrent requests, set `num_concurrent` > 1.
2024-10-20:17:04:37,945 INFO     [api_models.py:124] Using tokenizer None
2024-10-20:17:04:37,945 WARNING  [anthropic_llms.py:287] Chat completions does not support batching. Defaulting to batch size 1.
2024-10-20:17:04:37,945 WARNING  [anthropic_llms.py:292] Using Anthropic Version: 2023-06-01. Confirm the current version here: https://docs.anthropic.com/en/api/versioning
2024-10-20:17:04:44,347 INFO     [task.py:415] Building contexts for nq_open on rank 0...
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3610/3610 [00:00<00:00, 4208.38it/s]
2024-10-20:17:04:45,237 INFO     [evaluator.py:489] Running generate_until requests
Requesting API:   0%|                                                                                                           | 0/3610 [00:00<?, ?it/s]2024-10-20:17:04:45,445 WARNING  [api_models.py:347] API request failed with error message: {"type":"error","error":{"type":"invalid_request_error","message":"stop_sequences: each stop sequence must contain non-whitespace"}}. Retrying...
2024-10-20:17:04:46,666 WARNING  [api_models.py:347] API request failed with error message: {"type":"error","error":{"type":"invalid_request_error","message":"stop_sequences: each stop sequence must contain non-whitespace"}}. Retrying...
2024-10-20:17:04:47,884 WARNING  [api_models.py:347] API request failed with error message: {"type":"error","error":{"type":"invalid_request_error","message":"stop_sequences: each stop sequence must contain non-whitespace"}}. Retrying...
Traceback (most recent call last):
  File "{path_to_code}/lm-evaluation-harness/.venv/bin/lm_eval", line 8, in <module>
    sys.exit(cli_evaluate())
             ^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/__main__.py", line 382, in cli_evaluate
    results = evaluator.simple_evaluate(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/evaluator.py", line 301, in simple_evaluate
    results = evaluate(
              ^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/utils.py", line 397, in _wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/evaluator.py", line 500, in evaluate
    resps = getattr(lm, reqtype)(cloned_reqs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/models/api_models.py", line 569, in generate_until
    outputs = retry(
              ^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 418, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 185, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "{path_to_python}/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "{path_to_python}/Python.framework/Versions/3.12/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "{path_to_code}/lm-evaluation-harness/lm_eval/models/api_models.py", line 350, in model_call
    response.raise_for_status()
  File "{path_to_code}/lm-evaluation-harness/.venv/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://api.anthropic.com/v1/messages
Requesting API:   0%|                                                                                                           | 0/3610 [00:02<?, ?it/s]
```

Hi! Thanks for reporting this, and your solution seems very reasonable. If you have the time, a PR will be greatly appreciated. Otherwise, I'll try implementing it. :)

@baberabb I'll do that and submit a PR 

is there any plans to support this? i am running into the same problem when using `anthropic` engine.