Repository: stanford-crfm/helm
Issue: helm-run fails with OpenAI `max_tokens or model output limit` error when running `openai/o4-mini`
Root cause: Interface Contract Mismatch

**Issue Description**

**What happened**
When invoking `helm-run` with the `openai/o4-mini` model, the pipeline repeatedly fails with a 400 error, indicating that the model output hit its token limit:

```
OpenAI error: Error code: 400 - {
  "error": {
    "message": "Could not finish the message because max_tokens or model output limit was reached. Please try again with higher max_tokens.",
    "type": "invalid_request_error"
  }
}
```

It retries several times before ultimately aborting.

**Expected behavior**
`helm-run` should either automatically increase the `max_tokens` parameter or surface a clear configuration option so that the user can adjust token limits without modifying deep library defaults.

**Steps to reproduce**

1. Create a run configuration file `run_entries_medhlm.conf`.
```
entries: [
  {description: "aci_bench", priority: 1},
  {description: "ehr_sql", priority: 1},
  {description: "head_qa:language=en,category=medicine", priority: 1},
  {description: "head_qa:language=en,category=nursing", priority: 1},
  {description: "head_qa:language=en,category=psychology", priority: 1},
  {description: "head_qa:language=en,category=chemistry", priority: 1},
  {description: "head_qa:language=en,category=pharmacology", priority: 1},
  {description: "head_qa:language=en,category=biology", priority: 1},
  {description: "med_dialog:subset=healthcaremagic", priority: 1},
  {description: "med_dialog:subset=icliniq", priority: 1},
]
```

2. Execute:

   ```bash
   helm-run \
     -c ./run_entries_medhlm.conf \
     --suite test-0.0.1 \
     --max-eval-instances 1 \
     --local-path ./config \
     --models-to-run openai/o4-mini
   ```

3. Observe the repeated 400 errors indicating token limits have been reached.

**Configuration**

* helm-run version: 0.5.5
* run entries file: `run_entries_medhlm.conf` (contains long prompt blocks)
* Model: `openai/o4-mini`
* Default `max_tokens` not explicitly set in CLI



@yifanmai @unnawut @percyliang @dlwh 

You can use the `increase_max_tokens` run expander for this, e.g.

```
entries: [
  {description: "aci_bench:increase_max_tokens=20000", priority: 1},
  {description: "ehr_sql:increase_max_tokens=20000", priority: 1},
  {description: "head_qa:language=en,category=medicine,increase_max_tokens=20000", priority: 1},
  {description: "head_qa:language=en,category=nursing,increase_max_tokens=20000", priority: 1},
  {description: "head_qa:language=en,category=psychology,increase_max_tokens=20000", priority: 1},
  {description: "head_qa:language=en,category=chemistry,increase_max_tokens=20000", priority: 1},
  {description: "head_qa:language=en,category=pharmacology,increase_max_tokens=20000", priority: 1},
  {description: "head_qa:language=en,category=biology,increase_max_tokens=20000", priority: 1},
  {description: "med_dialog:subset=healthcaremagic,increase_max_tokens=20000", priority: 1},
  {description: "med_dialog:subset=icliniq,increase_max_tokens=20000", priority: 1},
]
```

@yifanmai 

```bash                                                                                                                                                                                                             

Running med_dialog,subset=healthcaremagic:model=openai_o4-mini {
    scenario.get_instances {
      ensure_file_downloaded {
        Not downloading https://worksheets.codalab.org/rest/bundles/0x82f0c47f6d3e4462ae9ef8ea39eebe64/contents/blob/healthcaremagic/test.json because benchmark_output/scenarios/med_dialog/healthcaremagic/test.json already exists
      } [0.003s]
    } [0.382s]
    22642 instances, 0 train instances, 1/22642 eval instances
    DataPreprocessor.preprocess {
    } [0.0s]
    GenerationAdapter.adapt {
      1 instances, choosing 0/0 train instances, 1 eval instances
      Adapting with train_trial_index=0 {
        Sampled 0 examples for trial #0.
        Parallelizing computation on 1 items over 4 threads {
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 26.07it/s]
        } [0.039s]                                                                                                                                                               | 0/1 [00:00<?, ?it/s]
        Sample prompts {
          reference index = None, request_mode = None {
            Generate a one sentence summary of this patient-doctor conversation.
            
            Patient-Doctor: Patient: Can rabies be transferred through blood? I may have ran over something and when I came inside after cleaning it I forgot to wash my hands and began playing a video game then I realized I needed to wash my hands so I did then I came back and forgot to clean my mouse and I have a bad habit of biting my nails. So if the animal did have rabies could I have rabies now? Doctor: Hello thanx to contact healthcare magic. If I am your suggestion giving Doctor I suggest you that you should not be worry regarding rabies. Rabies only spreads through damaged nerves, which happens in the case of biting by dogs. But it wont spread by biting of nails. But if suppose when you biting the nail you got injured the fingerbed you might be infected with rabies. But I think you have not injured your finger during biting the nail so do not worry regarding that. If you have anything else to ask kindly contact me Dr Arun Tank, Infectious Disease specialist.
            Summary:
          } [0.0s]
        } [0.0s]
      } [0.039s]
      1 requests
    } [0.039s]
    Executor.execute {
      Parallelizing computation on 1 items over 4 threads {
                                                                                                                                                                                                              OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
        Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
        OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
        Request failed. Retrying (attempt #3) in 20 seconds... (See above for error details)
        OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
        Request failed. Retrying (attempt #4) in 40 seconds... (See above for error details)
        OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
        Request failed. Retrying (attempt #5) in 80 seconds... (See above for error details)
```

I believe this is correctly handled in the latest version of HELM (see [here](https://github.com/stanford-crfm/helm/blob/c154602e879ca44dc3dea841aaeff7826f59c5d3/src/helm/clients/openai_client.py#L226-L236)). Could you try installing the latest version from the main branch? Alternatively you can wait for the next PyPI release (which should be next week), and then do `pip install`.

> I believe this is correctly handled in the latest version of HELM (see [here](https://github.com/stanford-crfm/helm/blob/c154602e879ca44dc3dea841aaeff7826f59c5d3/src/helm/clients/openai_client.py#L226-L236)). Could you try installing the latest version from the main branch? Alternatively you can wait for the next PyPI release (which should be next week), and then do `pip install`.

This did not fix the issue for me.

```
                                                                                                                                               2025-08-01T13:58:21 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
2025-08-01T13:58:21 INFO             Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
2025-08-01T13:58:21 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
2025-08-01T13:58:21 INFO             Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
2025-08-01T13:58:21 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
2025-08-01T13:58:21 INFO             Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
2025-08-01T13:58:21 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
2025-08-01T13:58:21 INFO             Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
2025-08-01T13:58:31 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
```

```Python
model_engine: str = request.model_engine
        if OpenAIClientUtils.is_reasoning_model(model_engine):
            # Avoid error:
            # "Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead."  # noqa: E501
            # Note that openai>=1.45 is needed for this
            if raw_request["max_tokens"]:
                raw_request["max_completion_tokens"] = raw_request["max_tokens"]
                raw_request.pop("max_tokens")
            # Avoid error:
            # "Invalid type for 'stop': expected an unsupported value, but got null instead."
            if raw_request["stop"] is None:
                raw_request.pop("stop")
            else:                                               # I had to add these two lines to fix it. for o3.
                raw_request.pop("stop")
```

> > I believe this is correctly handled in the latest version of HELM (see [here](https://github.com/stanford-crfm/helm/blob/c154602e879ca44dc3dea841aaeff7826f59c5d3/src/helm/clients/openai_client.py#L226-L236)). Could you try installing the latest version from the main branch? Alternatively you can wait for the next PyPI release (which should be next week), and then do `pip install`.
> 
> This did not fix the issue for me.
> 
> ```
>                                                                                                                                                2025-08-01T13:58:21 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
> 2025-08-01T13:58:21 INFO             Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
> 2025-08-01T13:58:21 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
> 2025-08-01T13:58:21 INFO             Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
> 2025-08-01T13:58:21 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
> 2025-08-01T13:58:21 INFO             Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
> 2025-08-01T13:58:21 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
> 2025-08-01T13:58:21 INFO             Request failed. Retrying (attempt #2) in 10 seconds... (See above for error details)
> 2025-08-01T13:58:31 INFO             OpenAI error: Error code: 400 - {'error': {'message': "Unsupported parameter: 'stop' is not supported with this model.", 'type': 'invalid_request_error', 'param': 'stop', 'code': 'unsupported_parameter'}}
> ```

Hi @NoahBPeterson, Try this by adding 'stop=None' in run_entries

```
entries: [
  {description: "aci_bench:increase_max_tokens=20000,stop=None", priority: 1},
]

```

That did not work.

```
# HELM scenarios.

entries: [
  # NarrativeQA
  {description: "narrative_qa:model=text,output_format_instructions=narrative_qa,temperature=0,increase_max_tokens=10000,stop=None", priority: 1}

  # NaturalQuestions
  {description: "natural_qa:model=text,mode=openbook_longans,output_format_instructions=natural_qa,temperature=0,increase_max_tokens=10000,stop=None", priority: 1}
  {description: "natural_qa:model=text,mode=closedbook,output_format_instructions=natural_qa,temperature=0,increase_max_tokens=10000,stop=None", priority: 1}

  # OpenbookQA
  {description: "commonsense:model=text_code,dataset=openbookqa,method=multiple_choice_joint,output_format_instructions=openbookqa,temperature=0,increase_max_tokens=10000,stop=None", priority: 1}
]
```

```
 ~/Desktop/quant-bench/daily-bench  main !4  daily-bench run                                                            ✔  daily-bench  
Running HELM Lite benchmark from /Users/noahpeterson/Desktop/quant-bench/daily-bench/src/daily_bench/helm_lite
2025-08-01T14:43:41 INFO     helm_run {
2025-08-01T14:43:42 INFO       Reading tokenizer configs from /Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/config/tokenizer_configs.yaml...
2025-08-01T14:43:42 INFO       Reading model deployments from /Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/config/model_deployments.yaml...
2025-08-01T14:43:42 INFO       Read 4 run entries from run_entries_lite_20240424_instruct.conf
2025-08-01T14:43:43 INFO     } [1.805s]
Traceback (most recent call last):
  File "/Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/bin/helm-run", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/benchmark/run.py", line 377, in main
    return helm_run(args)
           ^^^^^^^^^^^^^^
  File "/Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/common/hierarchical_logger.py", line 140, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/benchmark/run.py", line 256, in helm_run
    run_specs = run_entries_to_run_specs(
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/benchmark/run.py", line 43, in run_entries_to_run_specs
    for run_spec in construct_run_specs(parse_object_spec(entry.description)):
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/benchmark/run_spec_factory.py", line 74, in construct_run_specs
    child_run_spec for parent_run_spec in run_specs for child_run_spec in expander.expand(parent_run_spec)
                                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/benchmark/run_expander.py", line 220, in expand
    raise Exception(f"Unknown value: {self.value}")
Exception: Unknown value: None
2025-08-01T14:43:47 INFO     summarize: summarize {
2025-08-01T14:43:47 INFO       Reading tokenizer configs from /Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/config/tokenizer_configs.yaml...
2025-08-01T14:43:47 INFO       Reading model deployments from /Users/noahpeterson/Desktop/quant-bench/daily-bench/.venv/lib/python3.12/site-packages/helm/config/model_deployments.yaml...
2025-08-01T14:43:48 INFO       Reading schema file schema_lite.yaml...
0it [00:00, ?it/s]
2025-08-01T14:43:48 INFO       Summarizer.check_metrics_defined {
2025-08-01T14:43:48 INFO       } [0.0s]
2025-08-01T14:43:48 INFO       Parallelizing computation on 0 items over 8 threads {
0it [00:00, ?it/s]
2025-08-01T14:43:48 INFO       } [0.0s]
2025-08-01T14:43:48 INFO       Writing 49582 characters to benchmark_output/runs/results-20250801_144341/schema.json
2025-08-01T14:43:48 INFO       Summarizer.write_executive_summary {
2025-08-01T14:43:48 INFO         Writing 113 characters to benchmark_output/runs/results-20250801_144341/summary.json
2025-08-01T14:43:48 INFO       } [0.0s]
2025-08-01T14:43:48 INFO       Writing 2 characters to benchmark_output/runs/results-20250801_144341/runs.json
2025-08-01T14:43:48 INFO       Writing 2 characters to benchmark_output/runs/results-20250801_144341/run_specs.json
2025-08-01T14:43:48 INFO       Writing 2 characters to benchmark_output/runs/results-20250801_144341/runs_to_run_suites.json
2025-08-01T14:43:48 INFO       Writing 10442 characters to benchmark_output/runs/results-20250801_144341/groups.json
2025-08-01T14:43:48 INFO       Writing 5038 characters to benchmark_output/runs/results-20250801_144341/groups_metadata.json
2025-08-01T14:43:48 INFO       Summarizer.write_cost_report {
2025-08-01T14:43:48 INFO         Writing 2 characters to benchmark_output/runs/results-20250801_144341/costs.json
2025-08-01T14:43:48 INFO       } [0.0s]
2025-08-01T14:43:48 INFO       Symlinking benchmark_output/runs/results-20250801_144341 to latest.
2025-08-01T14:43:48 INFO       Done.
2025-08-01T14:43:48 INFO     } [0.93s]
 ~/De/quant-bench/daily-bench  main !4                                                                             ✔  8s  daily-bench  
```