zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Feature Request: Add tools and tool_choice config to Azure Openai provider
Root cause: Unimplemented Feature Gap

Description: https://github.com/promptfoo/promptfoo/blob/main/src/providers/azureopenai.ts#L38 Currently the Azure open ai provider only allow functions config in the provider config. Our team is trying to test using the tools and tool_choice parameters. 

The following 
```
prompts:
  - "Be helpful and tell me the past info in {{city}}."

providers:
  - id: azureopenai:chat:gpt4turbo
    config:
      apiHost: 
      apiKey: 
      temperature: 0.9
      tools: [
        {
        "type": "function",
          "function": {
            "name": "get_current_info",
            "description": "Get the current info in a given location",
            "parameters": {
              "type": "object",
                "properties": {
                  "location": {
                    "type": "string",
                      "description": "The city and state, e.g. San Francisco, CA"
                    },
                    "unit": {
                      "type": "string",
                      "enum": ["celsius", "fahrenheit"]
                    }
                  },
              "required": ["location"]
            }
          }
        }
      ]
      tool_choice: 'auto'

tests:
   - vars:
        city: Boston
     assert:
        - type: is-json
        - type: is-valid-openai-tools-call
        - type: javascript
          value: output[0].function.name === 'get_past_info'
        - type: javascript
          value: JSON.parse(output[0].function.arguments).location === 'Boston, MA'

   - vars:
        city: New York

```

is sending requests without the tools and tool_choice parameters:
```
npx promptfoo@latest eval -c tool_test.yaml --verbose
Calling Azure OpenAI API: {"model":"gpt4turbo","messages":[{"role":"user","content":"Be helpful and tell me the past info in Boston."}],"max_tokens":1024,"temperature":0.9,"top_p":1,"presence_penalty":0,"frequency_penalty":0} 
```

#553 adds support for Azure OpenAI `tools` and `tool_choice`.  It will be included in the next published release :)

Oops, totally missed your PR!  Sorry about that, will merge yours instead.

Your change is published in 0.47.0!
Repository: promptfoo/promptfoo
Issue: Some assertions in Gemini evaluation fail with system-only prompts
Root cause: Interface Contract Mismatch

**Describe the bug**

When using Google Gemini Test providers (both Google AI Studio and Vertex AI), specific evaluation assertions fail with a 400 Bad Request error.

The affected assertions are those that use LLM grading with system-only prompts: `model-graded-closedqa` and `select-best`. These assertions fail because Gemini API does not support executing prompts that contain only system messages.


**To Reproduce**

1. Create a configuration file with a Gemini provider and the affected assertion types:

```yaml
# file: gemini-reproduce.yaml
providers:
- id: vertex:gemini-2.0-flash
- id: vertex:gemini-2.5-flash-lite

defaultTest:
  options:
    provider:
      id: vertex:gemini-2.5-pro # this issue also happens in google:* providers

prompts:
- Please explain the good points about dogs.

tests:
- assert:
  - type: select-best
    value: Explains multiple positive aspects of dogs as companions and pets.
  - type: model-graded-closedqa
    value: Focuses on dogs without discussing cats.
```

2. Run the evaluation:
```bash
promptfoo eval -c gemini-reproduce.yaml
```

3. Observe the 400 Bad Request error:
```
API call error: Status INVALID_ARGUMENT, Code 400, Message:
Request contains an invalid argument.
```

**Expected behavior**

The evaluation assertions should execute successfully and return PASS/FAIL results, like they do with other providers (OpenAI, Anthropic, etc.).


**Screenshots**

<img width="1917" height="754" alt="Image" src="https://github.com/user-attachments/assets/898919b1-2bf1-45a6-8115-1655e0b076c5" />


**System information:**


- Promptfoo version: 0.118.3
- Providers: Both `vertex:*` and `google:*` Gemini providers


**Additional context**

Affected Assertions are `model-graded-closedqa` and `select-best`, those prompts are system-only prompts.

I checked almost all LLM Assertions, and it appears these two are the only implementations with system prompts only.

This issue originates from Gemini models, so it occurs with both `vertex:*` and `google:*` providers.

**Proposed Solution**

When system-only prompts are detected, the Gemini provider should convert them to user messages for API compatibility.

This fix may also resolve this issue
- https://github.com/promptfoo/promptfoo/issues/1393
Repository: promptfoo/promptfoo
Issue: When a test file is dereferenced, the test's Python assertion file is also dereferenced as an in-line script
Root cause: Configuration Error

**Describe the bug**
When `promptfooconfig.yaml` references `file://tests.yaml` which references `file://assertion.py`, the `assertion.py` file is loaded into the test in-line. When the assertion is later used, because it is in-line, promptfoo wraps it in a `main` function  and calls that instead of the `get_assert` function.

If the tests are included in the `promptfooconfig.yaml` directly, `file://assertion.py` is used during assertion and the `get_assert` function is called as expected.

**To Reproduce**

promptfooconfig.yaml
```yaml
providers:
  - id: "python:./provider.py"
tests:
  - file://tests.yaml
```

provider.py
```python
def call_api(prompt, options, context):
    return {"output": "LLM output."}
```

tests.yaml
```yaml
- vars:
    name: Should PASS
  assert:
  - type: python
    value: file://good_assertion.py
- vars:
    name: Should FAIL
  assert:
  - type: python
    value: file://bad_assertion.py
```

good_assertion.py
```python
def get_assert(output, context):
  return {
    "pass": True,
    "score": 1,
    "reason": f"Assertion function 'get_assert' was called.",
  }
```

bad_assertion.py
```python
import inspect
return {
  "pass": True,
  "score": 1,
  "reason": f"Assertion function '{inspect.stack()[0][3]}' was called.",
}
```

Execute
```bash
promptfoo eval -c promptfooconfig.yaml
```

**Expected behavior**
- "Should PASS" test passes.
- "Should FAIL" test fails with "[FAIL] Error running Python script: SyntaxError: 'return' outside function"

**Actual behavior**
- "Should PASS" test fails with "[FAIL] Python code execution failed: Python assertion must return a boolean, number, or {pass, score, reason} object. Got instead: {}"
- "Should FAIL" test passes.

**System information:**

```json
{
  "version": "0.118.3",
  "platform": {
    "os": "darwin",
    "release": "24.6.0",
    "arch": "arm64",
    "nodeVersion": "v24.7.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```

**Additional context**
Add any other context about the problem here.


@mark-richardson-ncino Thanks so much for reporting the issue. We released a fix for this in https://github.com/promptfoo/promptfoo/releases/tag/0.118.4 please pull the latest version / it a try and let me know if you run into any other issues! Feel free to book some time https://cal.com/michael-dangelo/15min if you'd like to debug further. 

Thanks @mldangelo! That's working as expected now.
Repository: promptfoo/promptfoo
Issue: Crescendo redteam strategy fails with "malformed response" error when target returns empty string output
Root cause: Validation Gap

**Describe the bug**
The Crescendo redteam strategy throws a "malformed response" error when the target provider returns an empty string as output, even though this should be a valid response. The error occurs in the `getTargetResponse` function which incorrectly treats empty strings as invalid responses.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Set up a Crescendo redteam strategy with any target provider
2. Configure the target provider to return empty string responses (this can happen due to content filtering, rate limiting, or model refusals)
3. Run the redteam evaluation: `promptfoo redteam`
4. See error: `Target returned malformed response: expected either \`output\` or \`error\` to be set. Instead got: {"output":""}`

**Expected behavior**
Empty string responses should be treated as valid output from the target provider. The Crescendo strategy should continue processing even when the target returns empty responses, as this is a legitimate scenario that can occur due to content filtering, model refusals, or other valid reasons.

**Screenshots**
```
    
[Crescendo] Error Running crescendo step: Error: 
    Target returned malformed response: expected either `output` or `error` to be set.

    Instead got: {"output":""}
```
**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: 0.118.3

**Additional context**
The root cause is in `/src/redteam/providers/shared.ts` in the `getTargetResponse` function (lines 133-147). The validation logic uses:

```typescript
if (targetRespRaw?.output) {
  // Handle output
} else if (targetRespRaw?.error) {
  // Handle error  
} else {
  // Throw error
}
```

The problem is that `targetRespRaw?.output` evaluates to `false` when `output` is an empty string `""`, causing the code to fall through to the error case even though the response is technically valid.

**Proposed fix:**
Change the validation logic to check for the existence of the `output` property rather than its truthiness:

```typescript
// Instead of: if (targetRespRaw?.output)
if (targetRespRaw?.hasOwnProperty('output')) {
  // Handle output (including empty strings)
}
```

This fix would allow empty string responses to be processed correctly while maintaining the same validation for truly malformed responses.

This was fixed in #5549 (merged as 0cdc01aca).

The root cause was exactly what you identified - the validation logic used truthiness checks (`if (targetRespRaw?.output)`) which evaluated to `false` for empty strings. We updated `getTargetResponse` to use `Object.prototype.hasOwnProperty.call()` instead, so empty string responses are now handled correctly.

The fix also propagated to Crescendo, Custom, Iterative, and IterativeTree providers, with tests covering the empty response scenarios.

Available in v0.118.10+. Thanks for the detailed bug report and proposed fix - made this straightforward to track down.
Repository: promptfoo/promptfoo
Issue: The custom Python assertions file can not be found
Root cause: External Dependency Breakage

**Describe the bug**
When using the custom Python function for assertion in an external file, the file can not be found. **The same configuration, same functions, files,... work fine in the version promptfoo@0.117.6**.

**Promptfoo Version: promptfoo@0.118.3**

**Steps to reproduce:**
1. Execute the command `promptfoo eval ....` to run the test case that has the Python assertion function in an external file.
2. Observe the error logs

**Expected behavior**
The test should work fine

The defined assertion in the test case in JSON:
```
      "assert":[
         {
            "type":"python",
            "value": "file://../assert.py:assert_source_web_field"
         }
      ]
```
The external function in assert.py:
```
   def assert_source_web_field(output: str, context):
        ##Coding logic...
```

**System information:**
```
opt/homebrew/lib/node_modules/promptfoo/dist/src/util/file.js:152
        throw new Error(`File does not exist: ${finalPath}`);
              ^
Error: File does not exist: /Path/to/custom/assert/function/assert.py:assert_source_web_field
    at maybeLoadFromExternalFile (/opt/homebrew/lib/node_modules/promptfoo/dist/src/util/file.js:152:15)
    at maybeLoadConfigFromExternalFile (/opt/homebrew/lib/node_modules/promptfoo/dist/src/util/file.js:201:12)
    at maybeLoadConfigFromExternalFile (/opt/homebrew/lib/node_modules/promptfoo/dist/src/util/file.js:197:27)
    at /opt/homebrew/lib/node_modules/promptfoo/dist/src/util/file.js:192:37
    at Array.map (<anonymous>)
    at maybeLoadConfigFromExternalFile (/opt/homebrew/lib/node_modules/promptfoo/dist/src/util/file.js:192:23)
    at maybeLoadConfigFromExternalFile (/opt/homebrew/lib/node_modules/promptfoo/dist/src/util/file.js:197:27)
    at /opt/homebrew/lib/node_modules/promptfoo/dist/src/util/file.js:192:37
    at Array.map (<anonymous>)
    at maybeLoadConfigFromExternalFile (/opt/homebrew/lib/node_modules/promptfoo/dist/src/util/file.js:192:23)
Node.js v23.11.0
```
Repository: promptfoo/promptfoo
Issue: Feature request: Load var from all *.txt files in a folder
Root cause: Unimplemented Feature Gap

First of all thank you very much for this tool, I've been looking for something like this and will be very useful for my workflow. There is one feature that I' missing (or perhaps it's possible to do already, but I don't know how). 

I'd like to be able to load vars from external files by providing a path with wildcards. 

Here is an example:
```
prompts:
  - "Identify language of the following text and extract all names of cities:\n{{text}}\n\nMake sure to anwer using valid JSON format."

providers:
  - openai:gpt-3.5-turbo
  - openai:gpt-4

tests:
#GENERAL tests - use for all texts
  - description: valid json
    vars:
      text: texts/*/*.txt
    assert:
      - type: is-json

#only for english texts
  - description: language check
    vars:
      text: texts/english/*.txt
    assert:
        - type: icontains-any
          value: ["english"]

#only for german texts
  - description: language check
    vars:
      text: texts/german/*.txt
    assert:
        - type: icontains-any
          value: ["german"]

#only for specific texts
  - description: correct city extracted?
    vars:
      text: file://texts/german/frankfurt.txt
    assert:
        - type: icontains-any
          value: ["frankfurt"]
```

In my use case I need to test against tens or hundreds of var values which can be categorized. Each category needs a different test. Some specific var values also need specific tests.

BTW I already sent this request on Discord, but thought here would be probably a better place for it. Thanks!

Thanks for creating this issue, the request makes sense!  It probably would require some glob/file logic to be added here https://github.com/promptfoo/promptfoo/blob/main/src/evaluator.ts#L49

This is now supported in [0.51.0](https://github.com/promptfoo/promptfoo/releases/tag/0.51.0)
Repository: promptfoo/promptfoo
Issue: How to generate separate report or a breakdown report per test suite instead of just one single report for all test suites?
Root cause: Unimplemented Feature Gap

For example, for the following configuration, how do I generate the report such that I get separate stats for each test-suite instead of combined stats for all test-suites.

```
prompts: [prompt.json]

providers:
  - providers/my-provider.js:

tests:
  - tests/generation/test-suite1.yaml
  - tests/generation/test-suite2.yaml
```

Hmm, I would probably just run them as separate evals.  Is there a reason you want to run them all together, but keep the stats separate?

Closing this out for now, feel free to reopen with your thoughts on this!
Repository: promptfoo/promptfoo
Issue: Feature request: Add CometAPI provider preset/docs (OpenAI-compatible)
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Many teams using promptfoo want optional provider redundancy, regional coverage, and cost control. Today, configuring alternative OpenAI-compatible providers is possible but not discoverable, and users often ask for clear presets/docs. Adding CometAPI as a documented provider/preset would reduce friction and improve evaluation portability.

**Describe the solution you'd like**
- Add a CometAPI provider preset and docs page (or a section under OpenAI) showing how to set `baseUrl` and API key.
- Ensure streaming and non-streaming modes work as expected.
- Optionally include a quick command/example for model listing if promptfoo supports that in providers.

**Describe alternatives you've considered**
- Manually configuring an OpenAI provider with a custom `baseUrl`. Works, but users frequently miss this option and it varies across tools.
- Relying on other aggregators. Adds an extra dependency and less control.

**Additional context**
CometAPI Resources:
- Website: https://www.cometapi.com/
- API Documentation: https://api.cometapi.com/doc
- Base URL: https://api.cometapi.com/v1/
- Model List: https://api.cometapi.com/v1/models
- Get API Key: https://api.cometapi.com/console/token
- Pricing: https://api.cometapi.com/pricing

Compatibility:
- CometAPI exposes OpenAI-compatible endpoints, so a provider config is typically a matter of setting `baseUrl` and the `COMETAPI_API_KEY`.

Minimal example (promptfooconfig.yaml):
```yaml
providers:
  - id: openai:gpt-5-mini
    config:
      baseUrl: https://api.cometapi.com/v1/
      apiKey: ${COMETAPI_API_KEY}

prompts:
  - "Summarize: {{ input }}"

tests:
  - vars:
      input: "Promptfoo is an open-source tool for evals."
    assert:
      - type: contains
        value: "open-source"
```

Implementation Offer:
- Happy to contribute a docs + preset PR if this aligns with your roadmap. We can also provide a temporary test API key to verify the integration.


@TensorNull we would welcome this contribution. Please have a look at https://www.promptfoo.dev/docs/contributing/#adding-a-new-provider - there are many examples in src/providers of thin wrappers around the openai chat provider and you might be able to one shot this with an llm. It should look something like the reverse of https://github.com/promptfoo/promptfoo/pull/5599 Send me an email when you're done (michael @ promptfoo.dev) for some promptfoo swag! Thank you so much! 

Looks like this is all set. CometAPI support was added in [#5721](https://github.com/promptfoo/promptfoo/pull/5721?utm_source=chatgpt.com).

Used as cometapi:chat with the COMETAPI_KEY environment variable.

Docs: [https://www.promptfoo.dev/docs/providers/cometapi/](https://www.promptfoo.dev/docs/providers/cometapi/?utm_source=chatgpt.com)

Released in the September 2025 build (around v0.118.x). Ready for close @mldangelo
Repository: promptfoo/promptfoo
Issue: Feature Request: Add max_tokens config to Azure OpenAI provider
Root cause: Unimplemented Feature Gap

You cannot configure max_tokens as a config option to Azure OpenAI providers. As of now the only way that I can set this is via the OPENAI_MAX_TOKEN env var.

options interface: https://github.com/promptfoo/promptfoo/blob/main/src/providers/azureopenai.ts#L14-L57

line in azure openai providers source where max_tokens is set to env or 1024: 
- AzureOpenAiCompletionProvider: https://github.com/promptfoo/promptfoo/blob/main/src/providers/azureopenai.ts#L241
- AzureOpenAiChatCompletionProvider: https://github.com/promptfoo/promptfoo/blob/main/src/providers/azureopenai.ts#L326

It would be pretty awesome to be able to control that via provider config options.

Nice catch.  #561 adds this param.  It'll be in the next release!

In the meanwhile, as a workaround, you can set
```yaml
config:
  passthrough:
    max_tokens: 2048
```
in the provider config.  `passthrough` will just blindly send whatever's in it to the API.

@typpo (first of all ... awesome username) thanks for the heads up on passthrough!

Released in [0.48.0](https://github.com/promptfoo/promptfoo/releases/tag/0.48.0)!
Repository: promptfoo/promptfoo
Issue: Internal monkey patches shouldn't leak into "user-level" implementations
Root cause: Architectural Constraint

**Describe the bug**
Some implementations of this library is leaking its own monkey patch to third-party implementations when importing the library. Ideally, it can bring some error prone behaviors (as the screenshot below, where the monkey patch was affecting the implementation of the flow).

**To Reproduce**
1. Install the library `npm i -D promptfoo@latest`
2. Import it locally in your file.
3. Execute the `evaluate` function from the library.
4. Has a fetch somewhere else during the flow, you will see the logs.

**Expected behavior**
Ideally, if the libraries needs any kind of monkey patch for some reason, it should ideally scope it into the library level, not sharing it with the user implementation, which can cause conflictions.

An option on that, should be having an abstraction over the `fetch` function, instead of patching it.

**Screenshots**

<img width="719" height="87" alt="Image" src="https://github.com/user-attachments/assets/90b01c33-6758-4296-b9e2-ccbf143e191a" />

https://github.com/promptfoo/promptfoo/blob/43c878c9aec0b61770a0fcaeb824e15da13fa5cb/src/util/fetch/monkeyPatchFetch.ts#L62

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

```json
{
    "promptfooVersion": "0.118.6",
    "nodeVersion": "v22.17.0",
    "platform": "darwin",
    "arch": "arm64",
}
```

**Additional context**
Add any other context about the problem here.


Thanks for the report @noghartt, looking into this now.

Hi @noghartt,

Sorry for the bug and for not closing this sooner - this was fixed in **PR #5665** on September 19, 2025 (1 day after you reported it), released in version 0.118.7.

Global fetch patches were leaking into user code. This has been resolved - promptfoo's internal modifications no longer affect user-level implementations, and there's better isolation between promptfoo internals and your code.

Please pull the latest version and your user code should no longer be affected by promptfoo's internal patches. If you notice any other unexpected behavior, feel free to comment or re-open.

Thanks for the excellent bug report!
Repository: promptfoo/promptfoo
Issue: Mistral provider is hard-coded to https://api.mistral.ai/v1/chat/completions
Root cause: Architectural Constraint

There is no way for the Mistral provider to override the API URL. So you can't test against, for example, Mistral hosted on Azure. It would be ideal if this provider supported the `apiHost`, `apiBaseUrl`, and `apiKeyEnvar` configuration parameters like the OpenAI provider does.

If there's interest, I can work on a PR for this.

Would definitely merge a PR for this.  Feel free to take a stab at it!