zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Provider output in 0.117.5 is a string instead of object (breaking change)
Root cause: Algorithmic Error

**Describe the bug**

After upgrading from `0.117.4` to `0.117.5`, the provider output type is now a string instead of an object.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
providers:
  - id: openrouter:google/gemini-2.5-flash-lite
    config:
      response_format:
        type: json_schema
        json_schema:
          name: answer
          strict: true
          schema:
            {
              "type": "object",
              "properties": {
                "answer": {
                  "type": "integer",
                },
              },
              "required": ["answer"],
              "additionalProperties": false
            }

prompts:
  - 'How much is 1 + 1?'

tests:
  - assert:
      - type: equals
        value: 2
        transform: output.answer
```

**Expected behavior**

I expected to be able to call `output.answer` directly from the `transformer`, but now we need to parse the string first (`JSON.parse(output).answer`).

**Screenshots**

Eval result from 0.117.4:

<img width="2090" height="798" alt="Image" src="https://github.com/user-attachments/assets/0dc33931-7ecf-4b87-943b-91398b0f59f9" />

Eval result from 0.117.5:

<img width="2092" height="594" alt="Image" src="https://github.com/user-attachments/assets/1c48aedc-5e3b-455a-b7d2-ec46c0dbdf8e" />

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: 0.117.5

**Additional context**
Add any other context about the problem here.


Hi, this should be resolved in the latest version (0.117.6).
Repository: promptfoo/promptfoo
Issue: GPT-5-mini response includes "Reasoning: []" text which breaks JSON parsing
Root cause: External Dependency Breakage

**Describe the bug**

I'm using gpt-4-mini provider for structured json outputs:

```
providers:
  - id: openai:responses:gpt-5-mini
    config:
      response_format:
        type: json_schema
        schema: file://./schema.yaml
```

I also use a transform step in one of the assertions, to get the value of a single response object field:

```
transform: output.MyField
```

When I run the evaluations, I get an error in the console:

```
Starting evaluation eval-7nu-2025-08-18T14:06:57
Running 1 test cases (up to 4 at a time)...
Failed to parse JSON output: SyntaxError: Unexpected token 'R', "Reasoning:"... is not valid JSON
Group 1/1 [████████████████████████████████████████] 100% | 1/1 | Complete | openai:responses:gpt-5-mini
```

The response that's shown in the promtfoo UI, when I remove the transform field (as otherwise I only get "Transform function did not return a value" error):

```
Reasoning: []{
  "MyField": "..."
}
```


**To Reproduce**
1. Use json_schema response format with the latest OpenAI reasoning models (e.g. gpt-5-mini)
2. This should already report an error in the console, but some assertions will still work
3. Additionally you can use an assertion with a transform option that reads a field of the response object
4. This will not work at all, and afaik there is no workaround atm

**Expected behavior**
I would expect promptfoo to better handle the new reasoning model outputs (strip or parse the reasoning tokens), or at least provide a way to overwrite response parsing as a fallback for scenarios like this.

**Screenshots**

<img width="1070" height="284" alt="Image" src="https://github.com/user-attachments/assets/3a52334d-b22f-4024-b425-bc4fd6121efe" />

**System information:**

```
=======================================================================================
Promptfoo Debug Information
=======================================================================================
{
  "version": "0.117.5",
  "platform": {
    "os": "darwin",
    "release": "23.5.0",
    "arch": "arm64",
    "nodeVersion": "v24.6.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```


- Promptfoo version: 0.117.5


Thanks for reporting, @bartolomej. This is fixed in https://github.com/promptfoo/promptfoo/pull/5235 which will be included in the next release.

> Thanks for reporting, @bartolomej. This is fixed in https://github.com/promptfoo/promptfoo/pull/5235 which will be included in the next release.

Thats great! Will keep track of the releases, thanks

@bartolomej [0.117.6](https://www.npmjs.com/package/promptfoo) has been released! 

> @bartolomej [0.117.6](https://www.npmjs.com/package/promptfoo) has been released! 

Whoah, that was fast! Cheers
Repository: promptfoo/promptfoo
Issue: `additionalOption` is not a dictionary in the `call_api` function from a python script
Root cause: Interface Contract Mismatch

I've recently tested the config from a provider defined in a python script

I've followed the instructions in [the doc](https://promptfoo.dev/docs/providers/python/) but realized that the options variable is a string dictionary, for example if I define something like the following in a config file:
```
providers:
   - id: python:providers/eval_chain.py
     config:
         model_name: "xyz"
```
I'll get something like this for options:
```
"{"id":"python:providers/eval_chain.py","config":{"model_name":"xyz","basePath":"."}, "env":{}}"
```

The workaround that I use to solve this issue is to parse the options as a dict using `json.loads`, ie
```
def call_api(prompt, options, context):
    options_dict = json.loads(options)
    options.get("config").get("model_name", "default_model_name")
....
```

Thanks for flagging this!  #532 should fix.
Repository: promptfoo/promptfoo
Issue: __expected column doesn't support blank cells
Root cause: Validation Gap

In the CSV test format, you cannot use the __expected column unless you have an assertion for every case. If some case has no assertion, the code adds an empty assertion which will fail the case.
Would be easy to fix by checking for truthy `value` before adding to asserts, in this code:
https://github.com/promptfoo/promptfoo/blob/792a5aa5a7b7cdf66513733fbd3fd07185e92413/src/util.ts#L350

Thanks!  This is fixed in [d398866](https://github.com/promptfoo/promptfoo/commit/d3988666d95528f36c85165509bf634d8848da83) and will be published in the next release.

This is fixed in version 0.17.0
Repository: promptfoo/promptfoo
Issue: v0.45 breaks input data files
Root cause: External Dependency Breakage

v0.45 seems to break the ability to load input data

`promptfooconfig.yaml`

```
prompts:
  - file://prompt3.json
  - file://prompt7.json

providers:
  # - id: openai:chat:gpt-4-0613
  # - id: anthropic:completion:claude-3-opus-20240229
  - id: openai:chat:gpt-4-vision-preview

tests:
  - vars:
      data: file://data1.json
  - vars:
      data: file://data2.json
  - vars:
      data: file://data3.json
  - vars:
      data: file://data4.json
  - vars:
      data: file://data5.json
```

`npx promptfoo@v0.45 eval --no-cache` results in the model not seeing any of the input data
`npx promptfoo@v0.44 eval --no-cache` works as expected

Thanks for the catch.  #531 fixes and adds relevant tests.
Repository: promptfoo/promptfoo
Issue: Can't run promptfoo 0.117.7 after upgrading
Root cause: External Dependency Breakage

In my package.json:
```
"promptfoo": "^0.117.7",
```

I deleted my node_modules and called `yarn` to reinstall.

When I run promptfoo:
```
/my-repo/node_modules/promptfoo/dist/src/commands/mcp/lib/performance.js:16
        this.cache = new lru_cache_1.LRUCache({
                     ^

TypeError: lru_cache_1.LRUCache is not a constructor
    at new EvaluationCache (/my-repo/node_modules/promptfoo/dist/src/commands/mcp/lib/performance.js:16:22)
    at Object.<anonymous> (/my-repo/node_modules/promptfoo/dist/src/commands/mcp/lib/performance.js:129:27)
    at Module._compile (node:internal/modules/cjs/loader:1546:14)
    at Object..js (node:internal/modules/cjs/loader:1689:10)
    at Module.load (node:internal/modules/cjs/loader:1318:32)
    at Function._load (node:internal/modules/cjs/loader:1128:12)
    at TracingChannel.traceSync (node:diagnostics_channel:315:14)
    at wrapModuleLoad (node:internal/modules/cjs/loader:218:24)
    at Module.require (node:internal/modules/cjs/loader:1340:12)
    at require (node:internal/modules/helpers:141:16)

Node.js v22.10.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
```

Thanks for flagging.  Should be fixed in #5309

Fixed in 0.117.8 :)
Repository: promptfoo/promptfoo
Issue: can not load txt file in vars
Root cause: External Dependency Breakage

using promptfoo 0.45.1
  vars: 
    prompt: file://testcases/015_simple_mahjong_prompt.txt

will get error:
![image](https://github.com/promptfoo/promptfoo/assets/1726527/9119471c-f8b4-4165-9c2c-11c44729c68c)

it worked before.

Which Anthropic provider are you using?  I'm guessing this is related to #526.  Could you try specifying your provider as `anthropic:completion:modelname`?

Should be fixed in 0.46.0
Repository: promptfoo/promptfoo
Issue: Header dropdowns collapse on hover transition and overlap when switching triggers

- **Summary**: The `Create` and `Evals` dropdowns in the header both have the same issue:
  - Moving the cursor from the trigger button into the dropdown causes the menu to close.

- **Steps to reproduce**
  1. Hover `Create` to open the dropdown.
  2. Move the mouse downward toward the menu items → the dropdown collapses before you can enter it.

- **Expected behavior**
  - The dropdown should remain open while the cursor moves from the trigger into the menu.

- **Actual behavior**
  - Dropdown closes during hover transition.

- **Environment**
  - Browsers: Chrome, Opera (latest)
  - OS: Linux
  - Relevant file: `src/app/src/components/Navigation.tsx` (MUI `Popper` + hover handlers)

- **Promptfoo version**
  - latest

- **Acceptance criteria**
  - Hovering from trigger to menu does not close the dropdown.
  - Switching triggers immediately closes any previously open dropdown.

- **Loom of observed behavior**
  - https://www.loom.com/share/e3f2c3c4e13f489db464f1bd0f85e8e0?sid=0535244d-4252-4bbb-864c-51669a5a090b

Thanks for the PR - will be in the next release
Repository: promptfoo/promptfoo
Issue: Allow use of Custom ApiProviders in DefaultTest Configuration
Root cause: Validation Gap

I have some code that is bundled up at compile time, but I want to leverage a custom validator ApiProvider class in my default test but it consistently crashes here because of the [loadApiProvider implementation depending on a provider path](https://github.com/promptfoo/promptfoo/blob/feaa4a8f9ab6f529bd0dcdcadc89ba968a34f00f/src/providers/index.ts#L19)

We have a slightly more complex flow for invoking our model endpoint from PromptFoo for evaluation due to workload identity federation and the disallowing of key access in our org, so I've been looking to replace the bolded section of  EvaluateTestSuite configuration like this:

```javascript
return {
      prompts: config.questions,
      providers: providers,
      tests: config.evaluations.map((evaluation) => ({
        vars: evaluation.vars,
        assert: evaluation.assert.map((assertion) => ({
          ...assertion,
          type: assertion.type as AssertType,
        })),
      })),
      defaultTest: {
        assert: config.defaultAssertions.map((assertion) => ({
          ...assertion,
          type: assertion.type as AssertType,
        })),
        options: {
          provider: {
            id: 'azureopenai:chat:gpt-4o-mini',
            config: {
              apiHost: 'xxxxxxxxxx.openai.azure.com',
              apiKey: openAIKey,
            },
          },
        },
      },
    };
  }
```

Replacing the defaultTest.options.provider section with an instance of an ApiProvider class like so:
```javascript 
provider: new VanillaValidator({ id: 'vanilla_validator' }),
```

If I modify the code here: https://github.com/promptfoo/promptfoo/blob/05c42d34cc4e599e5378b9b11f83270fd93d8a66/src/index.ts#L65 
to add a check like so, it handles my custom api provider properly.

```javascript
 // If the value is an ApiProvider class, use it directly.
        if(!providers_2.isApiProvider(constructedTestSuite.defaultTest?.options?.provider) && !providers_2.isApiProvider(constructedTestSuite.defaultTest?.provider))
        {
            // Resolve defaultTest.provider
            if (constructedTestSuite.defaultTest?.provider) {
                constructedTestSuite.defaultTest.provider = await (0, providers_1.resolveProvider)(constructedTestSuite.defaultTest.provider, providerMap, { env: testSuite.env });
            }
            // Resolve defaultTest.options.provider
            if (constructedTestSuite.defaultTest?.options?.provider) {
                constructedTestSuite.defaultTest.options.provider = await (0, providers_1.resolveProvider)(constructedTestSuite.defaultTest.options.provider, providerMap, { env: testSuite.env });
            }
        }   
    }
```

 I don't know if this is the right place for us to be handling this, but would love an option to pass in an ApiProvider properly to that defaultTest without having to fork the repo.

This is resolved with the change. Thanks so much!
Repository: promptfoo/promptfoo
Issue: example in assertion overview document 

 In configuration > Assertions & metrics > Overview #python
the example goes 
```
import sys
import json

output = sys.argv[1]
context = json.loads(sys.argv[2])

print(f'Received {output} with variables {context}')

return {
  'pass': True,
  'score': 0.5,
  'reason': 'Some custom reason',
}
```
but it should like 
```
import sys
import json

output = sys.argv[1]
context = json.loads(sys.argv[2])

# print(f'Received {output} with variables {context}')

print(json.dumps({
  'pass': False,
  'score': 0.5,
  'reason': 'Some custom reason',
}))
```
or it can not be run by copy & paste
I'm not sure if this minor issue should be opened here or there is some other way to do it. please let me know.

Thanks for catching this!  Fixed in a6259f5ac9ea37eb9c370ae5a632c1c64f08410b
Repository: promptfoo/promptfoo
Issue: Response schema file not working with Vertex AI provider
Root cause: Configuration Error

**Describe the bug**
When using `vertex` provider setting pointing `responseSchema` to a file does not seem to have any effect. While setting it through `generationConfig.response_schema` seem to work as expected.

**To Reproduce**
Create a minimal config

```
description: "My schema eval"

prompts:
  - "Write a tweet about {{topic}}"
  

providers:
  - id: vertex:gemini-2.5-flash
    config:
      region: europe-west1
      projectId: squidler-dev
      responseSchema: file://schema.json

tests:
  - vars:
      topic: bananas
    assert:
      - type: equals
        value: Bananas are great
        transform: JSON.parse(output).tweet
```

With schema (`schema.json`) 
```
{
  "type": "object",
  "properties": {
    "tweet": {
      "description": "The tweet",
      "type": "string"
    },
  },
  "required": [ "tweet" ]
}

```

**Expected behavior**
The _tweet_ is extracted and the assert fails nicely
```
[FAIL] Expected output "Bananas are the ultimate snack! Sweet, portable, and packed with potassium.
What's your favorite way to enjoy a banana? 🍌 #bananas #healthyfood" to equal "Bananas are great"
```

**Actual behavior**
The _tweet_ is NOT extracted
```
[ERROR] SyntaxError: Unexpected token 'H', "Here are a"... is not valid JSON
SyntaxError: Unexpected token 'H', "Here are a"... is not valid JSON
    at JSON.parse (<anonymous>)
    at eval (eval at getInlineTransformFunction (/opt/homebrew/Cellar/prompt...
```

**System information:**

```
  "version": "0.118.0",
  "platform": {
    "os": "darwin",
    "release": "24.6.0",
    "arch": "arm64",
    "nodeVersion": "v24.7.0"
  },
  "env": {},
```
Repository: promptfoo/promptfoo
Issue: Config file with commandLineOptions.envPath is not respected
Root cause: Configuration Error

## Summary

The `commandLineOptions.envPath` configuration option in promptfoo YAML config files is not properly processed, causing environment variables to not be loaded when specified via config file rather than command line.

## Root Cause

Environment variable loading occurs before configuration parsing:

1. **Early env setup**: `src/commands/eval.ts:418` calls `setupEnv(cmdObj.envPath)` before config loading
2. **Config loading**: `src/util/config/load.ts:546-698` processes config files but excludes `commandLineOptions` from the returned config object (line 546: `Omit<UnifiedConfig, 'evaluateOptions' | 'commandLineOptions'>`)
3. **Missing integration**: `commandLineOptions.envPath` from config files is never merged back into `cmdObj.envPath` before the `setupEnv()` call

## Expected vs Actual Behavior

**Expected**: This YAML config should load environment variables from the specified path:
```yaml
commandLineOptions:
  envPath: ../../.env.local
```

**Actual**: Environment variables are only loaded when using `--env-path` command line flag.

## Code References

- `src/commands/eval.ts:418` - Early `setupEnv(cmdObj.envPath)` call
- `src/util/config/load.ts:453-456` - Config file `commandLineOptions` are collected but not applied to `cmdObj`
- `src/util/config/load.ts:546` - Config object explicitly omits `commandLineOptions`
- `src/main.ts:63-67` - Command line env loading works via preAction hook
- `src/util/index.ts:setupEnv()` - Environment loading function

The issue affects promptfoo v0.117.11-0.118.0 and likely earlier versions.
Repository: promptfoo/promptfoo
Issue: Support Messages api in Bedrock for anthropic v3+
Root cause: Algorithmic Error

It's now supporting only messages API which is to be called like this:

```typescript
async function main() {
  const message = await client.messages.create({
    model: 'anthropic.claude-3-sonnet-20240229-v1:0',
    max_tokens: 256,
    messages=[{"role": "user", "content": "Hello, world"}]
  });
  console.log(message);
}
```

Looks like it's using same api as Anthropic provider which supports {{system_prompt}}: 
https://github.com/promptfoo/promptfoo/blob/main/site/docs/providers/anthropic.md#latest-api-messages

additional data:
https://docs.anthropic.com/claude/reference/claude-on-amazon-bedrock
https://community.aws/content/2dJmYpKlFNh6NOeC71GIZWZkfST/use?lang=en

Published in [0.49.0](https://github.com/promptfoo/promptfoo/releases/tag/0.49.0)!

I think there's a bug in split of model id using `:` , it doesn't send the last ":0" therefore it doesn't hit the correct endpoint.
it's in 
https://github.com/promptfoo/promptfoo/blob/94425e9a55e7dc1537d37b576b72bdf9af1e069a/src/providers.ts#L208
 I believe 

I see what you mean.  As a temporary workaround, I think you can specify it like so:
```
bedrock:completion:anthropic.claude-3-haiku-20240307-v1:0
```
Repository: promptfoo/promptfoo
Issue: [Feature Request] Support custom WebSocket URLs in OpenAI Realtime Provider
Root cause: Unimplemented Feature Gap

The current OpenAI Realtime Provider in promptfoo hardcodes the WebSocket URL to OpenAI's endpoint (`wss://api.openai.com/v1/realtime`). This prevents users from:

  - Using OpenAI-compatible realtime APIs from other providers (e.g., Azure OpenAI, custom deployments)
  - Testing against local or development realtime endpoints
  - Using proxy servers or custom routing for the realtime API

 Current Implementation
  Looking at `src/providers/openai/realtime.ts`, the WebSocket URLs are hardcoded:

  ```typescript
  // Line 162
  const wsUrl = `wss://api.openai.com/v1/realtime/socket?client_secret=${encodeURIComponent(clientSecret)}`;

  // Line 770 & 1208
  const wsUrl = `wss://api.openai.com/v1/realtime?model=${encodeURIComponent(this.modelName)}`;
```
  Proposed Solution

  Modify the realtime provider to respect the existing apiBaseUrl configuration (similar to how other OpenAI providers work), converting HTTP URLs to WebSocket URLs:
```
  private getWebSocketUrl(modelName: string): string {
    const apiUrl = this.getApiUrl(); // This already respects apiBaseUrl
    const wsUrl = apiUrl
      .replace('https://', 'wss://')
      .replace('http://', 'ws://');

    return `${wsUrl}/realtime?model=${encodeURIComponent(modelName)}`;
  }
```
Proposed Configuration

  This would allow configurations like:
```
  providers:
    - id: openai:realtime:gpt-4o-realtime-preview
      config:
        apiBaseUrl: 'https://my-custom-api.com/v1'
        # Would connect to: wss://my-custom-api.com/v1/realtime
        modalities: ['text']
        temperature: 0.7
```
  Use Cases

  1. Azure OpenAI Service: Users with Azure OpenAI deployments that support realtime APIs
  2. Custom OpenAI-compatible APIs: Services like vLLM, LocalAI, or custom implementations
  3. Development/Testing: Local development servers or staging environments
  4. Corporate Proxies: Organizations routing through internal proxy servers
  5. Multi-region Deployments: Using region-specific endpoints

  Additional Context

  - Other OpenAI providers already support custom base URLs via getApiUrl() method
  - The WebSocket provider exists but doesn't provide the same realtime protocol handling
  - This would enable broader adoption of promptfoo for testing OpenAI-compatible realtime implementations

Thank you for the feature request @yyamaguchi93!
Repository: promptfoo/promptfoo
Issue: Qwen3 missing from website Foundation Model Security Reports

Hello,

I know the list could be long, but  https://www.promptfoo.dev/models/ lacks Qwen3 models. It would be great to be able to compare with the others at least:
- Qwen/Qwen3-30B-A3B-Instruct-2507
- Qwen/Qwen3-235B-A22B-Instruct-2507
- Qwen/Qwen3-4B-Instruct-2507

Thanks!

Hi @gelim, late reply but Qwen red teams are available in the model reports view.  For example: https://promptfoo.dev/models/reports/qwen3-235b-a22b-2507
Repository: promptfoo/promptfoo
Issue: response error while using replicate's Private model
Root cause: Interface Contract Mismatch

![crop](https://github.com/promptfoo/promptfoo/assets/159236680/6b9f9c1c-b4cd-486b-a341-81ed3bec7c7a)

here is the replicate's model link : https://replicate.com/abrar-tericsoft/mistral
what type of error is this?

The implementation expects a list of strings in `output` object, which is what most popular models return, but it looks like this endpoint is returning a string.

#547 adds support for string responses.

Thank you for support @typpo 

Published in 0.47.0!
Repository: promptfoo/promptfoo
Issue: fix: Placeholder api key blocks usage of Helicone Cloud Gateway
Root cause: Validation Gap

https://github.com/promptfoo/promptfoo/blob/68974572a3500f857cfc6d1bf3d26142559fa3bf/src/providers/helicone.ts#L48

Due to the placeholder API key and not using `HELICONE_API_KEY` it is not possible to use helicone **cloud** AI gateway without hardcoding the api key.

This works:
```yaml
id: helicone:claude-sonnet-4/anthropic
label: Claude Sonnet 4
config:
  temperature: 0.0
  max_tokens: 2000
  baseUrl: https://ai-gateway.helicone.ai
  apiKey: sk-helicone-eu-1234-1234-1234
  headers:
          Helicone-Property-Environment: 'development'
          Helicone-User-Id: 'promptfoo'
          Helicone-Property-Feature: 'promptfoo_testing'
```

but you would want to set the key in the env var `HELICONE_API_KEY` and leave out the `apiKey` key from the config.

# fix

```typescript
apiKey: config.apiKey || process.env.HELICONE_API_KEY || 'placeholder-api-key',
```