zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Promptfoo is affected by the Shai-Hulud worm, through the posthog-node dependency
Root cause: External Dependency Breakage

npm ls posthog-node
pmw@1.0.0 /Users/postermywall/postermywall/git/webapp/server/react
└─┬ promptfoo@0.119.11
  └── posthog-node@5.13.3

<img width="201" height="190" alt="Image" src="https://github.com/user-attachments/assets/d2084716-25f3-4ac3-bc55-4421d202823e" />

<p><a href="https://linear.app/promptfooo/issue/ENG-1111/promptfoo-is-affected-by-the-shai-hulud-worm-through-the-posthod-node">ENG-1111</a></p>

Greetings!  The vulnerable version of posthog was taken down quite quickly, and replaced with a clean version per https://status.posthog.com/incidents/kv3nj636f59c.  This should only impact installations that happened during the window where the vulnerable version was available.  

That having been said - we've updated our version of posthog:
https://github.com/promptfoo/promptfoo/blob/main/package.json#L288

And released a new version with the fix: 
https://github.com/promptfoo/promptfoo/releases/tag/promptfoo-v0.119.13

Let us know if you run into any issues with the latest version!

good to hear 👍
Repository: promptfoo/promptfoo
Issue: Add verbosity support to Azure Responses provider (azure:responses)
Root cause: Unimplemented Feature Gap

The OpenAI Responses provider (`openai:responses`) supports the `verbosity` parameter directly in config.
  However, the Azure Responses provider (`azure:responses`) does not support `verbosity` in its type definitions,
  despite Azure OpenAI's Responses API supporting this parameter.

  ## Current Workaround

  To set verbosity with Azure, you must use `passthrough`, but this overwrites the entire `text` object, requiring
  you to re-specify the format:

  ```yaml
  # Workaround for azure:responses
  - id: azure:responses:my-deployment
    config:
      apiHost: "my-deployment.openai.azure.com"
      reasoning_effort: "low"
      passthrough:
        text:
          format:
            type: "text"
          verbosity: "high"
```

  This is problematic because:
  1. Can't easily add verbosity to configs using response_format for JSON schema output (passthrough would
  overwrite the schema)
  2. Requires knowledge of internal request body structure

  Expected Behavior

  - id: azure:responses:my-deployment
    config:
      reasoning_effort: "low"
      verbosity: "high"  # Should work natively

  Additional Context

  - Relevant code: src/providers/azure/responses.ts
  - The text object in the request body should include verbosity when configured
  - Azure API expects: { text: { format: { type: "text" }, verbosity: "high" } }

<p><a href="https://linear.app/promptfooo/issue/ENG-1144/add-verbosity-support-to-azure-responses-provider-azureresponses">ENG-1144</a></p>
Repository: promptfoo/promptfoo
Issue: `azure:responses` missing `isReasoningModel` config override (parity with `azure:chat`)
Root cause: Architectural Constraint

## Problem

  The `azure:responses` provider's `isReasoningModel()` method only checks deployment name prefixes, unlike
  `azure:chat` which also supports a `config.isReasoningModel` override.

  This means `reasoning_effort` is silently ignored when using custom Azure deployment names that don't start with
  `o1`, `o3`, `o4`, or `gpt-5`.

  ## Current Behavior

  **`azure:chat`** (correct):
  ```typescript
  // src/providers/azure/chat.ts
  protected isReasoningModel(): boolean {
    if (this.config.isReasoningModel || this.config.o1) {
      return true;  // ✅ Config override supported
    }
    // fallback to deployment name check
  }

  azure:responses (missing override):
  // src/providers/azure/responses.ts
  isReasoningModel(): boolean {
    return (
      this.deploymentName.startsWith('o1') ||
      this.deploymentName.startsWith('o3') ||
      this.deploymentName.startsWith('o4') ||
      this.deploymentName.startsWith('gpt-5')
    );  // ❌ No config override
  }
```

  Impact

  Users with custom Azure deployment names (common in enterprise) can't use reasoning_effort:

```yaml
  # This silently ignores reasoning_effort because deployment name
  # "my-company-gpt-5-prod" doesn't START with "gpt-5"
  - id: azure:responses:my-company-gpt-5-prod
    config:
      reasoning_effort: "low"  # ❌ Ignored silently
```

  Expected Behavior

```yaml
  # Should work with explicit config override
  - id: azure:responses:my-company-gpt-5-prod
    config:
      isReasoningModel: true   # Override auto-detection
      reasoning_effort: "low"  # ✅ Now applied
```

  Suggested Fix

  Update src/providers/azure/responses.ts to match azure:chat:

```typescript
  isReasoningModel(): boolean {
    if (this.config.isReasoningModel || this.config.o1) {
      return true;
    }
    return (
      this.deploymentName.startsWith('o1') ||
      this.deploymentName.startsWith('o3') ||
      this.deploymentName.startsWith('o4') ||
      this.deploymentName.startsWith('gpt-5')
    );
  }
```


<p><a href="https://linear.app/promptfooo/issue/ENG-1145/azureresponses-missing-isreasoningmodel-config-override-parity-with">ENG-1145</a></p>
Repository: promptfoo/promptfoo
Issue: MCP Tool Timeout (`-32001`)
Root cause: Configuration Error

## Bug Report: MCP Tool Timeout (`-32001`) for `tool_a` and `tool_b`

**Problem summary:**
When running a promptfoo evaluation that uses MCP-enabled providers, the MCP servers (`tool_a` and `tool_b`) consistently return:

```
MCP error -32001: Request timed out
```

This occurs even though both tools are expected to take **up to 15 minutes** to produce a valid response.

---

## 🔁 Steps to Reproduce

1. Use the following promptfoo configuration:

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: "Eval GPT-OSS"

prompts:
  - "langfuse://gaia-gpt-oss-e4-chat@production:chat"

providers:
  - id: openai:chat:openai/gpt-oss-120b
    config:
      apiBaseUrl: https://<MODEL_SERVER_ADDRESS>/v1
      apiKey: <MODEL_SERVER_API_KEY>
      mcp:
        enabled: true
        debug: true
        verbose: true
        timeout: 120000
        servers:
          - name: tool_a
            type: "streamable-http"
            url: "https://<TOOL_A_MCP_ENDPOINT>"
          - name: tool_b
            type: "streamable-http"
            url: "https://<TOOL_B_MCP_ENDPOINT>"
```

2. Run a test case


3. Observe that the model attempts to call `tool_a` (or `tool_b`).

---

## ❗ Actual Behavior

The MCP client returns:

```
Tool Error: MCP error -32001: Request timed out
```

This happens consistently for both `tool_a` and `tool_b`, regardless of query or prompt.

---

## ✅ Expected Behavior

* `tool_a` and `tool_b` should be callable even when they require **up to 15 minutes** to compute a result.
* The MCP client should keep the connection open and accept long-running responses.
* The model should receive the tool output and include it verbatim in its final answer.

---

## 🔍 Additional Context

* Both MCP servers (`tool_a` and `tool_b`) may legitimately take **5–15 minutes** to return output.
* Increasing the MCP timeout in promptfoo does not resolve the issue.
* The connection appears to terminate **before** the MCP server starts streaming any response.
* The issue occurs before any part of the tool result is received.
* The error always occur after 60 seconds

### .env content
```
PROMPTFOO_EVAL_TIMEOUT_MS=1200000
PROMPTFOO_MAX_EVAL_TIME_MS=1200000
REQUEST_TIMEOUT_MS=1200000
MCP_SERVER_REQUEST_TIMEOUT=1200000
MCP_CONNECTION_TIMEOUT=1200000

LANGFUSE_PUBLIC_KEY="pk"
LANGFUSE_SECRET_KEY="sk"
LANGFUSE_HOST=<url>
```
---


<p><a href="https://linear.app/promptfooo/issue/ENG-1146/mcp-tool-timeout-32001">ENG-1146</a></p>
Repository: promptfoo/promptfoo
Issue: Add Strands Agents SDK example
Root cause: Unimplemented Feature Gap

It would be great to add an example showing how to evaluate agents implemented with [Strands SDK](https://github.com/strands-agents/sdk-python) (maintained by AWS)!

<p><a href="https://linear.app/promptfooo/issue/ENG-1149/add-strands-agents-sdk-example">ENG-1149</a></p>
Repository: promptfoo/promptfoo
Issue: Github latest release not available anymore
Root cause: Configuration Error

Hello,
I was watching promptfoo releases using the [GH AP](https://docs.github.com/en/rest/releases/releases?apiVersion=2022-11-28) and since release [v0.119.13](https://github.com/promptfoo/promptfoo/releases/tag/promptfoo-v0.119.13) the AP returns a 404 on https://api.github.com/repos/promptfoo/promptfoo/releases/latest

Maybe something change on your side how you automate release publications?

**To Reproduce**

- not working anymore:
```
$ curl -I https://api.github.com/repos/promptfoo/promptfoo/releases/latest
HTTP/2 404
date: Fri, 28 Nov 2025 08:42:21 GMT
[...]
```

- working release on another project:
```
$ curl -I https://api.github.com/repos/projectdiscovery/httpx/releases/latest
HTTP/2 200
date: Fri, 28 Nov 2025 08:41:17 GMT
[...]
```

<p><a href="https://linear.app/promptfooo/issue/ENG-1152/giithub-releases-not-available-via-api-anymore">ENG-1152</a></p>
Repository: promptfoo/promptfoo
Issue: Type names and field names cannot be a keyword: 'pass'
Root cause: Validation Gap

Thanks to #638, I can now execute Python asserts with huge inputs.
However, the `GradingResult` class in Python has the following error:

```
ValueError: Type names and field names cannot be a keyword: 'pass'
```

Maybe renaming to other names e.g., `has_passed` or `passed` would resolve this issue.
Thank you.

I think this is just a documentation error, the return type is really just a `Dict[str, Any]` with a `"pass"` key.  I'll update the docs

```py
def get_assert(output, context) -> Union[bool, float, Dict[str, Any]]
    print('Prompt:', context['prompt'])
    print('Vars', context['vars']['topic']

    # Determine the result...
    result = test_output(output)

    # Here's an example GradingResult dict
    result = {
      'pass': True,
      'score': 0.6,
      'reason': 'Looks good to me',
    }
    return result
```

@typpo thanks for the comment.
But I don't think it's just a documentation error.
I get this message in the result columns from `npx promptfoo view`.
I didn't return `GradeResult` type in my grading Python script.

```
Error: Python script execution failed: Error: ValueError: Type names and field names cannot be a keyword: 'pass'

Error: Python script execution failed: Error: ValueError: Type names and field names cannot be a keyword: 'pass'
  at runAssertion(.../node_modules/promptfoo/dist/src/assertions.js:184:27)
  at process.processTicksAndRejections(node:internal/process/task_queues:95:5)
  at async .../node_modules/promptfoo/dist/src/assertions.js:83:24
```

What does your script look like? You should make sure that you are not
creating a class with a "pass" attribute.

On Mon, Apr 8, 2024, 7:00 AM Sangwoo Joh ***@***.***> wrote:

> @typpo <https://github.com/typpo> thanks for the comment.
> But I don't think it's just a documentation error.
> I get this message from npx promptfoo view result columns:
>
> Error: Python script execution failed: Error: ValueError: Type names and field names cannot be a keyword: 'pass'
>
> Error: Python script execution failed: Error: ValueError: Type names and field names cannot be a keyword: 'pass'
>   at runAssertion(.../node_modules/promptfoo/dist/src/assertions.js:184:27)
>   at process.processTicksAndRejections(node:internal/process/task_queues:95:5)
>   at async .../node_modules/promptfoo/dist/src/assertions.js:83:24
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/642#issuecomment-2042836451>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJSKMSDI2KM5JPS7MZ3Y4KPG7AVCNFSM6AAAAABF33YMR6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANBSHAZTMNBVGE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


@typpo Sorry, one of my scripts defined `GradeResult` class in it!! I've missed this.
After removing this declaration, I got the correct result.
Thank you so much!!
Repository: promptfoo/promptfoo
Issue: Show Result with Virtualization
Root cause: Unimplemented Feature Gap

When I have 10k+ eval results, `npx promptfoo view --yes` show a page that renders too long.
I found that promptfoo is using mui, and there is [Data Grid Virtualization](https://mui.com/x/react-data-grid/virtualization/) feature in mui, so this could improve the result page in an order of magnitude.

#649 adds pagination, which should solve the immediate problem of pages too large to render

@typpo thank you, that sounds wonderful!

Pagination released in [0.52.0](https://github.com/promptfoo/promptfoo/releases/tag/0.52.0)
Repository: promptfoo/promptfoo
Issue: Still Getting `E2BIG` Error
Root cause: Resource Mishandling

I thought #635 fixed this issue, but it didn't.
I'm still getting `E2BIG` errors in `promptfoo view` page.

```
FAILError: spawn E2BIG

Error: spawn E2BIG
    at ChildProcess.spawn (node:internal/child_process:420:11)
    at spawn (node:child_process:757:9)
    at new PythonShell (prompt-pioneer/node_modules/python-shell/index.js:130:55)
    at prompt-pioneer/node_modules/python-shell/index.js:252:27
    at new Promise (<anonymous>)
    at PythonShell.run (prompt-pioneer/node_modules/python-shell/index.js:251:16)
    at runPython (prompt-pioneer/node_modules/promptfoo/dist/src/python/wrapper.js:28:58)
    at PythonProvider.callApi (prompt-pioneer/node_modules/promptfoo/dist/src/providers/pythonCompletion.js:40:59)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Evaluator.runEval (prompt-pioneer/node_modules/promptfoo/dist/src/evaluator.js:248:30)
```

Seems like the implementation will have to be updated to send data through stdin

https://github.com/extrabacon/python-shell/issues/179
https://github.com/extrabacon/python-shell#exchanging-data-between-node-and-python

@typpo I have another workaround with a temporary json file, and if you don't mind can I make a pull request for this?

Yes, that would work too!  Definitely open to a PR @sangwoo-joh 

@typpo Thanks, I made a #660 ! It would be appreciated if you review this.
Repository: promptfoo/promptfoo
Issue: Support Cloudflare Workers AI
Root cause: Unimplemented Feature Gap

Leaving a ticket here for something I want to implement shortly: supporting cf workers AI.

They have beta support for uploading lora finetunes and could support very low latencies. So I think it's a great target to support in promptfoo.

Sounds good, the API looks nice and clean so hopefully not much of a lift!  

@CamdenClark any interest in helping me test my branch?

> @CamdenClark any interest in helping me test my branch?

Yeah I can this weekend!
Repository: promptfoo/promptfoo
Issue: Update new docker image on ghcr registry
Root cause: Configuration Error

the ghcr.io/promptfoo/promptfoo:latest points to this docker image (v0.119.11) https://github.com/promptfoo/promptfoo/pkgs/container/promptfoo 


<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1182/update-new-docker-image-on-ghcr-registry">ENG-1182</a></p>
Repository: promptfoo/promptfoo
Issue: Upgrade zod and zod-validation-error when possible
GitHub labels: dependencies

blocked

We are currently blocked on https://github.com/anthropics/claude-agent-sdk-typescript/issues/38, but I don't want to lose track. 

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1189/upgrade-zod-and-zod-validation-error-when-possible">ENG-1189</a></p>
Repository: promptfoo/promptfoo
Issue: Import YAML Preset For Redteamin
Root cause: Unimplemented Feature Gap

Every-time we need to fill the form for redteaming , so if we can have a feature to import YAML files and scan using the YAML file or we can save the yaml file for future use 




<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1196/import-yaml-preset-for-redteamin">ENG-1196</a></p>
Repository: promptfoo/promptfoo
Issue: Prompt file doesn't support multi-shot, multi-line prompts 
Root cause: Unimplemented Feature Gap

I would like to give a multi-shot prompt with multiple user/assistant messages.
For that, as the examples state, I use a JSON list of objects.
However, in this format, if I have newlines inside a prompt, the JSON isn't properly decoded (as officially, newlines should be encoded as \n in JSONs).

This makes it extremely uncomfortable to use, as copying prompts in/out of promptfoo requires a lot of manual work of transforming newlines to \n s.

I think there should be a way to write multi-line, multi-shot prompts in some non-JSON way.

Hi @liorn, can you please provide an example of a prompt file that is not behaving the way you want it to?

There are some examples of JSON prompts here https://github.com/promptfoo/promptfoo/tree/main/examples/multishot


Sure! I would like something like the following.

It doesn't work because JSONs shouldn't have newlines in them, and I get "unexpected token" errors.
I would love promptfoo to have some alternative way to provide multi-line, multi-shot prompts like the above.

```
[{"role": "system", "content": "This is a multiline prompt.
It has multiple lines in it.
Today, it doesn't work."},
{"role": "user", "content": "Another prompt,
which has several lines in it."}]```



Ah I see.  As you pointed out in your original post, that is not valid JSON.  I imagine most users won't want a tool that coerces malformed JSON to correct JSON, because then it won't work properly in their production application.

One potential solution is to add YAML support.  YAML has built-in support for multiline strings, and it maps directly to JSON.  For example:
```yaml
- role: system
  content: |
    This is a multiline prompt.
    It has multiple lines in it.
    Today, it doesn't work.
- role: user
  content: |
    Another prompt,
    which has several lines in it.
```

What do you think?

That would make a lot of sense. Thank you.

This will be in the next release.  Example usage here: https://github.com/promptfoo/promptfoo/tree/main/examples/openai-multiline-yaml

Thanks for the prompt fix (pun intended)!
Repository: promptfoo/promptfoo
Issue: Feature Request: Aggregate and/or Calculated Metrics
Root cause: Unimplemented Feature Gap

It would be fantastic to be able to aggregate and/or apply logic over Metrics in a custom way. 

For example, if you had a requirement that context be both `relevant` and `faithful`, it would be nice to be able to implement `context-relevant`, `context-faithful` , (`context-relevant` > n AND `context-faithful` > m), `context-relevant` * `context-faithful` et cetera. 

Thanks for the suggestion @thomascleberg.  Definitely interested in implementing this - will follow up once I open a PR

@thomascleberg WIP here: https://github.com/promptfoo/promptfoo/pull/670

This is now supported as of 0.53.0. Documentation here: https://promptfoo.dev/docs/configuration/expected-outputs/#creating-derived-metrics
Repository: promptfoo/promptfoo
Issue: [Feature] F1 score metric for tool calling evaluations
Root cause: Unimplemented Feature Gap

My use case requires a straightforward way to measure tool calling accuracy using F1 score, it would be great to have `promptfoo` support this.

When testing LLMs that call tools/functions, I'd like to know how well the model picks the right tools. For context, [F1 score](https://en.wikipedia.org/wiki/F-score) is a standard metric that combines precision (of the tools called, how many were correct?) and recall (of the expected tools, how many were called?) into a value between 0 and 1.

Something like:

```yaml
assert:
  - type: tool-call-f1
    value:
      - get_weather
    threshold: 0.8
```

This would compare the tools the model actually called against what I expected and return an F1 score.

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1227/feature-f1-score-metric-for-tool-calling-evaluations">ENG-1227</a></p>
Repository: promptfoo/promptfoo
Issue: Specify description for custom python providers, for display in web ui
Root cause: Documentation Deficiency

Hello,

At the moment, when using custom python script providers as follows:

```
providers:
  - id: 'python:../scripts/instructor_schema.py'
    config:
      llm_model: "gpt-4-0125-preview"
      evaluateOptions.maxConcurrency: 4
      evaluateOptions.repeat: 1
      evaluateOptions.delay: 500
      evaluateOptions.showProgressBar: true

  - id: 'python:../scripts/instructor_schema.py'
    config:
      llm_model: "gpt-3.5-turbo-0125"
      evaluateOptions.maxConcurrency: 4
      evaluateOptions.repeat: 1
      evaluateOptions.delay: 500
      evaluateOptions.showProgressBar: true
```

it doesn't appear to be possible to give each provider a useful descriptive name.  If, as above, running the same script with different `config`, it can be difficult to tell which column is for which provider.  Indeed, both columns are simple labelled `['{{ query }}']`.

Would it be possible to (i) give Python custom providers a custom name, and (ii) enable setting that custom name as column title in the web ui?

Many thanks for your attention!

Hey @alexclaydon,

I think what you're looking for is the `label` property:

```yaml
providers:
  - id: 'python:../scripts/instructor_schema.py'
    label: 'gpt-4-0125-preview via instructor_schema.py'
    config:
      llm_model: "gpt-4-0125-preview"
      evaluateOptions.maxConcurrency: 4
      evaluateOptions.repeat: 1
      evaluateOptions.delay: 500
      evaluateOptions.showProgressBar: true
  # ...
```

Let me know if that does the trick!

That works great - thank you very much!
Repository: promptfoo/promptfoo
Issue: Telemetry is enabled even when opted out

Steps to reproduce:
1) Run
```sh
mkdir empty_dir
cd empty_dir
docker run -it --rm -p 3000:3000 -v ./:/root/.promptfoo -e PROMPTFOO_DISABLE_TELEMETRY=1 -e PROMPTFOO_SHARE_STORE_TYPE=filesystem promptfoo-ui
```
2) Open the ui, put any config and run the evaluation

3) See the docker output
```
Writing environment variables to .env file...
Loaded environment variables:
NODE_VERSION=18.20.0
HOSTNAME=0.0.0.0
YARN_VERSION=1.22.19
SHLVL=2
PORT=3000
HOME=/root
PROMPTFOO_SHARE_STORE_TYPE=filesystem
TERM=xterm
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NEXT_TELEMETRY_DISABLED=1
PROMPTFOO_DISABLE_TELEMETRY=1
PWD=/app
Starting server...
- ready started server on 0.0.0.0:3000, url: http://localhost:3000
- info Loaded env from /app/.env
- info Loaded env from /app/.env
Listening on port 3000 url: http://0.0.0.0:3000
Using unknown OpenAI chat model: llamacpp
Anonymous telemetry is enabled. For more info, see https://www.promptfoo.dev/docs/configuration/telemetry
```


Should be fixed by #658 - thanks to @will-holley 

This appears to just be a logical error that causes the notice to display, but telemetry is not actually sent.