zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: llm-rubric assertion does not use script output for rubric text
Root cause: Algorithmic Error

**Describe the bug**
When using a file-based JS rubric (e.g., `value: file://index.js:rubric`) in a `llm-rubric` assertion, the output of the script is logged but not used as the rubric text in the LLM request. Instead, the rubric sent to the LLM is the literal script path string (e.g., `<Rubric>\nfile://index.js:rubric\n</Rubric>`).

**Expected behavior**
The rubric sent to the LLM should be the result of the evaluated JavaScript function, not the reference string. For example, if the JS evaluates to `abc`, the API request should include `<Rubric>\nabc\n</Rubric>`.

**How to reproduce**
- Use a test assertion like:
  ```yaml
  - type: llm-rubric
    value: file://index.js:rubric
    threshold: 0.9
  ```
- Observe logs showing the JS script output (e.g., `abc`), but the actual API request to the LLM contains `<Rubric>\nfile://index.js:rubric\n</Rubric>` instead of `<Rubric>\nabc\n</Rubric>`.

```
[logger.js:324] Javascript script /app/prompts/change_type__test/index.js output: abc
```

**Where the bug is**
In `handleLlmRubric`, only `renderedValue` is used for the rubric text. The resolved script output (`valueFromScript`) should be used if available.

**Suggested fix**
Update `handleLlmRubric` and its caller to use `valueFromScript` if present when assembling the LLM request content for rubric grading. E.g.:
```typescript
return matchesLlmRubric(
  valueFromScript ?? renderedValue ?? '',
  ...
);
```
And ensure callers pass `valueFromScript` in the handler params.

---
Reporter: @juniorb2ss
Repository: promptfoo/promptfoo
Issue: Multiple prompts with each mapped to separate set of images & test cases/assertions
Root cause: Documentation Deficiency

I have a vision QA use case with multiple prompts and each prompt has separate set of images & test cases/assertions. I was looking at recommended project structure as mentioned [here](https://www.promptfoo.dev/docs/configuration/test-cases/) 

project/
├── promptfooconfig.yaml
├── prompts/
│   └── main_prompt.txt
└── tests/
    ├── basic_functionality.csv
    ├── edge_cases.yaml
    └── regression_tests.json

It is not clear how to support multiple prompts and map tests to specific prompts ? Can someone please provide an example here. Creating separate project(promptfooconfig.yaml) per prompt seems overkill to me ?

Hi @khatwaniNikhil!

Great question - this is a common pattern for vision QA workflows. By default, promptfoo creates a matrix of all combinations (tests × prompts × providers), so every test runs against every prompt. There isn't a built-in way to map specific tests to specific prompts directly.

For your use case with different prompts analyzing different types of images, separate configs is actually the cleanest solution.

```
project/
├── document-clarity/
│   ├── promptfooconfig.yaml
│   ├── prompt.json
│   └── tests.yaml
└── chart-analysis/
    ├── promptfooconfig.yaml
    ├── prompt.json
    └── tests.yaml
```

Run them with:
```bash
promptfoo eval -c document-clarity/promptfooconfig.yaml
promptfoo eval -c chart-analysis/promptfooconfig.yaml
```

Each prompt represents a distinct eval task with its own success criteria - keeping them separate makes your results clearer and configs easier to maintain.

### Relevant Examples

- [`examples/openai-vision`](https://github.com/promptfoo/promptfoo/tree/main/examples/openai-vision) - Basic vision setup
- [`examples/image-classification`](https://github.com/promptfoo/promptfoo/tree/main/examples/image-classification) - Vision with structured output

Hope this helps! Let me know if you have any other questions.
Repository: promptfoo/promptfoo
Issue: promptfoo:simulated-user provider does not check for errors after calls to sendMessageToAgent()
Root cause: Validation Gap

**Describe the bug**

Hello,

The `promptfoo:simulated-user` provider does not check for errors after calls to  `sendMessageToAgent` in `promptfoo/dist/src/providers/simulatedUser.js`.  Notice how `sendMessageToUser` **does** perform the error check.

As a result, if an error occurs with `sendMessageToAgent()`, `promptfoo eval` will report back success (Because it's not returning an error as expected). 

We can observe two things with our screenshot below:
1. The value of **Assistant** is empty for the failed run (Invalid model used)
2. The results show 2 **Successful** runs (Expect one Error run)

<img width="856" height="370" alt="Image" src="https://github.com/user-attachments/assets/db8d2ef3-2061-4a57-b86d-f4f25a29ff08" />

**To Reproduce**

You'll notice the following config uses a model value that is **valid** and should work.  One that's **invalid**, which should fail, forcing an error.

**promptfooconfig.yaml**
```
description: "Minimal Test (Python) - Demonstrates simulated-user. One should Pass, one should Error"

prompts:
  - You are a helpful assistant.

providers:
  # Provider 1: Invalid model - should FAIL
  - id: 'file://minimal_provider.js'
    label: 'Invalid Model (Should FAIL)'
    config:
      model: 'anthropic.claude-INVALID-MODEL-NAME'
      region: 'us-east-1'

  # Provider 2: Valid model - should PASS
  - id: 'file://minimal_provider.js'
    label: 'Valid Model (Should PASS)'
    config:
      model: 'anthropic.claude-3-sonnet-20240229-v1:0'
      region: 'us-east-1'

defaultTest:
  provider:
    id: 'promptfoo:simulated-user'
    config:
      maxTurns: 1

tests:
  - vars:
      instructions: |
        Ask: What is 2+2?
```

**To Reproduce**

Create a simple custom provider
1.  Create a simple `minimal_provider.js`.  Want a working provider, as we can force an error by passing in an invalid model with promptfoo config (Example shown above)
2. `npx promptfoo eval`

**Expected behavior**
Expect `promptfoo eval` to report an error back to the user (See screenshot below). 

We see now we get: 
1. A Error displayed as expected. 
2. One Successful run and one Error run as expected.  

**Expected Results Screenshots**

<img width="798" height="386" alt="Image" src="https://github.com/user-attachments/assets/7d9bcbf8-6c5a-4abf-b51a-607ace3d1dac" />

The successful screenshot above is after applying the following fix in a couple places in `promptfoo/dist/src/providers/simulatedUser.js`

```
            agentResponse = await this.sendMessageToAgent(messages, context.originalProvider, context);
            // Check for errors from agent response
            if (agentResponse.error) {
                return {
                    error: agentResponse.error,
                    tokenUsage,
                };
            }
```

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version:
**0.119.6**

**Additional context**
Add any other context about the problem here.


Thank you!
Repository: promptfoo/promptfoo
Issue: Promptfoo on self-deployed machine becomes unreachable after uploading a large json file
Root cause: Resource Mishandling

- Sharing small files to the promptfoo machine works well
- I am using an on disk memory store
- When I share a large evaluation the promptfoo interface becomes unreachable
	- I cannot share the results because they contain private data but the json is an evaluation of 3 models over 253 questions.


PS: I am not sure it is the fact the file is large that produces the error.

Here is the log trace i have when I upload the large json file:
```

Using Memory store
214Storing eval result with id f:093d306e-eda2-40c2-bfab-35c5f0041330
215Storing eval result with id f:4190a09a-a9d4-4423-b6b4-50877006bf45
216Failed to read eval Error: Could not find eval
217at getDataForId (/app/.next/server/app/api/eval/[id]/route.js:283:19)
218at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
219at async GET (/app/.next/server/app/api/eval/[id]/route.js:306:22)
220at async /app/.next/server/chunks/2778.js:5604:37
221Failed to fetch data Error: Could not find eval
222at getDataForId (/app/.next/server/app/api/eval/[id]/route.js:283:19)
223at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
224at async GET (/app/.next/server/app/api/eval/[id]/route.js:306:22)
225at async /app/.next/server/chunks/2778.js:5604:37
226TypeError: fetch failed
227at Object.fetch (node:internal/deps/undici/undici:11731:11)
228at async invokeRequest (/app/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:21:12)
229at async invokeRender (/app/node_modules/next/dist/server/lib/router-server.js:229:29)
230at async handleRequest (/app/node_modules/next/dist/server/lib/router-server.js:422:24)
231at async requestHandler (/app/node_modules/next/dist/server/lib/router-server.js:439:13) {
232cause: SocketError: other side closed
233at Socket.onSocketEnd (/app/node_modules/next/dist/compiled/undici/index.js:1:63301)
234at Socket.emit (node:events:529:35)
235at endReadableNT (node:internal/streams/readable:1400:12)
236at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
237code: 'UND_ERR_SOCKET',
238socket: {
239localAddress: '127.0.0.1',
240localPort: 56806,
241remoteAddress: '127.0.0.1',
242remotePort: 44815,
243remoteFamily: 'IPv4',
244timeout: undefined,
245bytesWritten: 14245943,
246bytesRead: 144
247}
248}
249}
```

Interesting - how large is this file?

Hi @NoAchache, thank you for reporting this issue back in April 2024.

Since this was filed, we've made significant improvements to our sharing and upload functionality, including better handling of large evaluation files and improved error handling throughout the sharing flow.

Given the age of this issue and the changes we've made, I'm going to close this. If you or anyone else encounters similar problems with large file uploads, please feel free to open a new issue with details about your setup and we'll be happy to investigate.

Thanks again for taking the time to report this!
Repository: promptfoo/promptfoo
Issue: MCP Server: get_evaluation_details rejects eval IDs returned by list_evaluations due to regex validation mismatch
Root cause: Validation Gap

## Bug Description

The MCP server's `get_evaluation_details` tool cannot fetch details for evaluations listed by `list_evaluations` due to an eval ID format validation mismatch.

## Steps to Reproduce

1. Call `list_evaluations` via the Promptfoo MCP server
2. Get an eval ID from the results (e.g., `eval-8h1-2025-11-15T14:17:18`)
3. Call `get_evaluation_details` with that ID
4. Receive validation error: "Invalid eval ID format"

## Expected Behavior

`get_evaluation_details` should accept eval IDs returned by `list_evaluations`.

## Actual Behavior

The validation fails with error:
```json
{
  "validation": "regex",
  "code": "invalid_string",
  "message": "Invalid eval ID format",
  "path": ["id"]
}
```

## Root Cause

- `list_evaluations` returns eval IDs with **colons** in timestamps: `eval-8h1-2025-11-15T14:17:18`
- `get_evaluation_details` validates IDs with regex pattern `^[a-zA-Z0-9_-]+$` (no colons allowed)

## Suggested Fix

Either:
1. Update the `get_evaluation_details` regex to allow colons: `^[a-zA-Z0-9_:-]+$`
2. Or normalize eval IDs in `list_evaluations` to replace colons with hyphens or remove them

## Environment

- Promptfoo MCP Server (latest version)
- Using Claude Code IDE with MCP integration

Ha! @moltar you were quicker. I just hit the same bug. @mldangelo , how are things, is this release-worthy? :-) 


Thanks for catching and reporting this, @moltar and @mpasternak. The regex on get_evaluation_details now accepts the IDs returned by list_evaluations, and there are tests in place to cover this path. The fix is on main and included in the latest release, so please pull the latest version / upgrade Promptfoo and let me know if you still hit any issues.

If you run into more MCP bugs or have ideas for things you would like to see in the MCP server, please keep filing issues or feature requests. Feedback here is super helpful, and if you are doing heavier MCP integrations and want to talk through use cases, feel free to grab time on my calendar: https://cal.com/michael-dangelo/30min.

Thank you 🙏
Repository: promptfoo/promptfoo
Issue: Using python script assertions file path is not relative to config file
Root cause: Configuration Error

I am trying to maintain multiple prompt evaluations in separate folders. Each folder contains promptfoo config files along with python scripts that are used for prompts, variables, and assertions. 

When specifying the file path of python scripts for prompts and variables, the path relative to the config file is used. However, when the file path for python assertions is specified, it is relative to where the promptfoo eval is used. This behavior breaks the tests if the evaluation is not always run from a specific relative path to the python assertion scripts.

It would be helpful if the file path for python assertion scripts is relative to the config file specifying the path and not where the evaluation is run from.

Thanks in advance



I am not very familiar with JS but maybe it's related to using 
`runPythonCode` (https://github.com/promptfoo/promptfoo/blob/main/src/assertions.ts#L31) which is used only there
Instead of 
`runPython` (e.g. https://github.com/promptfoo/promptfoo/blob/main/src/prompts.ts#L10) which is used in other places I looked into.

Thanks for flagging this.  The problem was that `basePath` was not plumbed through to the assertions logic - something that I've been putting off a while.  I've got a fix here https://github.com/promptfoo/promptfoo/pull/624

That's awesome :) thanks for the speedy response and fix

Change is released in [0.51.0](https://github.com/promptfoo/promptfoo/releases/tag/0.51.0)
Repository: promptfoo/promptfoo
Issue: defaultTest property gets deleted from .yaml file when it is edited through the web ui
Root cause: Configuration Error

https://promptfoo.dev/docs/configuration/guide/#default-test-cases
when I try to add this property to the config file in the UI via the "edit yaml" button, it just disappears after clicking "save"

Thanks for catching, fixed in #629 

Thanks for a quick fix! I wanted to try multi-attempt evaluations, but seems like "evaluateOptions" also disappears from the config if it is edited in the UI  https://promptfoo.dev/docs/configuration/reference#config 

added this one too :+1: 

Change is released in [0.51.0](https://github.com/promptfoo/promptfoo/releases/tag/0.51.0)

Hi, I checked out the updated version and these options are indeed in the UI, but have no effect and are ignored during the evalutation.  

> Hi, I checked out the updated version and these options are indeed in the UI, but have no effect and are ignored during the evalutation.

#829
Repository: promptfoo/promptfoo
Issue: How to leave `{{x}}` in prompt template when `x` is not a variable?
Root cause: Documentation Deficiency

For instance, if I have a prompt template like the following:

```
Please generate {{language}} for {{target}}
```

And I only have `language` in my tests.
I want the final prompt look like

```
Please generate French for {{target}}
```

but currently, it gives me

```
Please generate French for
```

How can I leave `{{x}}` like this?

In nunjucks templates, you can put a string literal inside of double curly braces.  For example, this will do what you want:
```
prompts:
  - "Please generate {{language}} for {{'{{target}}'}}"
```

Thank you very much!!
Repository: promptfoo/promptfoo
Issue: Local Models on Promptfoo
Root cause: Documentation Deficiency

how can we make use of promptfoo on local llm models on my machine?

It depends what you're using to run local models.  See instructions on [ollama](https://promptfoo.dev/docs/providers/ollama/) and [LocalAI](https://promptfoo.dev/docs/providers/localai), for example.

Closing this out.  Let me know if you have any other questions!
Repository: promptfoo/promptfoo
Issue: Env var LITELLM_API_KEY no longer picked up by the latest promptfoo version
Root cause: Configuration Error

Here is a minimal example to reproduce the problem:

`promptfooconfig.yaml`:
```yaml
prompts:
  - "Write a tweet about {{topic}}"
providers:
  - id: litellm:gpt-4.1-mini
    config:
      apiBaseUrl: http://localhost:4000
tests:
  - vars:
      topic: bananas
```

Running

```bash
export LITELLM_API_KEY=...
promptfoo eval
```

with promptfoo version `0.119.8` works fine.

But when using version `0.119.9`, I get the following error:

```
Error: API key is not set. Set the LITELLM_API_KEY environment variable or add `apiKey` to the provider config.
    at OpenAiChatCompletionProvider.callApi (/home/tobias/.nvm/versions/node/v22.12.0/lib/node_modules/promptfoo/dist/src/providers/openai/chat.js:269:19)
    at LiteLLMProvider.callApi (/home/tobias/.nvm/versions/node/v22.12.0/lib/node_modules/promptfoo/dist/src/providers/litellm.js:43:30)
    at runEval (/home/tobias/.nvm/versions/node/v22.12.0/lib/node_modules/promptfoo/dist/src/evaluator.js:341:49)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async processEvalStep (/home/tobias/.nvm/versions/node/v22.12.0/lib/node_modules/promptfoo/dist/src/evaluator.js:982:26)
    at async /home/tobias/.nvm/versions/node/v22.12.0/lib/node_modules/promptfoo/dist/src/evaluator.js:1285:17Show less
```
Repository: promptfoo/promptfoo
Issue: Error: spawn python ENOENT when user try to run promptfoo with python:provider
Root cause: Environment Incompatibility

Hi! 
I try to run sample code from examples/python-provider/promptfooconfig.yaml and I get an error 
```
node:events:492
      throw er; // Unhandled 'error' event
      ^

Error: spawn python ENOENT
    at ChildProcess._handle.onexit (node:internal/child_process:286:19)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on PythonShell instance at:
    at ChildProcess.<anonymous> (/Users/ogorbachenko/.npm/_npx/81bbc6515d992ace/node_modules/python-shell/index.js:180:18)
    at ChildProcess.emit (node:events:514:28)
    at ChildProcess._handle.onexit (node:internal/child_process:292:12)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'spawn python',
  path: 'python',
  spawnargs: [
    '/Users/name/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/python/wrapper.py',
    '/Users/name/Projects/dataQA/promptfoo_samples/examples/python-provider/provider.py',
    'call_api',
    '"Write a tweet about bananas"',
    '{"id":"python:provider.py","config":{"someOption":"foobar","basePath":""}}',
    '{"vars":{"topic":"bananas"}}'
  ]
}

Node.js v20.5.1

```
promptfoo version 0.50.1

Any idea why this is happening?

This means that node isn't able to find the Python executable.  By default, this is just `python`.  Ensure that python is installed on your system and that you're in a virtualenv (or whatever is necessary to locate the executable).

If desired, you can manually override the python binary path using the environment variable `PROMPTFOO_PYTHON`.  For example:
```
PROMPTFOO_PYTHON=venv/bin/python3.9 npx promptfoo@latest eval
```

Thanks!!
Repository: promptfoo/promptfoo
Issue: `eval` successfully done but error code is not 0
Root cause: Documentation Deficiency

I ran `npx promptfoo eval` and I can see the result with `npx prompt view`, but `npx promptfoo eval` itself returned `100` not `0` (checked by `echo $?`). What is the reason for this?

The process exits with status code 100 if there are any failures in the tests. This was intentionally added in #335, useful for CI

Thanks for the clarification!
Repository: promptfoo/promptfoo
Issue: Assert `FAIL` due to `Error: spawn E2BIG`
Root cause: Resource Mishandling

It looks like the context and model output I used for my experiments exceeded `ARG_MAX`. I'm currently asserting using an external Python script, and I'm getting this error for some tests:
```
Error: spawn E2BIG
    at ChildProcess.spawn (node:internal/child_process:420:11)
    at spawn (node:child_process:757:9)
    at executive (node:child_process:348:17)
    at pythonScriptOutput (node_modules/promptfoo/dist/src/assertions.js:181:50)
    at new Promise (<anonymous>)
    at runAssertion (node_modules/promptfoo/dist/src/assertions.js:180:50)
    at node_modules/promptfoo/dist/src/assertions.js:82:30
    at node_modules/async/dist/async.js:151:38
    at replenish (node_modules/async/dist/async.js:448:21)
    at node_modules/async/dist/async.js:453:13
```

Is it possible to make the Python assert read a file instead of the command line argument?

This is probably fixed by #638, which runs invokes python functions more elegantly instead of the current approach, which executes directly through the shell.

@typpo yes, I think that should fix this too, and maybe that fix will be released shortly?
Thanks as always.

Aiming to release this tomorrow morning :)

Change is released in [0.51.0](https://github.com/promptfoo/promptfoo/releases/tag/0.51.0)

Many thanks!!!!
Repository: promptfoo/promptfoo
Issue: Test generation hangs when run from CI/CD
Root cause: Resource Mishandling

**Describe the bug**
During test generation, promptfoo appears to sometimes(?) hang when run from inside a CI/CD pipeline (we're using Bitbucket pipelines). It does not hang when run locally.

We first observed this behaviour around v0.119.0, prior to that we had no issues with hanging pipelines.

**To Reproduce**

1. From a CI/CD pipeline, run a test generation command such as `npx --yes promptfoo@latest redteam generate -c promptfoo-configs/beavertails-config.yaml -o promptfoo-configs/aisafety-beavertails-redteam-input.yaml`
2. The stdout will show that test generation completed ("Wrote n cases... Run promptfoo redteam eval... to run the red team!"), but then that command just stalls and never exits

We experienced the same issue with different config files, so unsure that it's an issue with the config file.

The only way we've gotten it to terminate is by wrapping the command in a `timeout`.

There are also no error or debug logs. We've set PROMPTFOO_DISABLE_DEBUG_LOG and PROMPTFOO_DISABLE_ERROR_LOG to 0.

**Expected behavior**
Command exits on completion of test generation.

**System information:**

- Promptfoo version: > 0.118.17

**Additional context**
Unsure if this is somehow related to https://github.com/promptfoo/promptfoo/issues/5893 (last comment suggests this is not yet fixed even though the issue is closed)

<p><a href="https://linear.app/promptfooo/issue/ENG-1095/test-generation-hangs-when-run-from-cicd">ENG-1095</a></p>
Repository: promptfoo/promptfoo
Issue: Selectively Hide `Variables` Column in View
Root cause: Documentation Deficiency

I have a huge `tests.csv` with more than 20 columns. When I tested with these settings, `npx promptfoo view` showed too many columns and I could not control this section.
It would be really helpful if I could hide some of the variable columns selectively.
Oh, and I just want to say thank you so much for creating such a great tool, I really appreciate it, thank you.

Oh, I can select specific columns in Columns box. I didn't notice that. Thank you!