zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Cannot Use More Than One Replica in Kubernetes Deployment — Eval Jobs Not Shared Across Pods
Root cause: Architectural Constraint

Hi, we are trying to use a self-hosted version of Promptfoo in a Kubernetes environment, but we're encountering a significant limitation: we cannot run more than one replica of the application.
When deploying multiple replicas (2 or more), job-related state (evalJobs defined at src/server/routes/eval.ts, line 27) is stored in pod memory. This state is not shared between pods. As a result:
- If an evaluation job is created on pod-1, and a subsequent request (e.g., to check status) is handled by pod-2, it fails with an error like: "Job not found"

This essentially makes horizontal scaling impossible and also creates performance bottlenecks:
- Running just a single replica causes the application to hang during I/O operations.
- Parallel evaluation jobs become impractical, as the single-instance architecture locks up under load.

**Reproduction Steps:**
1. Deploy Promptfoo with 2+ replicas in Kubernetes.
2. Try to create an evaluation job.
3. Query the job status, which may be picked up by another pod
4. Observe the "Job not found" error.

**Expected Behavior**
Evaluation jobs should be accessible and consistent across all replicas.

**Observed Behavior**
Job data is stored in memory of the originating pod and is not shared with others, breaking job tracking logic in a multi-replica setup.

**Potential Solutions Considered**
We assume the in-memory storage of evalJobs might need to be replaced or supplemented by:
- A shared storage mechanism (Redis, DB, or filesystem).
- Central job queue/state store.

**Questions**
1. How can we make self-hosted Promptfoo support multiple replicas?
2. Is there a way to sync evalJobs across all pods?
3. How can we start a few evaluations simultaneously without blocking the app by I/O?

![Image](https://github.com/user-attachments/assets/d5967bc6-d9a0-40d6-ab0f-1e901647271e)

![Image](https://github.com/user-attachments/assets/957b1289-d3ba-4726-9343-7b22c95c099f)

![Image](https://github.com/user-attachments/assets/12b9e7b0-6d1b-4555-a7f2-5b6a34de373e)

Hi @mldangelo , could you help me solve this issue? 
Let me know if more details are needed. Thanks! 🙏

Hey @yurchik11,  thanks again for the chat earlier today and for writing this up. Promptfoo OSS keeps eval state in a single sqlite database that is stored on the local file system, so only one API pod works today. Multi-pod job queuing isn’t on the open-source roadmap right now, but it’s a perfect fit for our cloud / enterprise.  Closing this for now to keep the board tidy, and we can reopen if plans change. Appreciate the detailed report! 🙏
Repository: promptfoo/promptfoo
Issue: Help me  ths, how to start promptfoo
Root cause: Documentation Deficiency

background：
  I download the source code of promptfoo version 0.115.4.
  
    Because i don't know what language the promptfoo project is written, i tried the following: use "npm run build",then package it into dist folder, but i don't know how to run dist.

    I hope the students who know can help guide me.

My target:
  I hope to launch the project promptfoo-0.115.4.zip for download
  

Hi @wk1583241477, thanks for checking out Promptfoo! You don’t need to build from source. The published package already has everything compiled. You can run it directly with `npx promptfoo@latest init --example getting-started`

Or install globally

```
npm install -g promptfoo
promptfoo --version
```

If you really want to build from source
	1.	Make sure Node 18+ is installed.
	2.	Run `npm install`
	3.	Run `npm run build` to compile TypeScript, React UI, etc.
	4.	Execute the built CLI: `node dist/src/main.js --version`  


Docs if you need them:
	•	Installation: https://www.promptfoo.dev/docs/installation
	•	Getting started: https://www.promptfoo.dev/docs/getting-started

Closing this issue as it’s a usage question, but feel free to open a new one if anything is unclear after trying the recommended install. Welcome aboard!

Hi @mldangelo ,thanks for you review me message, because i need to modify the export excel funcation for source ,so i need to build from source and run it.
I think "4. Execute the built CLI: node dist/src/main.js --version" may help me. Thank you again for your reply.
And I want to ask if executing "2. Run npm install" and "3. Run npm run build to compile TypeScript, React UI, etc." is  done in the root directory.

And i would like to ask about the introduction of the "src" and "site" directories，ths
Repository: promptfoo/promptfoo
Issue: Simulated-user provider does not pass vars/context to custom provider calls
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I'm trying to use the `promptfoo:simulated-user` provider to simulate multi-turn conversations while routing responses through my custom Python provider. However, `vars` defined in the test (like `workflow_id`, or instructions) are not passed into the provider’s `context`. Normally, these are available in `context["vars"]` during single-turn tests, but they are missing entirely in multi-turn simulated-user flows. This prevents dynamic routing or payload construction based on test-specific context

**Describe the solution you'd like**
I’d like test-level `vars` to be included in the `context["vars"]` parameter when `call_api()` is invoked on a custom provider, even during simulated-user runs. This would make context handling consistent between single-turn and multi-turn evaluations, and allow dynamic values like `workflow_id` to be accessible throughout the conversation.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
```yaml
providers:
  - id: file://provider.py
    config:
      base_url: base_url

defaultTest:
  provider:
    id: promptfoo:simulated-user
    config:
      maxTurns: 5

tests:
  - vars:
      workflow_id: 123-workflow
      session_id: 123-session
      instructions: |
        You are booking a one-way flight from New York to Seattle on May 20.
```
In this setup, `workflow_id` is not available inside the `context` argument of `call_api()` when using `simulated-user`
Repository: promptfoo/promptfoo
Issue: Invalid option: a
Root cause: Validation Gap

I'm getting "Invalid option: a" error when running eval with model graded "factuality" assertion (btw docs are unambiguous about factuality, should I use `factuality` or `model-graded-factuality`?). I'm using default test provider  bedrock:completion:anthropic.claude-v2.

Thanks for the catch - #468 should resolve this issue.  It was happening because Claude is grading with a lowercase letter, but the code expects an uppercase letter.
Repository: promptfoo/promptfoo
Issue: Answer-relevance assertion underreports token usage
Root cause: Algorithmic Error

**Describe the bug**

The answer-relevance assertion only logs token usage from failed API calls during candidate question generation, ignoring tokens used in successful calls.

**To Reproduce**
Steps to reproduce the behavior:

1. Set up a promptfoo config with `answer-relevance` assertions:
```yaml
prompts:
  - "{{query}}"
providers:
  - openai:gpt-4o-mini
tests:
  - vars:
      query: "What are your return policies?"
    assert:
      - type: answer-relevance
        threshold: 0.7
```

2. Run `promptfoo eval`
3. Check token usage output
4. See suspiciously low token usage in grading

**System information:**

- Promptfoo version: 0.115.4

**Additional context**
Token accumulation only happens in the error handling blocks (`if (resp.error || !resp.output)`). Successful API responses don't result in token accumulation within the loop. 
https://github.com/promptfoo/promptfoo/blob/39bb0239641b64ed5c0561e7fe37fcaee464964d/src/matchers.ts#L898-L936


Thanks for the report, this should fix https://github.com/promptfoo/promptfoo/pull/4676

This should be fixed in the next version with https://github.com/promptfoo/promptfoo/pull/4677
Repository: promptfoo/promptfoo
Issue: Add bedrock:completion:anthropic.claude-v2.1
Root cause: Unimplemented Feature Gap

As in title

I don't think I have access to this model yet - what is the current behavior when you try to run it with promptfoo?

I think it should be added here: https://github.com/promptfoo/promptfoo/blob/bc83295b59c307b4764d14af6fd15ad2a78d8541/src/providers/bedrock.ts#L67

Currently I'm getting "TypeError: Cannot read properties of undefined (reading 'params') TypeError: Cannot read properties of undefined (reading 'params') at AwsBedrockCompletionProvider.callApi"

But this'll be more complicatede - claude v2.1 supports system prompts (no "Human:" at the beggining of the prompt). 

#470 solves the error at least - so you can use claude-2.1 without a system prompt.  

As for supporting system prompt, what do you think is the best way?  Some thoughts:
  - Completely remove the logic that prepends `Human:` and appends `Assistant:`.  This means that users will have to set custom prompts for Anthropic, but minimizes the amount of magic stuff that's happening
  - Support OpenAI-style `role`, `content` JSON objects
  - Some other approach?

Adding `Human:` `Assistant:` is similar to langchain - we could try to map langchain solution to promptfoo codebase. From the other side, I agree that this "magic" isn't something I like in langchain. And we have functionality that adds prefix/suffix to the prompt in promptfoo if I remember correctly. So probably good documentation without "magic stuff" sounds reasonable.

Closing this out as this format is now legacy with Claude 3
Repository: promptfoo/promptfoo
Issue: Bedrock anthropic Claude v2 - `llm-rubric produced malformed response:`
Root cause: External Dependency Breakage

If I understand correctly, grader response for llm-rubric should be json? Fun fact that I'm getting this error with 2 of 3 prompts used for evaluation - claude was trained on XML data, so it might not follow guidelines  

The Anthropic grader has been improved to use tools.  And Claude 3-3.5 is much better!  Let me know if this continues to be a problem.
Repository: promptfoo/promptfoo
Issue: About I run red team error
Root cause: Documentation Deficiency

Sorry，i don't have permission to reopen issue ，https://github.com/promptfoo/promptfoo/issues/4663。

About my second error,
I'm sorry for i didn't make myself clear。
Because i run  red team  on page，it always error，i look the code，error code is my second error。
And error message is "Error running redteam: Error: Unable to proceed with test generation: Network error: Request timed out after 5000 ms
URL: https://api.promptfoo.app/health
Please check your API configuration or try again later.
[POST /providers/test] Error calling agent helper
error: fetch failed
providerOptions: {"id":"http","label":"111","config":{"useHttps":false,"url":"http://localhost:11434/api/chat","method":"POST","headers":{"Content-Type":"application/json"},"body":"{\"model\": \"qwen3:0.6b\", \"prompt\": \"{{prompt}}\",\"stream\":false}","validateStatus":"","transformResponse":"(json,text) => json.response || text"}}".


And i run "Option 2: Run Directly in Browser", error message is 
<img width="963" alt="Image" src="https://github.com/user-attachments/assets/2cf25f21-b012-45d1-8990-9d81e6efe4f9" />

Does your machine have access to `api.promptfoo.app`

Can you run `curl https://api.promptfoo.app/health`?

Yes, i can run it.

<img width="462" alt="Image" src="https://github.com/user-attachments/assets/2828a00d-33c6-4f2e-8d7b-b36e565c48d5" />

<img width="617" alt="Image" src="https://github.com/user-attachments/assets/92b42e4a-df7b-482a-aa7a-fa62ec40d620" />

But i think my network machine  accessing  api.promptfoo.app is too slow， it may be the reason for red team error.
If my machine don't accessing  api.promptfoo.app, can i run red team success？
Can i create https://api.promptfoo.app/health and https://api.promptfoo.app/providers/test？
Make the red team run successfully.
Or do i have to do anything else？
Thanks！


Can I choose not to access the domain 'apipromptfoo.app'? 
Because our company requires that external websites cannot be accessed, can I localize and deploy the functionality on this domain name? 
Or how can I deploy the API. promptfoo. app feature locally?
Thanks！

Hi @wk1583241477,

Thanks for reporting this - I understand the frustration with company network restrictions blocking access to api.promptfoo.app.

Good news: you can run red team offline without accessing any external domains. The offline mode is working correctly.

To run red team without accessing api.promptfoo.app:

```bash
export PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true
promptfoo redteam run
```

This will generate test cases locally without any external API calls. Please pull the latest version and give it a try.

See the [Red Team Configuration documentation](https://www.promptfoo.dev/docs/red-team/configuration) for more details on offline mode.

If you encounter any issues, feel free to comment or re-open.

Thanks for your patience!
Repository: promptfoo/promptfoo
Issue: Can't load multiple scenarios using a star wildcard character
Root cause: Validation Gap

**Describe the bug**
If I try to load all the scenarios from a folder with a wildcard pattern promptfoo breaks with a 'Can't find file' error. If I specify the files one by one, it works correctly.

**To Reproduce**
1. Add scenarios from location with a wildcard e.g.   `- file://scenarios/*.yaml`
2. Run `npx promptfoo@latest eval`

**Expected behavior**
Scenarios are loaded successfully and the evaluation runs.

**Screenshots**
config file:

```
description: 'Reports 1-5, 30-34'

scenarios:
  - file://scenarios/*.yaml
      
prompts:
  - file://prompts/experiments/prompt_01072025_1.txt

providers:
  - id: azure:chat:gpt-4o
    config:
      apiHost: 'https://***.openai.azure.com/'
      temperature: 0.1
  - id: azure:chat:gpt-4.1
    config:
      apiHost: 'https://***.openai.azure.com/'
      temperature: 0.1
  - id: vertex:gemini-2.0-flash-001
    config:
      region: europe-west4
      projectId: ***
```
Error:
```
PS E:\{path}\reports1_2_3_4_5_31_32_33_34_45> npx promptfoo@latest eval
C:\Users\adm-bartosz\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\util\file.js:93
        throw new Error(`File does not exist: ${finalPath}`);
              ^

Error: File does not exist: E:\{path}\reports1_2_3_4_5_31_32_33_34_45\scenarios\*.yaml
    at maybeLoadFromExternalFile (C:\Users\adm-bartosz\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\util\file.js:93:15)
    at C:\Users\adm-bartosz\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\util\file.js:79:29
    at Array.map (<anonymous>)
    at maybeLoadFromExternalFile (C:\Users\adm-bartosz\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\util\file.js:78:25)
    at runEvaluation (C:\Users\adm-bartosz\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\commands\eval.js:225:79)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
    at async doEval (C:\Users\adm-bartosz\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\commands\eval.js:567:12)

Node.js v22.16.0
```

**System information:**
```
{
  "version": "0.116.0",
  "platform": {
    "os": "win32",
    "release": "10.0.26100",
    "arch": "x64",
    "nodeVersion": "v22.16.0"
  },
  "env": {},
  "configInfo": {
    "defaultConfigPath": "E:\\{path}\\reports1_2_3_4_5_31_32_33_34_45\\promptfooconfig.yaml",
    "configExists": true,
    "configContent": "Error loading config: Error: File does not exist: E:\\{path}\\reports1_2_3_4_5_31_32_33_34_45\\scenarios\\*.yaml"
  }
}
```

- Promptfoo version:
0.116.0

**Additional context**
Add any other context about the problem here.
Repository: promptfoo/promptfoo
Issue: Is doRedteamRun supposed to be exposed in the package export?
Root cause: Documentation Deficiency

Hello 👋, I was trying to use `doRedteamRun` programmatically, but it looks like it's not exposed via the main package export, or at least I couldn't import it like this:

```typescript
import { doRedteamRun } from 'promptfoo';
```

is this function intentionally not exported? If not, would it be possible to add it to the main exports?

Thanks

@ossamakhiar This will be in the next release! Thanks for the suggestion!

@ossamakhiar this will be exported under the redteam namespace as { generate } in the next release. Note that this is mostly an internal function and the API here may change over time.  See https://github.com/promptfoo/promptfoo/pull/4793 

I'll update the issue when we release it. Thanks! 

Hi @ossamakhiar, great news! 🎉

The `doRedteamRun` function is now exposed under the `redteam` namespace. You can import and use it like this:

```typescript
import { redteam } from 'promptfoo';

// Use the function
await redteam.run(config, options);
```

Or if you prefer destructuring:

```typescript
import { redteam } from 'promptfoo';
const { run: doRedteamRun } = redteam;
```

The redteam namespace also includes other useful exports:
- `redteam.generate` - Generate red team tests
- `redteam.Plugins` - Available plugins
- `redteam.Strategies` - Available strategies
- `redteam.Graders` - Grading functions
- `redteam.Extractors` - Entity extraction utilities

This was added to support programmatic use cases like yours. Thanks for the suggestion - it helped us improve the API! 

Let us know if you run into any issues using it programmatically.
Repository: promptfoo/promptfoo
Issue: Allow output into javascript (and in future python maybe)
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
In our use case we have many promptfoo suites.
Sometimes when running one some of them have many failures due to TPM issues or tokens expiring.
We then have to open `promptfoo view` to dind these and know to run again or update the token

**Describe the solution you'd like**
A simple solution would be to be able to set the output (one that is normally sent to csv, json ect file) to a javascript function (change in promptfoo) and then we can set this function to collate the error metadata and output it to console.

I also think that this could streamline data analysis in other cases for other people

**Describe alternatives you've considered**
We could add steps we manually run after each run but this is cumbersome to do.
We could create scripts that do the run, write to a file and run the analysis on top but this seems ineffecent and also cumbersom given multiple suites, it would be nice to modify the promptfooconfig.yaml and then have the info each time.

Happy to implement this if I get guidance on how we want this to fit in
Repository: promptfoo/promptfoo
Issue: Error: OPENAI_API_KEY is not set using docker compose
Root cause: Documentation Deficiency

As per the docs I have a `docker-compose.yml` with the `OPENAI_API_KEY` set as an environment variable:
```
services:
    dc_promptfoo: # Consistent service and container name
        image: ghcr.io/promptfoo/promptfoo:latest # Or pin to a specific version tag
        ports:
            - '3000:3000' # Map host port 3000 to container port 3000
        volumes:
            - promptfoo_data:/home/promptfoo/.promptfoo
        environment:
            # Optional: Adjust chunk size for large evals (See Troubleshooting)
            - PROMPTFOO_SHARE_CHUNK_SIZE=10
            # Add other necessary environment variables (e.g., API keys)
            - OPENAI_API_KEY=sk-xxxx
            # Example: Google API Key
            # - GOOGLE_API_KEY=your_google_key_here
volumes:
    promptfoo_data:
        driver: local
```
I have installed Promptfoo as a local node client using `npx promptfoo@latest` via `nvm` using `lts/hydrogen`

I have set up the default `promptfooconfig.yaml` using `npx promptfoo@latest init`

I have added the correct sharing URLs to the config:
```
sharing:
    apiBaseUrl: http://localhost:3000
    appBaseUrl: http://localhost:3000
```
But when I run `promptfoo eval` it will run but all tests return `Error: OPENAI_API_KEY is not set`

I have checked the environment variable is correct in the container.

If I run `OPENAI_API_KEY=sk-xxx promptfoo eval` it will work

Is this an issue with me using `nvm` or why is the env variable in the container not being used?

Promptfoo doesn't share secrets from the view server to the cli.  You’re running `promptfoo eval` from your host shell (the node version you installed via nvm), so it can’t see the OPENAI_API_KEY that’s defined inside the Docker container. You have two options:

1. Export `OPENAI_API_KEY=sk-xxx` in the shell where you run promptfoo eval.
2. Or, execute promptfoo eval inside the container using docker exec so it inherits the container’s environment.

Either approach will give the CLI access to the API key and resolve the error. Let me know if this helps or if you have other questions. 

Closing for now. Feel free to re-open or throw some time on my calendar if you need help!
Repository: promptfoo/promptfoo
Issue: Customizing the grader provider doesn't work
Root cause: Unimplemented Feature Gap

I've been trying to [customize the grading provider](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded#overriding-the-llm-grader), but it doesn't seem to be working. Am I doing something wrong? Or is this not working as intended.

I created an [example here, that tries to set a grading provider a couple ways](https://github.com/adrianlyjak/promptfoo/commit/82febf2b7450fd13e78d2367d956c4fc41f42f69).

However the verbose output shows that its using gpt-4:

```
$ promptfoo eval --verbose
...

Calling OpenAI API: {"model":"gpt-4-0125-preview","messages":[{"role":"system","content":"You are comparing multiple pieces of text to see which best fits the following criteria: choose the funniest tweet\n\nHere are the pieces of text:\n\n\n<Text index=\"0\">\nJust discovered that bananas are a great source of potassium and vitamin B6! 🍌 Not only are they delicious, but they're also a powerhouse of nutrients. #HealthyEating #BananaLove\n</Text>\n\n<Text index=\"1\">\n\"Just found out bananas are a great source of potassium. Guess I'm one step closer to being a fruit ninja. #BananaBattles\"\n</Text>\n\n\nOutput the index of the text that best fits the criteria. You must output a single integer."}],"seed":0,"max_tokens":1024,"temperature":0,"top_p":1,"presence_penalty":0,"frequency_penalty":0}
...
Model-graded comparison #1 of 2 complete
Calling OpenAI API: {"model":"gpt-4-0125-preview","messages":[{"role":"system","content":"You are comparing multiple pieces of text to see which best fits the following criteria: choose the tweet that contains the most facts\n\nHere are the pieces of text:\n\n\n<Text index=\"0\">\nExploring the concrete jungle where dreams are made of - #NYC. The city's energy is infectious, its diversity inspiring, and its skyline breathtaking. Truly, there's no place like New York! 🗽🍎 #ILoveNY #BigApple #CityThatNeverSleeps\n</Text>\n\n<Text index=\"1\">\n\"Just saw a rat on the subway carrying a whole slice of pizza. Even the wildlife in NYC hustles harder. #OnlyInNYC\"\n</Text>\n\n\nOutput the index of the text that best fits the criteria. You must output a single integer."}],"seed":0,"max_tokens":1024,"temperature":0,"top_p":1,"presence_penalty":0,"frequency_penalty":0}
```

If you think this is a bug, I can try taking a stab at a fix

Thank you for providing a repro example!  It made it pretty easy to troubleshoot, so I just wound up making a PR.

`select-best` doesn't support provider overrides right now because it is a bit of a special case (output comparisons run after all LLM outputs have been received, as opposed to on-the-fly like other assertions).  

#478 fixes this issue and will be included in the next release.

Awesome! Looking forward to the next releases