zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Problem with my evals
Root cause: Environment Incompatibility

I ran my eval on the command line and all worked perfectly, also i have the output file, but when i run promptfoo view i don´t see any eval 

![Image](https://github.com/user-attachments/assets/171bb5cb-dc90-4bbc-85f3-91ca153fadc1)

Please, could you help me to solve this issue?

@josemakampa could you please check the console for any error messages? What's the response from the network request?

What version of promptfoo are you running?

@josemakampa Is there anything in you chrome console? If possible, can you `promptfoo export` the eval and send it to me (michael @ promptfoo.dev) along with your config file? 


@sklein12 I didn´t received any error in the console.

@mldangelo I have the version 0.115.0, i send you the output file and the config file.

Thanks so much both for the quick response.

Thanks both, i just resolved it cleaning the bronwser cache 

Btw i have a doubt, i check the config file, i used prompt from instruct module from CyberSecEval, but all the prompt were generated when i generated the config file using the plugings.

So the prompt from the file are useless right?
________________________________
De: Michael ***@***.***>
Enviado: domingo, 15 de junio de 2025 18:25
Para: promptfoo/promptfoo ***@***.***>
Cc: josemakampa ***@***.***>; Mention ***@***.***>
Asunto: Re: [promptfoo/promptfoo] Problem with my evals (Issue #4492)

[https://avatars.githubusercontent.com/u/7235481?s=20&v=4]mldangelo left a comment (promptfoo/promptfoo#4492)<https://github.com/promptfoo/promptfoo/issues/4492#issuecomment-2974194850>

What version of promptfoo are you running?

@josemakampa<https://github.com/josemakampa> Is there anything in you chrome console? If possible, can you promptfoo export the eval and send it to me (michael @ promptfoo.dev) along with your config file?

—
Reply to this email directly, view it on GitHub<https://github.com/promptfoo/promptfoo/issues/4492#issuecomment-2974194850>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A3ZPTCMNLL5EEOVM5KLWW2T3DWNANAVCNFSM6AAAAAB7LOPX6CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNZUGE4TIOBVGA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
Repository: promptfoo/promptfoo
Issue: Doubt with prompts

Hi, i have a doubt, i used some plugings and i used redteam generate for create a config file.

In the test section are the prompts generated by the tool and in the prompts section i especified a prompt file that i got from datasets from CyberSecEval, the doubt that i have IS why i should choose a prompt file, if the prompts that i want to test are those generated by promptfoo with redteam generate with the plugings that i choose.

Maybe is a bit confuse, i Hope someone can help me with this.

best regard and thanks in advance
Repository: promptfoo/promptfoo
Issue: ReliableGPT
Root cause: Unimplemented Feature Gap

While using promptfoo, the requests can often fail because of OpenAI's flakiness. I recommend us putting in something like 
https://github.com/BerriAI/reliableGPT
to improve reliability of failing requests. 

Please correct me if there is already a provision for retries. Thank you!

Thanks for the suggestion!  I've added retries with exponential backoff, which should help reduce some of the "model is overloaded" and similar OpenAI failures.
Repository: promptfoo/promptfoo
Issue: List of environment variables for redteaming
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
I find the [doc, where the generic environment variables are collected](https://www.promptfoo.dev/docs/usage/command-line/#ascii-only-outputs:~:text=bar%20%2D%2Dno%2Dtable-,Environment%20variables,-These%20general%2Dpurpose), really helpful. Could we have a similar collection for redteam environment variables? It's pretty tiring to read 10 pages and a bunch of code to find whether someting is controlled by an env var.

**Describe the solution you'd like**
Extend the documents with all used environment variables and their default behaviour, most importantly the ones concerning redteaming.

**Additional context**
The project I'm working on is very strict on what can be shared with 3rd parties, e.g. I see now that target discovery is disabled by default, but it was't a few weeks ago when I last generated tests and I couldn't easily find an env var or description what to do to disable it.


Thanks for surfacing this, @vira-s! I merged 90dc862 which adds a full red-team env-var reference and troubleshooting page. Give it a look and let me know if anything is missing. Closing this out, but happy to reopen if needed. 🙏

Thank you! It looks good 🙌
Repository: promptfoo/promptfoo
Issue: A more detailed description what Target Discovery really is
GitHub labels: documentation

**Is your feature request related to a problem? Please describe.**
I was trying to find out what Target Discovey is, but the [doc](https://www.promptfoo.dev/docs/red-team/discovery/) is very short about it. I have no idea what it stores about our LLMs or generation process/custom plugins or anything, it's not even in the API reference to see at least its request/response structure.

**Describe the solution you'd like**
I'd like a more detailed description about the Target Discovery process, what it does and what it stores about the evaluated LLMs.

**Additional context**
We have very strict policies on the project, and can't use anything that shares vital data with 3rd parties, but we like using Promptfoo and its features, I'd be sad to see it dropped.


Target Discovery works in an iterative fashion: it sends probing questions to a target LLM and evaluates whether it's satisfied with the target's answers, asking additional questions until it's satisfied.  No information about the target LLM is stored except in the case that an error is encountered: the LLM's responses to the questions will be stored in Promptfoo Cloud's error log.

Note that Target Discovery is disabled by default.
Repository: promptfoo/promptfoo
Issue: Unable to set config for google ai studio gemini api 
Root cause: Configuration Error


![image](https://github.com/promptfoo/promptfoo/assets/57288492/a12a4dc2-0758-4757-a546-b98365f4df63)
I am setting the config like below:

```
providers:
  - id: google:gemini-pro   
    config:
      temperature: 0
      topP: 1
```

Thanks for the catch.  I feel like Google has kept moving things around in their API.  #454 will fix support for `temperature` and `topP` and other generation config options.

This fix is published in 0.40.0
Repository: promptfoo/promptfoo
Issue: How to use self-hosted Promptfoo for eval execution and viewing results?
Root cause: Documentation Deficiency

Hello,

I'm a bit confused about the correct way to use the self-hosted Promptfoo setup, and I’d really appreciate some clarification.

I’ve pulled the official image (ghcr.io/promptfoo/promptfoo:latest) and started the container using Docker, with volume mappings as below. The web UI at http://localhost:3000 works fine.

```
docker pull ghcr.io/promptfoo/promptfoo:latest

docker run -d \
  --name promptfoo_container \
  -p 3000:3000 \
  -v C:\Users\den\data:/home/promptfoo/.promptfoo \
  ghcr.io/promptfoo/promptfoo:latest
```


However, I’m not sure what the proper workflow is for running evals and being able to view the results in this self-hosted container UI. Here are my main questions:

Should I still run promptfoo eval from my local CLI?

If I run the evals locally, how can I make sure the results are sent to the self-hosted container so I can view them at localhost:3000?

I’ve tried setting PROMPTFOO_REMOTE_API_BASE_URL and PROMPTFOO_REMOTE_APP_BASE_URL environment variables to point to the self-hosted instance(http://localhost:3000/), but the results still get saved in my local .promptfoo folder and only show up on localhost:15500 when I run the evals with "promptfo eval".

What is the correct and recommended setup to both run the evals and view results in the self-hosted UI?

I want to make sure I’m using self hosted Promptfoo in the right way and not mixing local and container usage in an unintended way.

Thanks a lot in advance!

after running the eval you have to upload it. For that, execute:
```
promptfoo share 
```

Also you can oneline it in the eval command:
```
promptfoo eval -j 16 --share --config promptfoo/tests/xyz.yaml
```

If you see some error you can dd the `-v` flag for more details. 
this is barely documented here: https://www.promptfoo.dev/docs/usage/self-hosting/#configuring-the-cli

Hi @denniseng88, thanks for the detailed questions, and thank you @GuillermoBlasco for pointing out the missing step. I’ve just merged #4730, which adds a **“Sharing eval results”** section and troubleshooting notes to the self-hosting docs: https://www.promptfoo.dev/docs/usage/self-hosting/#sharing-eval-results Closing the issue for now, but feel free to reopen if you have further questions.
Repository: promptfoo/promptfoo
Issue: Do we have native support for cohere AI models?
Root cause: Unimplemented Feature Gap

Hi! First of all I am sorry for asking two questions in short term. I am quite new here.

I wonder if we have native support for cohere AI, another big name in the field? Or so far we need to write our local python scripts for the models from them via [this guideline](https://www.promptfoo.dev/docs/providers/custom-api)?

You're correct that there isn't native support for Cohere right now.  But, it shouldn't be too difficult to implement native support - this is something I can help with.

Are you interested in the `chat` endpoint specifically? https://docs.cohere.com/reference/chat

Native support for Cohere Chat API is available in 0.40.0!  See https://promptfoo.dev/docs/providers/cohere

@typpo So many thanks! I will give it a shot very promptly. If there is any other problem encountered, I will open separate issue.
Repository: promptfoo/promptfoo
Issue: Frequent Error Message: NotFoundError: Not Found when running `promptfoo view`
Root cause: Validation Gap

**Describe the bug**
In my web browser, and on the console, I frequently see this error when Promptfoo is runing:

NotFoundError: Not Found
    at createHttpError (/Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/send/index.js:861:12)
    at SendStream.error (/Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/send/index.js:168:31)
    at SendStream.pipe (/Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/send/index.js:468:14)
    at sendfile (/Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/express/lib/response.js:1000:8)
    at ServerResponse.sendFile (/Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/express/lib/response.js:409:3)
    at /Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/promptfoo/dist/src/server/server.js:242:13
    at Layer.handleRequest (/Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/router/lib/layer.js:152:17)
    at next (/Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/router/lib/route.js:157:13)
    at Route.dispatch (/Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/router/lib/route.js:117:3)
    at handle (/Users/paul.prescod/.npm/_npx/1ddbeaccabfe00a4/node_modules/router/index.js:435:11)


**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run 'npx promptfoo eval'
2. Run 'npx promptfoo view'
3. Refresh the browser page.
4. You will see the error in the browser (or at least I always seem to)
**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

```
=============================================================================================================================================
Promptfoo Debug Information
=============================================================================================================================================
{
  "version": "0.115.1",
  "platform": {
    "os": "darwin",
    "release": "24.3.0",
    "arch": "arm64",
    "nodeVersion": "v23.7.0"
  },
  "env": {},
```

- Promptfoo version:0.115.1

**Additional context**

Browser is Chrome on Mac.



@prescod I had the same problem. I ended up downgrading to an older version (I used version 0.114.5) and that fixed the problem for me. Most likely a bug in the latest version.


Thanks for filing this, @prescod, and my apologies that you and @josiahxocyber have been experiencing it!  I've identified the issue and its fix will be included in the next release.
Repository: promptfoo/promptfoo
Issue: [Feat] Please support on web operation to config the full parameters of ai provider
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
yes, when running in command line, i could editing the yaml to let it support my relayed AI or local ai by provide `config.apiKey` and `config.apiBaseUrl` , but these value could not be edit in web view, since it even has not been displayed

**Describe the solution you'd like**
add more editable value in web view just like in yaml file, or allow user to edit in yaml format (in some web editor or a simple textarea?) and then been accept by the web view

**Describe alternatives you've considered**
none

**Additional context**
none


thanks very much
Repository: promptfoo/promptfoo
Issue: Promptfoo does not escape characters in HTML reports
Root cause: Validation Gap

**Describe the bug**
If a prompt contains a character that has special meaning in HTML (for example, `<`), promptfoo copies this character into HTML reports verbatim.

**To Reproduce**

Create the file “a.yaml”:
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json

description: "test"

prompts:
  - 'Is 1<2?'

providers:
  - "openai:gpt-4.1-nano"

tests:
  - vars:
      a: b
```

Generate an HTML report:
```bash
promptfoo eval -c a.yaml -o a.html
```

The file “a.html” will contain
```
<th>[openai:gpt-4.1-nano] Is 1<2?</th>
```
This is invalid HTML.

**Expected behavior**
The file “a.html” contains
```html
<th>[openai:gpt-4.1-nano] Is 1&lt;2?</th>
```

**System information:**

```
=====================================================================================================================================================================================================================================
Promptfoo Debug Information
=====================================================================================================================================================================================================================================
{
  "version": "0.115.1",
  "platform": {
    "os": "linux",
    "release": "6.14.10-arch1-1",
    "arch": "x64",
    "nodeVersion": "v24.1.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
=====================================================================================================================================================================================================================================
```

- Promptfoo version: 0.115.1

Thanks @beroal!  This issue should be resolved in the next release
Repository: promptfoo/promptfoo
Issue: Support for OpenAI chat completions "n" parameter
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When testing prompts, I often find that I need to run each prompt several times to ensure that I get consistent results.  Currently, using --repeat 5 makes 5 separate API calls, which works but takes some time due to rate limits and incurs some costs with large prompts.

OpenAI chat completions api accepts a "n"-paramter that generates several completions to the same prompt in one request. It would be great if we could use that to make repeat testing more efficient

**Describe the solution you'd like**
A solution coult be to be able to configure the OpenAI provider with an n parameter that generates multiple completions per API request. For example:
```
  providers:
    - id: openai:gpt-4.1
      config:
        n: 5  # Generate 5 completions in one API call
```

This should:
  - Include the n parameter in the OpenAI API request
  - Handle multiple choices in the response
  - Evaluate each completion separately
  - Work seamlessly with the existing --repeat flag (e.g., --repeat 2 with n: 3 would make 2 API calls for 6 total completions)

**Describe alternatives you've considered**
Instead of "n" being a provider config parameter, we could have --repeat use "n" for providers that support multiple outputs in one request. 

**Additional context**
Documentation for the "n" parameter is here:
https://platform.openai.com/docs/api-reference/chat/create#chat-create-n

Do you think this would be a good idea? If so I could have a stab at a PR that adds this feature
Repository: promptfoo/promptfoo
Issue: Allow progress bar in debug/verbose mode

A progressbar is helpful even in debug mode.

I think that disconnecting the two will be helpful.

Ref: https://github.com/promptfoo/promptfoo/blob/2a8eb9349bb7d041ca22439a98820039ee7e1edf/src/main.ts#L544

The reason for this logic is that there cannot be CLI output while the progress bar is dynamically updating, otherwise it gets mangled/overwritten by the bar.

I think seeing the progress while seeing the debug values has higher value than hiding it:
When I debug, and something happens after 1000-2000 (like unresponsiveness, or empty results, etc' - the sort of things that debug exists for..)
I still want to know *when* it happens, and I still want to see - in the happy path - that I'm nearing the end of the processing.


In general, I think that debug/verbose mode should mostly just add information

#461 adds progress information in debug output :)

(not as a progress bar, but just as regular console output)
Repository: promptfoo/promptfoo
Issue: Derived metrics doesn't work if eval config is uploaded via WebUI
Root cause: Configuration Error

**Describe the bug**
Derived metrics are not being processed or calculated when eval configurations are uploaded via the web UI. After uploading a configuration YAML file that includes derived metrics, the derived metrics section is stripped from the config and the metrics are not calculated in the evaluation results, even when manually added back to the configuration.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations:

1. Navigate to the Promptfoo web UI
2. Go to Create -> Eval
3. Click on "Edit Yaml"
4. Upload or paste the YAML configuration from the F1 metrics example at https://github.com/promptfoo/promptfoo/blob/main/examples/f-score/promptfooconfig.yaml
5. Run the evaluation
6. Observe that the derived metrics are not present in the uploaded config and F1 scores are not calculated
7. Manually add the derived metrics back to the configuration
8. Run the evaluation again
9. See that derived metrics are still ignored and not calculated

**Expected behavior**
When uploading a configuration file with derived metrics via the web UI, the derived metrics should be preserved in the configuration and calculated during evaluation. The F1 score (and other derived metrics) should appear in the evaluation results alongside the base metrics like precision and recall.

**Screenshots**

<img width="722" alt="Image" src="https://github.com/user-attachments/assets/d96c9a85-4996-4003-b0be-111ba337b5e1" />

<img width="1885" alt="Image" src="https://github.com/user-attachments/assets/65297378-7ff0-4c39-96fb-a8dcaa29d5a7" />

**System information:**
N/A - Bug is in WebUI eval only and there is no debug command available there.

- Promptfoo version:
0.115.1

**Additional context**
Additional context
This issue specifically affects the web UI workflow. The same configuration file works correctly when run via CLI. The derived metrics functionality appears to be completely bypassed when configurations are processed through the web interface, making it impossible to calculate composite metrics like F1 scores through the web UI workflow. I initially discovered this issue on my config, but then reproduced on the Promptfoo example - so I use only promptfoo original example in this description.
Test case reference: The issue can be reproduced using the official F1 score example configuration from the promptfoo repository at examples 
https://github.com/promptfoo/promptfoo/blob/main/examples/f-score/promptfooconfig.yaml 

Thank you so much @yexingyao @mldangelo !
Repository: promptfoo/promptfoo
Issue: Remote generation URL is still called after disabling remote generation with env vars
Root cause: Configuration Error

**Describe the bug**
A clear and concise description of what the bug is.

I don't want to use remote generation for now, I've set the `PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true` and `PROMPTFOO_DISABLE_REDTEAM_TARGET_DISCOVERY_AGENT=true`, however the goal extraction ignores these env vars

https://github.com/promptfoo/promptfoo/blob/1932ee2e03fcfdc30598249ea2fc5c10e51aaefb/src/redteam/util.ts#L236

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

Use e.g. the `contracts` plugin with `numTests: 2`. I've set the` redteam.provider` and `defaultTest.options.provider` to Bedrock Sonnet 3.7 and I use a custom python target.

1. Run 'promptfoo redteam generate -v'
2. See log:
```
[base.js:126] ContractPlugin generated test cases from 2 prompts
[index.js:605] Extracting goal for 2 tests from contracts...
[util.js:228] Extracting goal from prompt. Request URL: https://api.promptfoo.app/api/v1/task
```

**Expected behavior**
A clear and concise description of what you expected to happen.

I expected the generate to execute without calling anything remote (including the promptfoo.app task API)

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version:
- 0.115.1

**Additional context**
Add any other context about the problem here.


Thank you for the quick solution! I've tried the new version and it works as expected.
Repository: promptfoo/promptfoo
Issue: View server
Root cause: Configuration Error

Hi, I was trying to create a self-hosted server with promptfoo so anyone can execute it. However, I was run some local eval from the ui and I am having a `500: Internal Server Error`. Any clue about what is happening inside? 
Thank you in advance

Any output on the server side?  Can you provide more information about the request that is returning the 500?

The error that I am getting is that is not getting to fetch the api.
```
TypeError: fetch failed
    at Object.fetch (node:internal/deps/undici/undici:11730:11)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  cause: Error: getaddrinfo ENOTFOUND auth
      at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:107:26)
      at GetAddrInfoReqWrap.callbackTrampoline (node:internal/async_hooks:128:17) {
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'auth'
  }
}
- error {
  message: 'TypeError: fetch failed',
  details: 'TypeError: fetch failed\n' +
    '    at Object.fetch (node:internal/deps/undici/undici:11730:11)\n' +
    '    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)',
  hint: '',
  code: ''
}
```

I tried to put all the variables that you indicated in for the supabase, also the api and app but still not reaching it. I also create the docker following the one provided in the repository. I checked the api and it seems is up but the view cannot reach it. Any tip to solve this?


The Dockerfile released in 0.42.0 contains updates that should resolve issues with self-hosting the share functionality described here: https://promptfoo.dev/docs/usage/sharing/#self-hosting.  Please let me know if not the case