zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: How to use promptfoo installed from pypi.org
Root cause: Environment Incompatibility

Hi.
I've installed promptfoo form tthis package https://pypi.org/project/promptfoo/. Running promptfoo ends with the following exception:
```fish
(promptfoo-self-learning-py3.13) damian@dgiebas-ubu-i4b-pl ~/A/promptfoo-self-learning (master)> promptfoo --version
Traceback (most recent call last):
  File "/home/damian/.cache/pypoetry/virtualenvs/promptfoo-self-learning-nABErWj3-py3.13/bin/promptfoo", line 5, in <module>
    from promptfoo.main import main
ModuleNotFoundError: No module named 'promptfoo'
(promptfoo-self-learning-py3.13) damian@dgiebas-ubu-i4b-pl ~/A/promptfoo-self-learning (master) [1]> 
```

Am I doing something wrong? Is it possible to add to README.md a short description how use it?
Repository: promptfoo/promptfoo
Issue: HTML Output rendering does not have all information
Root cause: Unimplemented Feature Gap

Hello,

When I rendering my results to HTML the title of columns say object Object (I am using the custom provider). Is there a way I can get the provider instead, maybe the file name that I use could be better instead. Moreover, there is no information about what assertion type was executed against the test case as well. It would be nice to add that information as well.

Thanks for the report.  This is fixed in 0.35.1
Repository: promptfoo/promptfoo
Issue: o4-mini is supported only on responses api and not chat completions
Root cause: Validation Gap

**Describe the bug**
o4-mini is not working with yaml configuration when using `openai:o4-mini` or `openai:chat:o4-mini`.
It is always throws an exception in openai API about using `max_tokens` is not supported, use `max_completion_tokens`

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

use this configuration:
```
  - id: openai:o4-mini // or openai:chat:o4-mini
    label: 'o4-medium'
    config:
      isReasoningModel: true
      max_completion_tokens: 60000
      reasoning_effort: medium
      response_format:
        type: json_object
```

And you will get an error:
```
API error: 400 Bad Request
{"error":{"message":"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.","type":"invalid_request_error","param":"max_tokens","code":"unsupported_parameter"}}
```

**Expected behavior**
Expect that max_tokens not be sent and the request to work

**Screenshots**

<img width="513" alt="Image" src="https://github.com/user-attachments/assets/817fc98a-0d97-47f1-be9f-df0f38c0bcf1" />

**System information:**
```
"version": "0.110.0",
  "platform": {
    "os": "darwin",
    "release": "24.4.0",
    "arch": "arm64",
    "nodeVersion": "v22.14.0"
  },

```
- Promptfoo version: 0.110.0



This is supported just needs a release I believe.  Coming shortly

Should be good in 0.110.1!
Repository: promptfoo/promptfoo
Issue: Error when calling gemini-pro
Root cause: Environment Incompatibility

I'm setting the config like this:
providers:
- vertex:gemini-pro:
      id: gemini-pro
      config:
        apiKey: 
        projectId: 
        temperature: 0

But the runtime report the following error:
 {"error":{"code":429,"message":"Quota exceeded for aiplatform.googleapis.com/online_prediction_requests_per_base_model with base model: gemini-pro. Please submit a quota increase request. https://cloud.google.com/vertex-ai/docs/quotas.","status":"RESOURCE_EXHAUSTED"}}

After search google, it seems to be calling the wrong endpoint:
https://www.googlecloudcommunity.com/gc/AI-ML/Gemini-Pro-Quota-Exceeded/m-p/693185

I'm wondering if anyone has successfully called gemini-pro using promptfoo? Thanks!

Can you make sure you're on the latest version?  `gemini-pro` uses an API that is different from the `bison` models, and support was added a few versions ago in 0.32.0.

I am able to use `gemini-pro` locally with this setup: https://github.com/promptfoo/promptfoo/tree/main/examples/google-vertex

yes, you are right. I was using the old version.
Repository: promptfoo/promptfoo
Issue: Better CLI Visuals when Passing many Threads
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When many threads are passed in, more than lines are available in the CLI, the visuals basically and you mostly see black screen with occasional updates for a split second.
`promptfoo eval -j 50`

**Describe the solution you'd like**
Some better visualization for many threads.
The current visualization for 5-15 threads is fine, but that doesn't scale to 100 threads.

**Describe alternatives you've considered**
Mostly I just sit and watch the black screen with occasional flashing lines, but I can't really track the progress this way. I just have to look back from time to time and see if my CLI is back.
Repository: promptfoo/promptfoo
Issue: Configuring Provider for Assertions (i.e. g-eval) Doesn't work with FunctionProvider
Root cause: Validation Gap

**Describe the bug**

I'd like to configure a default function provider for model-graded metrics using the node API but it doesn't work.

Somewhere to look first would be https://github.com/promptfoo/promptfoo/blob/main/src/index.ts#L53 - it's calling loadApiProvider if the parameter is a function _but_ loadApiProvider states that it currently does not handle functions https://github.com/promptfoo/promptfoo/blob/main/src/providers/index.ts#L18

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

Example node code:
```js
const results = await promptfoo.evaluate({
    prompts: ['Rephrase this in French: {body}', 'Rephrase this like a pirate: {body}'],
    defaultTest: {
       options: {
         provider: (prompt, context) => { ... }
      }
      assert: [
        {
          type: "g-eval",
          value: [
            "test to ensure the model provides one response",
            "ensure that the model responds in a polite manner",
          ],
        }
      ]
    },
```

**Expected behavior**

My provider function is called to process the g-eval assert.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

```
{
  "version": "0.111.1",
  "platform": {
    "os": "darwin",
    "release": "24.4.0",
    "arch": "arm64",
    "nodeVersion": "v20.18.1"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```
Repository: promptfoo/promptfoo
Issue: Running any command shows deprecation warning
Root cause: External Dependency Breakage

This happens on `promptfoo init`, `promptfoo help`, `promptfoo view`, etc

```
$ promptfoo help
(node:144661) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
```

Seems to be `ajv`'s fault

```
$ npm ls punycode
promptfoo@0.35.0 /home/kainoa/Git/promptfoo
└─┬ ajv@8.12.0
  └─┬ uri-js@4.4.1
    └── punycode@2.3.0
```

ref: https://github.com/ajv-validator/ajv/issues/2350
will be fixed by: https://github.com/garycourt/uri-js/pull/95

Closing as invalid since it's not an issue pertaining to this repo directly.
Repository: promptfoo/promptfoo
Issue: Truncated Test Name in CLI Progress Bar
Root cause: Architectural Constraint

**Describe the bug**
The CLI progress bar doesn't display enough characters to see which test is being run.

**To Reproduce**
Run promptfoo eval

Sample Configuration:
```YAML
description: "Promptfoo test cases for different stories"
prompts:
  - ./test/promptfoo/common/prompts.mjs:a
  - ./test/promptfoo/common/prompts.mjs:b
providers:
  - id: file://./test/promptfoo/providers/MockAppProvider.mjs
    label: "MockApp"
defaultTest:
  options:
    provider:
      id: file://./test/promptfoo/providers/CustomGPTProvider.mjs
      label: "CustomGPT"
      config:
        api_version: "2024-02-01"
        chat_completions_model: "gpt-4o"
        embeddings_model: "text-embedding-ada-002"
        temperature: 0.7
        max_tokens: 2000
  assert:
    - type: answer-relevance
      threshold: 0.7
tests:
  - file://./test/promptfoo/*-stories.*
outputPath: ./test-reports/promptfoo/promptfoo-result.json
evaluateOptions:
  maxConcurrency: 8
```

**Expected behavior**
Expected more characters reserved for displaying variables, because our vars contain the input differences.  Therefor displaying more vars would indicate better which test is being run.

**Screenshots**

<img width="848" alt="Image" src="https://github.com/user-attachments/assets/d82e0487-6370-4043-a0cc-81e808bd154b" />

**System information:**

- Promptfoo version: 0.107.7



BTW Promptfoo is excellent.

@sklein12 implemented a small fix for this issue, would appreciate your thoughts on this!
Repository: promptfoo/promptfoo
Issue: Support for openai chat completions / non zero-shot prompts
Root cause: Documentation Deficiency

Support for prompts that are beyond zero-shot would be great. To give an example, see the snippet below:

```
openai.ChatCompletion.create(
  model="gpt-3.5-turbo",
  messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in 2020?"}
  ]
)
```

The current examples eg. e-commerce assistant are all zero-shot. 

Hi Murat!  This is actually supported right now.

Your `prompt.txt` (or `prompt.json`) would be:

```json
[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Who won the world series in {{ year }}?"}
]
```

Note that `{{ year }}` will be substituted with the appropriate test variable.

I'll keep this issue open as a reminder to update the docs/example.

**Phase 1 Complete ✅**

Successfully fixed the Vite/Terser build error.

**Changes Made**:
1. Switched from `minify: 'terser'` to `minify: 'esbuild'` (Vite 7 default minifier)
2. Added external configuration to exclude `vite-plugin-node-polyfills/shims` from bundling
3. Simplified terserOptions (no longer needed with esbuild)

**File Modified**:
- `src/app/vite.config.ts` (lines 49-55)

**Build Result**: ✅ Success
- Build completes without errors
- dist/src/app/ folder created with all assets
- Build time: 26.42s

**Validation**:
- [x] Vite configuration error resolved
- [x] `npm run build` completes without Vite errors  
- [x] `dist/` folder created with frontend assets

**Progress**: 100% | **Status**: COMPLETE
Repository: promptfoo/promptfoo
Issue: Scenarios don't use separate _conversation keys
Root cause: Architectural Constraint

I was surprised to see chat messages stored in `_conversation` leaking across different scenarios.

Is this a bug, or is there an example that can illustrate how to ensure different scenarios have their own `_conversation` states?

Is there any update on this?

Or is there a way to reset the `_conversation` variable? I can trying to test multiple rounds of a chat/conversation. However, at the moment, I limited to one conversation per `.yaml` configuration.
Repository: promptfoo/promptfoo
Issue: Whitespace unexpectedly matters in csv assertions
Root cause: Validation Gap

**Describe the bug**
When defining assertions in csv, like this from the docs:

```csv
input,__expected1,__expected2,__expected3
"What's the capital of France?","contains: Paris","llm-rubric: Is the response accurate and factual?","python: file://verify_capitals.py"
"What is the sum of 15 + 27","equals: 42","contains: sum","javascript: file://check_math.js"
```

promptfoo will look for ` Paris` including the leading space. I find that confusing. I recommend we either adjust the docs to remove whitespace, or `trim()` the expected values.

Thanks!

- Promptfoo version:  0.112.1


Thanks for reporting this, @kevinschaul! The fix will ship in our next release.

Amazing, thank you

Thanks for reporting it, @kevinschaul. Check out 0.112.2 and let me know if you encounter any more issues!
Repository: promptfoo/promptfoo
Issue: Support text-generation-webui via its openai API
Root cause: Unimplemented Feature Gap

The documentation doesn't list [text-gen-webui](https://github.com/oobabooga/text-generation-webui) as supported, however getting it to run isn't difficult as it has an OpenAI API [extension](https://github.com/oobabooga/text-generation-webui/wiki/12-%E2%80%90-OpenAI-API).

If it works, what is this issue about?
Currently, promptfoo doesn't forward additional `config` parameters to the API if they are outside the official OpenAI parameters.

The text-gen-webui extension can be activated from with the UI or via command line. Here is an example of command line usage.

```
python server.py --loader <LOADER-NAME> --model <MODEL-NAME> --api
#optionally replace `python server.py` with ./start_linux if simple installer is used
```

In promptfoo we can address the API as follows.

```
  - openai:chat:<MODEL-NAME>:
      id: <MODEL-ID>
      config:
        apiKey: foo
        apiBaseUrl: http://localhost:5000
        temperature: 0.8 # works
        max_tokens: 1024 # works
        mode: instruct # ignored
        instruction_template: LLama-v2 # ignored
```

We can test the API with curl, here the additional non-standard arguments are adhered to.

```
curl http://127.0.0.1:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
	  {
	    "role": "system",
	    "content": "You give pizza recipies."
	  },
      {
        "role": "user",
        "content": "Pepperoni pizza."
      }
    ],
    "mode": "instruct",
    "instruction_template": "Llama-v2",
    "temperature": 0.0,
    "max_tokens": 200
  }'
```
Is a pass through of arguments feasible?

Thanks for the suggestion, could definitely see this being useful. #388 adds this functionality, I will land it once I update the documentation too.

Right now the implementation would allow for the following:

```yaml
  - openai:chat:<MODEL-NAME>:
      id: <MODEL-ID>
      config:
        apiKey: foo
        apiBaseUrl: http://localhost:5000
        temperature: 0.8 
        max_tokens: 1024 
        passthrough:
          mode: instruct 
          instruction_template: LLama-v2 
          # anything else you want...
```
alternatively I could make it work without the `passthrough` key like in your original example.  Just a bit messier in the code..

Passthrough looks good. 
Thank you and happy new year!

`passthrough` support landed in 0.35.1, and I've updated the documentation
Repository: promptfoo/promptfoo
Issue: Providing "completion" to custom factuality prompt
Root cause: Configuration Error

I need to change the default factuality prompt and I tried doing that by modifying the prompt as suggested [here](https://www.promptfoo.dev/docs/guides/factuality-eval/#customizing-the-evaluation-prompt). It also says in the docs:

> You must implement the following template variables:
> 
> {{input}}: The original prompt/question
> {{ideal}}: The reference answer (from the value field)
> {{completion}}: The LLM's actual response

I could pass input and ideal as part of `vars` since they are static and no model calls are involved, but I'm unable to do that. One way of course, would be to manually call the model when populating the `test_cases` dict, but I'm wondering if there is a way to keep that (i.e. the API call to the defined model) automated instead of doing that explicitly. I assume that the library behind the scenes calls whatever is provided in options[provider][id].

I just want to use the good old
```
{
    'type': 'factuality',
    'value': correct_answer,
    'metric': 'some_name'
},
```

Just to make sure we are on the same page

for a factuality test like this

```
providers:
  - openai:gpt-4.1-mini
prompts:
  - |
    Please answer the following question accurately:
    Question: What is the capital of {{location}}?
tests:
  - vars:
      location: California
    assert:
      - type: factuality
        value: The capital of California is Sacramento
```

what will happen is there will be a llm call being made 

```
....
      Question:  Please answer the following question accurately:\nQuestion: What is the capital of California?\n\n
      Reference answer: The capital of California is Sacramento
      Submitted answer: <What ever the llm responded with>
....
```

so the  Reference answer and Question are static in the example. How does your example differ that you are unable to do have these parts be static or am i missing something.

Hey @ghadialhajj - this should be working now!

The issue was that `rubricPrompt` was being rendered before the `{{completion}}` variable was available. We fixed this in #5340 by deferring the prompt rendering until after the LLM response comes back.

You can now use custom factuality prompts like:

```yaml
defaultTest:
  options:
    rubricPrompt: |
      Input: {{input}}
      Reference: {{ideal}}
      Completion: {{completion}}

      Evaluate the factual consistency...
```

All three variables get populated automatically - no need to make manual API calls.

Docs were also updated to clarify this: https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/factuality/#customizing-the-prompt

Let me know if you run into any issues!
Repository: promptfoo/promptfoo
Issue: Consistant colors on eval list and eval view pages

**Describe the bug**
Consistent colors used for evaluation results and evaluation view page. having consistent colors helps visually identify the Eval i am looking for. Opening as an issue first before modifying anything to discuss ideal outcome

**To Reproduce**

**Expected behavior**
Colors in eval list and on eval view page should be the same 

**Screenshots**
Eval view page seems to use some sort of green gradient, where as the eval list page use block yellow,red,gren

<img width="377" alt="Image" src="https://github.com/user-attachments/assets/bc76fe27-fa70-42a4-8d37-d3d10b710cc0" />

<img width="275" alt="Image" src="https://github.com/user-attachments/assets/c864ba70-a6ef-42e7-8750-76489413736f" />

<img width="268" alt="Image" src="https://github.com/user-attachments/assets/96be6e78-fbca-45cd-a2ed-b03c4b0e1bd5" />

<img width="256" alt="Image" src="https://github.com/user-attachments/assets/d1bf1207-dcb8-4c45-a198-870244354997" />


Thanks for the detailed issue and screenshots, @billybonks! You're right that the color treatments differ between these views.

After discussing this, we think the difference is actually intentional rather than inconsistent:

- **Eval list page**: Optimized for scanning/filtering many evals. Heavy color-coding here would add visual noise when users are trying to find a specific eval. The sortable columns let users find what they need.

- **Eval view pills**: These serve as a quick dashboard summary when you're analyzing one eval in depth. Color helps you immediately spot which prompts are underperforming.

The different contexts call for different visual weight — similar to how a spreadsheet stays neutral while a dashboard uses color for emphasis.

That said, if you're running into a specific workflow where this creates confusion, we'd love to hear more about it. For now we're going to close this, but feel free to reopen with more details if the current UX is causing friction in practice.

Thanks again for thinking about the user experience!
Repository: promptfoo/promptfoo
Issue: better indication as to what is going on during evaluations
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
1. When running the eval in terminal, i have no idea what the threads are doing sometimes they look like they are hanging for no reason.

2. It would be good to know how many requests were cached or errored


**Describe the solution you'd like**
**For the firs issue:**
my examples are using 100% but this is for in progress and only changing the messaging. i could only think of these 3 cases

when an api call is being made it can  display as normal
`[████████████████████████████████████████] 100% | ETA: 0s | 1/1 | http://localhost:3000/admin/llm/classify_comment "llama-3.3-" remote_id=`

when it is backing off due to rate limit or 500 retry
`[████████████████████████████████████████] 100% | ETA: 0s | 1/1 | backing off 500/429 5s`

when waiting for the delay to complete
`[████████████████████████████████████████] 100% | ETA: 0s | 1/1 | waiting for delay timeout 4000ms`

**For the second issue:**
Could use colors for example blue for cache default for default and red for errors

**Describe alternatives you've considered**
I have not considered other solutions


I usually use the verbose logging (`promptfoo eval --verbose`), which will nicely print the exact requests sent to the LLMs (both the one you evaluate and the grader), and also whether the response was cached or returned from cache (e.g. "Returning cached response for _provider_": _the full data object_). 
If you decide to use it, I also suggest to pipe it into a file, since it'll spam you with every single reques/response and your terminal won't show it (`promptfoo eval --verbose > output.log`).
Repository: promptfoo/promptfoo
Issue: Ability to rerun an evaulation
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
Lets say my api call had an error, i would like to fix it and rerun that evaluation, as of today i am stuck with the error cases for ever

**Describe the solution you'd like**
another button on eval_result that reruns the evaluation

**Describe alternatives you've considered**
I have not considered alternatives

**Additional context**
While i am focused on the error case here, users could end up modifying the prompt and rerunning an evaluation to correct it, while this could break the integrity of the evaluation, it might not be as important as what is most important is the evaluation when committing to the change versus when you are iterating.


Hey @billybonks! This functionality now exists in promptfoo. You have a few options depending on your use case:

**1. Retry errors from a specific evaluation:**
```bash
promptfoo retry <evalId>
```
This will retry all ERROR results (e.g., API timeouts, provider errors) from that evaluation.

**2. Retry errors from your latest evaluation:**
```bash
promptfoo eval --retry-errors
```

**3. Re-run only the failing tests in a new evaluation:**
```bash
promptfoo eval --filter-failing <evalId>
```
This filters to tests that didn't pass (both errors and assertion failures).

**4. Re-run only the error tests in a new evaluation:**
```bash
promptfoo eval --filter-errors-only <evalId>
```

The `retry` command modifies the original evaluation in place, while the `--filter-*` flags create a new evaluation with only the filtered tests.

Closing as complete. Hope all is well.