zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: command line option to only retry failed requests (eg. 500 server error or API returning nothing)
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I am abusing Promptfoo to perform benchmarking. If the model FAILs, I most often don't want to retry the prompt. I would like to retry only the cases, that actually failed because of network error, server error, API error etc. 

**Describe the solution you'd like**
Please add a command line option to allow to retry only failed cases. 

**Describe alternatives you've considered**
An alternative I considered is trying to code this myself but I am Python coder XD 

Thanks for a great piece of software. 

Hi @mpasternak,

Thank you for the kind words and for suggesting this great idea! We'll add this feature request to our backlog, though it may take some time before we can get to it. In the meantime, a potential workaround is to implement a custom provider with your own retry logic (which you can do in Python). You can find more details here: https://www.promptfoo.dev/docs/providers/python/

If you decide to give it a try, I'd be happy to help scope it out or even pair with you on it.

Thanks again for your feedback and support!

Upvote on this one but I have a slightly different use case. I'm running 2 prompts through 2 providers with 60 test cases each. Right now I'm hitting failures on only 2 cases. It would be great to be able to re-run those failed ones only without commenting out my test suite each time after tweaking a prompt slightly.

@ImTheDeveloper @mpasternak this is being worked on in https://github.com/promptfoo/promptfoo/pull/2214 It will go in soon! 

> Upvote on this one but I have a slightly different use case. I'm running 2 prompts through 2 providers with 60 test cases each. Right now I'm hitting failures on only 2 cases. It would be great to be able to re-run those failed ones only without commenting out my test suite each time after tweaking a prompt slightly.

+1 I also have this usecase. I tried understanding the description of https://github.com/promptfoo/promptfoo/pull/2214 but I wasn't sure. @mldangelo Could you please clarify that, with this merge would it be possible to pick and re-run only the failed (or errored) test caseswith promptfoo eval? I tried looking at the README in the merge but it looks incomplete? I know that there is already for e.g. --filter-metadata option for promptfoo eval to run specific test cases, so would there be a similar option to run only the failing/ errored test cases?

Finally circling back to close this one out - this has been implemented for a while now!

For retrying only errors (network/API issues, not test failures): `--filter-errors-only latest` or `--retry-errors`

For re-running all failed tests after prompt tweaks: `--filter-failing latest`

Both accept an eval ID or path to a JSON output file.

Docs: https://www.promptfoo.dev/docs/usage/command-line/#promptfoo-eval

Please feel free to comment or re-open if you run into any other issues!
Repository: promptfoo/promptfoo
Issue: Support reasoning_steps in some way
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Not related to a problem. 

**Describe the solution you'd like**
I would like an option to see reasoning steps for failed prompts. 

For example, if a prompt fails, I would actually love to have a a button in the web UI that would send a request to that AI model that would - if supported - show reasoning steps. Like the one described here: https://openai.com/index/introducing-structured-outputs-in-the-api/

**Describe alternatives you've considered**
The alternative is to copy and paste the prompt, manually enter the variables into it and retry the request. 

**Additional context**
Such feature would actually be great. Really.
Repository: promptfoo/promptfoo
Issue: feat: Vitest Integration ?
GitHub labels: good-first-issue

Since we have Jest and Mocha, it would be to consider to have a [Vitest](https://vitest.dev/) integration as well.

Going to treat this one as low priority, since we have instructions for jest and vitest has a jest-compatible api

Deployed in [0.69.1](https://github.com/promptfoo/promptfoo/releases/tag/0.69.1). See https://github.com/promptfoo/promptfoo/tree/main/examples/jest-integration 

Thanks @MentalGear!
Repository: promptfoo/promptfoo
Issue: Get "ERR_UNKNOWN_ENCODING" while import some modules in python script used as assertions
Root cause: Environment Incompatibility

**Describe the bug**
When I use External.py as assertion and import the module below in my script:

> from evaluate import EvaluationModule

It will return the error message:
<img width="957" alt="image" src="https://github.com/user-attachments/assets/dcc3d4f0-4dd1-492a-9ef9-637f9f2168de">

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Import the module in any External.py 
<img width="1090" alt="image" src="https://github.com/user-attachments/assets/48f22247-80e1-42d8-a424-ee6d04d6d286">

2. Run evaluation with command "npx promptfoo@latest eval"
<img width="981" alt="image" src="https://github.com/user-attachments/assets/aeedc7b9-8573-436d-b2f0-a4d0988d62e3">

**Expected behavior**
I expect it should return the evaluation without this error

**System information (please complete the following information):**
- evaluate module version:  <img width="254" alt="image" src="https://github.com/user-attachments/assets/d0e15e95-aa88-4293-b762-2e3ad602e11d">
- Promptfoo version: 0.81.3

**Additional context**
- I try to use some old versions of Promptfoo like 0.70.0 or 0.61.0 and they will return the error, neither.
- I try to follow the error message and edit the source code in the index.js, it could work without return the error. I'm not sure if the error occurs more than once in the loop in the source code L:66 so that it would return the error even though it has @ts-ignore in L:55.
- Original Code:
<img width="639" alt="image" src="https://github.com/user-attachments/assets/a7cb2364-5853-4892-909c-b6bea37ce442">

- After Editing the Code:
<img width="673" alt="image" src="https://github.com/user-attachments/assets/2e6cc9f8-e57e-46f4-b0a1-8a16135630e4">
 


After I try to downgrade the node.js version from v22.7.0 to v21.0.0, the encoding error won't occur anymore.

Hey @kevinyhl, thanks for reporting this! I believe the issue should be resolved now. If you're still experiencing the problem, feel free to reflag the issue, and I'll take another look. Appreciate your patience and contributions!
Repository: promptfoo/promptfoo
Issue: Is it possible to delete public promptfoo result?
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
Hi, I want to delete associated data and shareable url showing my promptfoo result after running following command

```
promptfoo share -y
```

It seems that url is something like `https://app.promptfoo.dev/eval/f:{id}`. Is it possible to delete this promptfoo result by myself ? After seeing [official docs](https://www.promptfoo.dev/docs/usage/sharing/#privacy) related to this topic, I cannot find how to delete associated data with the url.

**Describe the solution you'd like**
It will be great if there is any feature that I can delete my promptfoo result by myself.

**Describe alternatives you've considered**
NA

**Additional context**
NA


Shared results are automatically deleted after 2 weeks. If you've shared something sensitive and need it removed sooner, feel free to reach out via DM in our discord, and we can manually delete it for you. 

This has been handled 

@mldangelo @typpo 
thank you for handling this issue so quickly!
Repository: promptfoo/promptfoo
Issue: Allow markdown input for prompts
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Markdown is popular format used for creating prompts. Currently, it's only possible to load markdown files when giving them a txt extensions, which defeats the proper syntax highlighting in IDEs and cause issues with default prompt splitter `---`.

**Describe the solution you'd like**
Allows to load `.md` files directly, without applying prompt delimiter when loading such files.

**Describe alternatives you've considered**
Using .txt files and overriding default prompt delimiter.




Hi @sinedied, I merged a PR implementing support for markdown prompts, and we’ll ship it in a release on Monday. Thanks again for the great suggestion. Please feel free to share any more feedback or ideas!

Wow, thanks for the quick turnaround I'll right away!

Hi @sinedied, we’ve just released [0.85.1](https://github.com/promptfoo/promptfoo/releases/tag/0.85.1), which includes markdown support for prompts. Let me know if you run into any issues or have further feedback. Thanks for the great suggestion!
Repository: promptfoo/promptfoo
Issue: What is default for `--delay`?

A request would be to document the default for `--delay`, and that it's an `int`. I think from [here](https://github.com/promptfoo/promptfoo/blob/da8c07643766e68dcc3744103c9225eb16ea3ee1/src/main.ts#L378) it's default is `0`.

I will gladly contribute this, just wanting to confirm the default delay is 0 ms.

While I am at it, let's document `--max-concurrency`'s default, which I believe is an `int` with default of 4?  If you can also confirm this

Yes, both are correct!

Would you rather:

- Just add the default to the string message (how it's done for `--config`)
- Update the default into the source for `option()` (how it's done for `--table-cell-max-length`)

Yeah, it is a bit inconsistent right now.  Let's go with the `option` when possible.  I've pushed 82d8f5f52ba7d86221bdacf3b081da098d40c846 to address max concurrency, delay, and repeat.

Sounds good, thank you!
Repository: promptfoo/promptfoo
Issue: Please suport Mistral Nemo
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Cannot use open-mistral-nemo with promptfoo

**Describe the solution you'd like**
Id like  to be able to add open-mistral-nemo to my config file

**Describe alternatives you've considered**
No alternatives. Devs must help :-P

**Additional context**
https://docs.mistral.ai/getting-started/models/


Hi @mpasternak, Thanks for bringing this up! We’ve just released [0.85.0](https://github.com/promptfoo/promptfoo/releases/tag/0.85.0), which includes support for Mistral Nemo. You should now be able to add it to your promptfooconfig files. Feel free to check it out and let us know if you encounter any further issues. We really appreciate your feedback!

You guys are great. 

Do you realize, that I'm using promptfoo in a way that has probably not been thought of by the creators of it? Would you be interested in a short call?

@mpasternak Awesome! We'd love to hear more. Feel free to book a time on our calendar [here](https://cal.com/team/promptfoo/intro) and we can chat!
Repository: promptfoo/promptfoo
Issue: Errors when hitting rate limit when using Azure OpenAI
Root cause: Validation Gap

**Describe the bug**
When using model graded metrics such as `llm-rubric` with Azure OpenAI provider, when you hit the model rate limit all assertions fails with errors instead of waiting and retrying.

**To Reproduce**

1. Create a test case using Azure OpenAI provider for assertions
2. Add many assertions using `llm-rubric`
3. Run the test on multiple models or use high repeat like `--repeat 20`

**Expected behavior**
No error thrown, but model should wait until the end of rate limit and retry.

**Screenshots**
<img width="1095" alt="image" src="https://github.com/user-attachments/assets/9acfed30-c910-4aae-9317-fcd30ff77ce2">

**System information (please complete the following information):**

- Promptfoo version: 0.84.1

**Additional context**
N/A

Hi @sinedied,

Thanks for reporting this, and I apologize for the inconvenience. I’ll look into:

1. Whether we’re respecting Azure's rate limit headers.
2. Adding support for caching on repeats.

In the meantime, I recommend rerunning the eval with lower concurrency (-j 1) or adding a delay (--delay) to avoid hitting the rate limit. Another option is to implement a custom provider as a JS or Python file that supports caching or custom retry logic for your use case.

Let me know if you have any further questions! I’ll tag you in related PRs, though implementing new features may take some time.

Thanks! Let me know if you need help with the PR

The `fetchWithCache` function does respect 429s with exponential backoff, assuming Azure uses standard rate limit headers.  We'd definitely accept a PR for this if there is room for improvement (see [azureopenai.ts](https://github.com/promptfoo/promptfoo/blob/main/src/providers/azureopenai.ts)).  

(side note to @mldangelo's 2 point: we intentionally do not cache when repeat is >1)

@typpo Looking into the code, I've found a few issues:
- https://github.com/promptfoo/promptfoo/blob/main/src/fetch.ts#L50 : Azure uses 2 different headers for rate limiting, `X-Ratelimit-Remaining-Requests` and `X-Ratelimit-Remaining-Tokens`
- https://github.com/promptfoo/promptfoo/blob/main/src/fetch.ts#L54 : When rate limited, the `X-Ratelimit-Reset-Tokens` is set, instead of the one here. Alternatively, the header `Retry-After` can be used.

I've open a PR to improve these behaviors, as custom headers may vary depending of the provider.

PR sent 😉 

While adding unit tests, I also found out that the rate limiting logic was never called, so I fixed it.

Resolved by #1633
Repository: promptfoo/promptfoo
Issue: Can web crawlers find `promptfoo share` results?

I realized `promptfoo share` is creating a public URL.  Just two questions:
- Can web crawlers find the `promptfoo share` results?
- Per [here](https://promptfoo.dev/docs/usage/sharing/#privacy), it lasts for 2 weeks.  Can that be shortened/configured?

25081b04c60aef6d16a6460d77fed167bf44b91a adds `noindex` to all promptfoo eval pages, which disallows indexing.  Thanks for flagging!

There isn't a way to adjust the lifetime right now.
Repository: promptfoo/promptfoo
Issue: 0.85.2 breaks output formatting in view with Together.ai provider, Llama 3.1 70B
Root cause: External Dependency Breakage

**Describe the bug**
After upgrading to 0.85.2, prompt outupt in view changed to this:
```
{"role":"assistant","content":"<table id=\"3\">\n  <tr>\n    <td id=\"982\"></td>\n    <td id=\"117\">Total Revenue</td>\n    <td id=\"118\">Total Expenses</td>\n    <td id=\"119\">Net Income</td>\n  <td id=\"982\">New Acquisitions</td>\n    <td id=\"351\"></td>\n  </tr>\n  <tr>\n    <td id=\"117\">Ending Balance 12/31/2023</td>\n    <td id=\"493\"></td>\n  </tr>\n  <tr>\n    <td id=\"658\">Reserve Funds</td>\n    <td id=\"219\"></td>\n  </tr>\n  <tr>\n    <td id=\"421\">Beginning Balance 1/1/2023</td>\n    <td id=\"753\"></td>\n  </tr>\n  <tr>\n    <td id=\"936\">Impairment Provisions</td>\n    <td id=\"118\"></td>\n  </tr>\n  <tr>\n    <td id=\"275\">Ending Balance 12/31/2023</td>\n    <td id=\"982\"></td>\n  </tr>\n  <tr>\n    <td id=\"467\">Net Worth</td>\n    <td id=\"542\"></td>\n  </tr>\n  <tr>\n    <td id=\"624\">Ending Balance 12/31/2023</td>\n    <td id=\"819\"></td>\n  </tr>\n  <tr>\n    <td id=\"351\">Ending Balance 12/31/2022</td>\n    <td id=\"138\">528,524,535</td>\n  </tr>\n</table>","tool_calls":[]}
```

Instead of just the unescaped content like before.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Upgrade to 0.85.2
2. Run existing prompts against Llama 3.1 70B, together.ai
3. Open output in view
4. Inspect an individual response from LLM

**Expected behavior**
I should see unescaped content of the assistant's message

**System information (please complete the following information):**
- node 22.8.0
- promptfoo 0.85.2
- Llama 3.1 70B with Together.ai provider

- Promptfoo version:
-0.85.2

**Additional context**
Interesting, after downgrading back to 0.85.1 this problem persists. Tried with OpenAI provider and it formats correctly.


Any chance you could provide a full config to repro this issue?  That would be very helpful!

If downgrading the package version and re-running the eval (without cache, e.g. `promptfoo eval --no-cache`) still produces this issue, then it's likely not associated with the version bump 😱 

Hi @kkarski, I’m going to close this issue due to inactivity. If you’re still experiencing the problem, please feel free to reopen it with additional details or a reproducible config, and we’ll be happy to help investigate further. Thank you!
Repository: promptfoo/promptfoo
Issue: Request: adding `-y`/`--yes` option to `promptfoo view`

When running `promptfoo view`, it prompts one with y/N to open the browser at the view's URL.

The request is to add `-y`/`--yes` kwarg to skip the confirmation and auto-open the URL.

A workaround for now is `yes | promptfoo view`.

Implemented in #166 

Wow quick turnaround, thanks for this! 👍 🚀
Repository: promptfoo/promptfoo
Issue: `export PROMPTFOO_CACHE_ENABLED=false` not disabling cache
Root cause: Validation Gap

Running:

```bash
> export PROMPTFOO_CACHE_ENABLED=true
> promptfoo eval
...  # Takes < 1 sec
> export PROMPTFOO_CACHE_ENABLED=false
> promptfoo eval
...  # Takes < 1 sec
> rm -r ~/.promptfoo/cache
> promptfoo eval
...  # Takes >> 1 sec
```

I think this means `export PROMPTFOO_CACHE_ENABLED=false` doesn't get respected.

Should be fixed with 63eebeee45bcd1266c9d09ec7a78b670cd6c3be0

The original `Boolean(process.env.PROMPTFOO_CACHE_ENABLED)`, what was the value for `PROMPTFOO_CACHE_ENABLED` supposed to be?  I guess just `1`?

And looks like your change will work, thanks

Any non-empty string is truthy, so it didn't behave correctly without parsing.  Thanks for catching this!
Repository: promptfoo/promptfoo
Issue: Python assertions - Improve debugging and logging
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I am trying to use Python assertions via an external file (see https://www.promptfoo.dev/docs/configuration/expected-outputs/python/#external-py) but it is a black box and there seems to be no way to `print` or `logger` statements for progress related stuff like it seems to work for Python providers (https://www.promptfoo.dev/docs/providers/python/#viewing-python-output).

When I try to use it the same way via setting env `LOG_LEVEL=debug`, I only see the Typescript debug log statements before and after (see https://github.com/promptfoo/promptfoo/blob/a852acdadab32cbd0b1fd63c684450cec0aef06b/src/python/wrapper.ts#L37-L39). 

**Describe the solution you'd like**
It would be great to have one of both solutions:
1. Better debugging capacities (I have tried similar things like those described here: https://github.com/promptfoo/promptfoo/issues/955)
2. Not erasing `print` or `log` statements

**Describe alternatives you've considered**
* Using a remote debugger like [remote-pdb](https://pypi.org/project/remote-pdb/) but that's a lot of extra effort and cumbersome

**Additional context**
* IDE: VSCode 1.92.2
* OS: MacBook (Sonoma, 14.6.1)
* Python 3.12.5
* Terminal: Integrated of IDE
* Promptfoo 0.86.1 (installed via [volta](https://volta.sh/))

![image](https://github.com/user-attachments/assets/d1577d1c-695c-40ae-b471-fed99322de63)


Facing the same issue. Think its in progress https://github.com/promptfoo/promptfoo/issues/1705 @hci2 

@hci2 I've never used remote-pdb and am interested in trying this out. Do you have a guideline on how to use that?

@albertlieyingadrian Sorry for the delay. You can npm link my branch https://github.com/promptfoo/promptfoo/compare/feat/python-logging?expand=1 

I am working on fixing the tests now and will issue a release when done.

You can also write to something other than stdout in your current python file (for example, a file, a websocket, etc.) if you want to do it on main 

@albertlieyingadrian
No, I haven't looked into it but I just came across it through this comment https://github.com/promptfoo/promptfoo/issues/955#issuecomment-2289592449. It seems not so difficult to try out from what I see.

Awesome and thanks for your work guys. This tool is really promising and I like the YAML configuration approach compared to other toolings. 👏 

Hey @hci2, thanks for your patience! We just released the fix in [0.91.2](https://github.com/promptfoo/promptfoo/releases/tag/0.91.2), which improves logging for Python assertions. You can now view the outputs by setting the --verbose flag or using LOG_LEVEL=debug. Please give it a try and let us know how it works for you. Feel free to comment or reopen the issue if you encounter any further issues.
Repository: promptfoo/promptfoo
Issue: Self Hosting: Fetching shared link does not work, Cannot GET /eval/<id>
Root cause: Configuration Error

**Describe the bug**
The shared link produced now has the structure:
```
 Evaluation complete: https://promptfoo.mycorp.net/eval/eval-2024-09-12T14:50:30
```
however trying always returns 400 `Cannot GET /eval/eval-2024-09-12T14:50:30` 

I am able to see the report in the UI at `https://promptfoo.mycorp.net/eval/?evalId=eval-2024-09-12T14:50:30`


**To Reproduce**
Self hosting
- Build 0.86.1 docker image 
- run it with set envvar 
```
PROMPTFOO_REMOTE_API_BASE_URL=https:///promptfoo.mycorp.net
```


**System information (please complete the following information):**

- Promptfoo version: 0.86.1

I can clearly see there is a huge pace in versions, but I am struggling to find a combination that works for sharing the reports. Do you have any pointers how this is done?


Hi @AlexRRR, can you try the latest Dockerfile on main?  That should solve the issue, which is a UI bug in how the share ID is presented.

Thanks, that did the trick. I had some versions mixed up so it did not work at the first attempt (had to edit this comment). 

However I found a small bug while building this commit's docker file:
```
26.60 **************************************************
26.60 Building next.js
26.60 **************************************************
26.78 Attention: Next.js now collects completely anonymous telemetry regarding usage.
26.78 This information is used to shape Next.js' roadmap and prioritize features.
26.78 You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL:
26.78 https://nextjs.org/telemetry
26.78 
26.92 - info Creating an optimized production build...
63.71 thread '<unnamed>' panicked at 'internal error: entered unreachable code', /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/rayon-core-1.11.0/src/job.rs:230:32
63.71 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
63.73 fatal runtime error: failed to initiate panic, error 5
63.78 Aborted
64.07 cp: no such file or directory: src/web/nextui/out/*
```
I've just done `touch src/web/nextui/out/foo`  and then dockerfile worked again.




Hi @AlexRRR, we’ve just shipped [0.88.0](https://github.com/promptfoo/promptfoo/releases/tag/0.88.0), which includes a fix for the shared link issue you reported and the missing directory. It should now be resolved. Thanks again for bringing this to our attention, and let us know if you run into any further issues!

Thank you so much!