zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Unknown format for Gemini: {"role":"user","parts" -- in promptfoo redteam eval
Root cause: Validation Gap

**Describe the bug**
I successfully run promptfoo redteam init + promptfoo redteam generate for Google VertexAI Gemini pro. However when I run promptfoo eval, an unknown format for Gemini shows up.

Also, promptfoo gives errors when I add Indirect Prompt Injection and Prompt extraction. If I remove these from redteam init, it fails on the `eval`

**To Reproduce**
nvm use 20.10.0
npm install -g promptfoo
promptfoo redteam init (+Y generate)
promptfoo redteam eval

See error:

```
Unknown format for Gemini: {"role":"user","parts":{"text":"You are a travel agent specialized in budget trips to Europe\n\nUser q}
```
My redteam.yaml looks like this:

```
description: My first red team
prompts:
  - |+
    You are a travel agent specialized in budget trips to Europe.

    User query: {{prompt}}
```

My promptfooconfig.yaml looks like this:
```
description: "My first red team"

prompts:
  - "You are a travel agent specialized in budget trips to Europe.\n\nUser query: {{prompt}}\n\n"
  
targets:
  - vertex:gemini-pro
```

**Expected behavior**
I was expecting to run the redteam eval without issues

**Screenshots**
![Screenshot from 2024-10-17 10-20-51](https://github.com/user-attachments/assets/e308e440-9b29-4a7b-a913-a8eb49789b09)

**System information (please complete the following information):**

- Promptfoo version: 0.93.3 running on nvm 20.10.0 in Ubuntu 22.04, VSCode


Hey @RubensZimbres, this warning _should_ be safe to ignore.  It was triggering erroneously and is removed in bdf01b6499ec0738861ddf9bb3f970baaf1fb676

You can test a smaller number of inputs e.g. with `promptfoo redteam eval -n 2` (first 2) just to double check that everything is going through ok.

Let me know if you run into any issues with the final result.

Hi @RubensZimbres,

The warning for the Gemini format has been fixed in version [0.94.0](https://github.com/promptfoo/promptfoo/releases/tag/0.94.0). Let us know if you encounter any further issues with the final results!
Repository: promptfoo/promptfoo
Issue: --watch triggers on change, but doesn't pull changes
Root cause: Algorithmic Error

**Describe the bug**
When using --watch, changes to promptfooconfig.yaml correctly trigger a reload, but evals are run with the initial version of the promptfooconfig.yaml that was loaded.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. promptfoo init
2. promptfoo eval --watch
3. Change promptfooconfig.yaml to include "Don't " as a prefix to each prompt (so it becomes "Don't Write a concise, funny tweet about {{topic}}"
4. Wait until eval finishes
5. Old prompts are used ("Write a concise, funny tweet about {{topic}}").

**Expected behavior**
New prompts are used ("Don't Write a concise, funny tweet about {{topic}}").

**System information (please complete the following information):**

- Promptfoo version: 0.93.3


Thanks for flagging!  #1929 fixes this bug, which was apparently introduced a couple weeks ago.  It will be included in the next release.
Repository: promptfoo/promptfoo
Issue: --watch should also trigger for changes in prompt paths
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I have a project with many large prompts. The prompts are stored in separate files in different subdirectories and are pointed to in the config. When I run promptfoo eval -w, it doesn't react to changes in the prompt files, only to changes in the config.

**Describe the solution you'd like**
-w should also watch all files linked to in config. 

**Describe alternatives you've considered**
I might write my own --w functionality and use promptfoo as a NodeJS module. 


This behavior is fixed with #1929.  Thanks!

Awesome, thanks! Will it also watch linked Google Sheets for changes? It would be amazing if it did, but if not, maybe a warning is in order?

Fix is released!  Unfortunately we can't watch Google Sheets yet
Repository: promptfoo/promptfoo
Issue: Request: support Ollama `num_predict` and/or `options` to `/generate`
Root cause: Documentation Deficiency

I am using the `similar` assertion and Ollama `/generate` (provider) and LocalAI `/embeddings` (grader), and it's working. Now it's time to really dial in the test assertions.

My assertion prompt is short (20 words) but my Ollama generated response is massive (100+ words).  I believe this is impacting my `similar` score.

Ollama has a field for this [`num_predict`](https://github.com/jmorganca/ollama/blob/v0.0.21/api/types.go#L185) that corresponds with [`n-predict` from Llama.cpp](https://github.com/ggerganov/llama.cpp/tree/master/examples/main#number-of-tokens-to-predict).  I don't see it in the current [Ollama config options](https://github.com/promptfoo/promptfoo/blob/0.24.2/src/providers/ollama.ts#L7).

You can set it via `curl` with `options`:

```bash
curl http://localhost:11434/api/generate -X POST -H "Content-Type: application/json" -d '{
         "prompt": "some prompt",
         "model": "llama2",
         "options": {
             "num_predict": 42
         }
     }'
```

Is there any way we can either:
- Add `num_predict` support to Ollama config?
- Expose `options` for a given `/generate` request?

Closing this out - there's an example here https://github.com/promptfoo/promptfoo/blob/main/examples/ollama-comparison/promptfooconfig.yaml#L8-L9

Oh awesome, I will starting using this capability tomorrow! Much appreciated
Repository: promptfoo/promptfoo
Issue: Issue with Creating a Test Scenario
Root cause: Documentation Deficiency

I tried to read through the documentation, but I'm not sure how to construct the following test scenario:

Let's assume I have a long document along with a question about the document. Using "prompt1.txt," I would like to ask the model to summarize the document and keep the most important information necessary to answer the question. Later, I want to test whether the response contains the information required for a correct answer to the question. 

Is it possible to set up such a flow using promptfoo to accomplish this?

You may need to break the scenario into two tests. One for summarization and one for asking questions.

Do you mean two separate test files or the entire tests? If so, how can it be done? I'm stuck on this.

I mean, how can I get the model's response from Promptfoo and transfer that response to another test? The only thing that comes to mind is to transfer the response manually, but that would be somewhat pointless.

> I mean, how can I get the model's response from Promptfoo and transfer that response to another test? The only thing that comes to mind is to transfer the response manually, but that would be somewhat pointless.

You can write a custom provider and do what ever logic inside.
Repository: promptfoo/promptfoo
Issue: Allow referencing a particular sheet of a Google Sheet as evaluation source
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I need to test multiple prompts with different evaluations, where evaluations:prompts have a n:1 relationship. I want to keep all evaluations in a single Google Sheet for convenience of the end-users. At the moment, when I reference a particular sheet of a Google Sheet via URL, it still grabs the first sheet.

**Describe the solution you'd like**
In config, if I put in a reference to the "bird" sheet of the example Google Sheets file and then run `promptfoo eval`, it should run tests from that particular sheet.
```
tests: https://docs.google.com/spreadsheets/d/1xHVjPg8nGU96FV-KXIhtD7GfvUzyy6Y1WW4kX8aLK9s/edit?gid=1649698752#gid=1649698752
```


**Describe alternatives you've considered**
I'll use separate Google Sheets and keep another Google Sheet as the table of contents - but that's very brittle.

Added in #1942
Repository: promptfoo/promptfoo
Issue: Ability to add notes on the UI of the eval report
Root cause: Documentation Deficiency

<img width="656" alt="Screenshot 2024-10-18 at 12 01 38 PM" src="https://github.com/user-attachments/assets/b026ebc6-7a3b-49fa-8e10-e4c11f9896da">

Would love to have the ability to add note on the UI for errors

Hey! Good news - this feature actually already exists! It was introduced in September 2024 (PR #1637), about a month before this issue was opened.

To add notes/comments to any eval result:

1. Hover over any cell in the results table
2. Click the pencil icon (you'll see an "Edit comment" tooltip)
3. Type your note and hit Save

Your comment will show up in italics below the cell content, and it's saved to the database so it persists across sessions.

**Pro tip:** If you prefix your comment with `!highlight`, it'll highlight the cell background - handy for flagging things you want to come back to!

Closing this out as part of an issue cleanup. Let us know if you have any trouble finding it or if the current implementation doesn't meet your needs - happy to reopen!
Repository: promptfoo/promptfoo
Issue: Allow running llm-graded tests multiple times to find flickering tests
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Sometimes prompts and inputs result in unpredictable LLM behaviour, especially at higher temperatures. This means that both the LLM and the evaluator might produce vastly different outputs. 

As a synthetic example, a prompt `think of a number between 0 and 1 and return it` will produce both 0s and 1s. An assert checking if the output contains 0 will sometimes pass and sometimes fail.

Another synthetic example: an assert `llm-rubric: does this contain harmful material` might produce different results for edge cases. The assert will fail intermittently even if the inputs are the same.

**Describe the solution you'd like**
- A config flag that allows running a particular prompt against the same assert more than once (n times).
- A config flag that allows running a particular test against the same prompt more than once (n times).

**Describe alternatives you've considered**
I just run the eval several times and see if it flickers, but this is very brittle, and flickering tests still come through.

**Additional context**
This has been one of the main challenges when building prompts for long-form conversational agents for Enterprise customers. 

A workaround is available here: https://github.com/promptfoo/promptfoo/issues/1888
Repository: promptfoo/promptfoo
Issue: Support AWS Bedrock access key in config file
Root cause: Unimplemented Feature Gap

I tried to set `aws_access_key_id` and `aws_secret_access_key` in promptfooconfig.yaml, but it does not work. Set env is ok. Not sure if set in config supported?

Hi @zhlmmc,

Support for AWS credentials in the config file has been added in [#1936](https://github.com/promptfoo/promptfoo/pull/1936). Please update to version [0.94.0](https://github.com/promptfoo/promptfoo/releases/tag/0.94.0), and it should now work as expected.

Let us know if you have any further issues!
Repository: promptfoo/promptfoo
Issue: __description is ignored when using Google Sheets
Root cause: Configuration Error

**Describe the bug**
If a Google Sheets file contains a __description column, its content is ignored.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

Use the following config:
```
# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
description: "Line break test"

prompts:
  - 'A short description of Adele'
  
providers:
  - "openai:gpt-4o"
  
tests: https://docs.google.com/spreadsheets/d/1--15puWzEGnhCcuuLG-NPsTyvS6RHCCcrzA5BJl8Lbs/edit?gid=0#gid=0
```

Run the eval and view it (either in browser or in CLI). Link to shareable results: https://app.promptfoo.dev/eval/f:d26ab224-2964-491b-94c1-e4f4b400b433/

There is no "Description" column in the table.

**Expected behavior**
A "Description" column in the table

**System information (please complete the following information):**

- Promptfoo version: 0.93.3

Should be fixed with #2000.  Thanks for flagging - found an underlying issue with all descriptions after the latest db migration
Repository: promptfoo/promptfoo
Issue: Question: is `---` in `prompts.txt` for one conversation many prompts, or many conversations one prompt?

Please forgive me for a noobish question, but I am questioning my understanding of something.

Within `prompts.txt`, is `---` designed for:

- Delimiting within one conversation (one conversation, many prompts)
- Delimiting across conversations (many conversations of one prompt)

----

To reword, from https://promptfoo.dev/docs/configuration/parameters#prompt-files:

```txt
Translate the following text to French: "{{name}}: {{text}}"
---
Translate the following text to German: "{{name}}: {{text}}"
```

Is this like having a two chat conversation, or two separate one chat conversation?

It's split into multiple prompts.  It's as though you had two prompt files containing each line.

Thank you!  Opened a docs PR to clarify this.
Repository: promptfoo/promptfoo
Issue: Better support for rate limited calls
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

We are using OpenAI's GPT-4o model for evaluation, and are often getting rate limited. This manifests as the run for a prompt returning "API call error: TypeError: Cannot read properties of undefined (reading 'message')". Then I re-run the same config, and thanks to caching of the not-failed responses, it usually gets farther along.

**Describe the solution you'd like**

I think my ideal solution would be to wait when rate-limited and try again after the rate limit has expired, and to retry queries that fail in this way (perhaps up to a certain amount.)

It's also possible there are other intermittent errors (503? 5xx?) that we could handle in a similar way. But this is the most painful one for us at the moment.

**Describe alternatives you've considered**

Our workaround is to just re-run it manually until there are no API-related errors, but this seems less than desirable. It's possible we could parse through the recent runs to detect this and re-run with delay, but that seems like something the framework should handle. I've also turned down the concurrency and used the delay parameter, but this just makes it happen less often at the cost of potentially more delay than needed.

**Additional context**

Happy to try to look into this, especially if you have some pointers on where we could do this.

Based on searching the codebase for that error string, it looks like the code for my case is around:
 https://github.com/promptfoo/promptfoo/blob/5e500625e3da15478e9f2838323a7d832bb23958/src/providers/openai.ts#L562

(although there are other providers where this would likely be useful as well)

I also see some tests for rate limiting in: https://github.com/promptfoo/promptfoo/blob/5e500625e3da15478e9f2838323a7d832bb23958/test/rateLimit.test.ts

And the associated retry logic is in https://github.com/promptfoo/promptfoo/blob/5e500625e3da15478e9f2838323a7d832bb23958/src/fetch.ts

Currently it looks like it's looking at a few header values, but not the ones in https://platform.openai.com/docs/guides/rate-limits/rate-limits-in-headers (which allow for request-level and token-level limits). My guess is OpenAI updated their headers at some point. So I wonder if updating the code to use that would fix this issue using the current retry logic.

Hey @panozzaj, thanks for the pointers. I was in that area of the codebase anyway so I made some improvements in #1981.  Hopefully that helps
Repository: promptfoo/promptfoo
Issue: Show score and reason even when passes llm-rubric
Root cause: Unimplemented Feature Gap

Noticed when using llm-rubric it only displays the reason when assertions fail, however I would like to also see the reason the model gives even when it passes to get a understand why.

Update rendering UI to show reason and score

Let me know if this is a good idea and I could try to come up with a PR too.


The reason should be visible in the Details (magnifying glass) view.  Do you mean that you want to see it shown in the eval table itself?

<img width="1185" alt="image" src="https://github.com/user-attachments/assets/818cca63-0811-4037-8c08-6f4ed8c1672d">


yep, thats what i meant


We'd love this feature! 
Our LLM-Rubric eval is rating responses on multiple dimensions and we'd like to see which one didn't apply to any given response. We dont want to run 5 llm-rubric asserts and call the grader multiple times, because it gets costly very quickly on a bigger test set.
Repository: promptfoo/promptfoo
Issue: [BUG]No tableOutput.html file in dist.
Root cause: Validation Gap

```
Error: ENOENT: no such file or directory, open 'E:\dev\promptfoo\dist/tableOutput.html'
    at Object.openSync (node:fs:599:3)
    at Module.readFileSync (node:fs:467:35)
    at writeOutput (file:///E:/dev/promptfoo/dist/util.js:51:29)
    at Command.<anonymous> (file:///E:/dev/promptfoo/dist/main.js:136:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  errno: -4058,
  syscall: 'open',
  code: 'ENOENT',
  path: 'E:\\dev\\promptfoo\\dist/tableOutput.html'
}
```
Repository: promptfoo/promptfoo
Issue: Request: `--output=file.txt` defaulting to non-CSV
Root cause: Unimplemented Feature Gap

I discovered using `--output=file.txt` as being pretty non-readable, mainly cuz the CSV output isn't that friendly overall.

```txt
"{""raw"":""<redacted>{{ area }}<redacted>{{ area }}<redacted>"",""display"":""<redacted>{ area }}<redacted>{{ area }}<redacted>""}",area,expected,result
"[PASS] (0.65)  <redacted>",<redacted>,"<redacted>
",<redacted>

```

Thoughts on having `.txt` extension default to YAML formatting, as opposed to CSV formatting?

That's fair - not sure how many people use .txt outputs.  I've updated it accordingly in https://github.com/promptfoo/promptfoo/commit/cd9533f9daa5dc30317a8b1fd739fa589998f833

Thanks again!
Repository: promptfoo/promptfoo
Issue: Model Not Found Error with Multiple IDs in Red Teaming
Root cause: Configuration Error

I'm encountering an error when trying to run the red teaming feature of Promptfoo with specific model IDs. The error message indicates that the model does not exist or access is denied, even though I’ve verified the IDs individually.

To Reproduce
Steps to reproduce the behavior, including example Promptfoo configurations:

1. Use the following configuration in promptfoo (.yaml) file:

![image](https://github.com/user-attachments/assets/4840b8be-1e15-40e5-8e11-1dcb4317446e)

2. Run the command: promptfoo redteam run

3. See the following error message:
![image](https://github.com/user-attachments/assets/d9396b0a-6adf-4ad4-b41d-49cdeaff7899)

Expected behavior
I expected the command to run successfully and return results for the specified models, without encountering the "model not found" error.

System information:

- Promptfoo version: 0.94.1

Additional context
I have tried each model ID separately, and I am still receiving the same error message. This suggests there may be an issue with model accessibility or the IDs being used.


![gpt](https://github.com/user-attachments/assets/06c0fe5d-1e8f-4323-91ed-31fe38e1c87c)





Hmm, it's using your OpenAI API key to try to access a standard model, `gpt-4o-2024-05-13` ([OpenAI docs](https://platform.openai.com/docs/models/gpt-4o)).  Is it possible that your OpenAI organization does not have access to this specific model for some reason?

In that case, please try 

```
promptfoo redteam run --remote
```
And let me know how that works

Thanks for the screenshots! The error is happening during purpose extraction, not when calling your target. Promptfoo uses a separate "attacker" model for generating tests, and in v0.94.1 that defaulted to `gpt-4o-2024-05-13`. Your OpenAI account doesn't have access to that specific snapshot.

Easiest fix - use our hosted inference:

```bash
promptfoo redteam run --remote
```

Or set a model your account can access:

```yaml
redteam:
  provider: openai:chat:gpt-4o
```

Docs: https://www.promptfoo.dev/docs/red-team/configuration#providers

Closing since this is a config issue rather than a bug. Let us know if `--remote` doesn't work for you.
Repository: promptfoo/promptfoo
Issue: Support authenticated HTTP providers
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
I am using the promptfoo web interface. I want to query an HTTP provider at an API that I host, which requires authentication.  My evaluation results come back with every query response as "401 unauthorized", since the Promptfoo server making the api call is not sending a valid bearer token. 

**Describe the solution you'd like**
I'd like to be able to register my promptfoo deployment as a client in keycloak, so that when I deploy my promptfoo instance I can provide it with its keycloak client credentials.  Then I would like the promptfoo server to go through a proper auth flow to retrieve an authorization token from keycloak which it can then pass to my HTTP provider when it makes a query during evaluation.

**Describe alternatives you've considered**
For the time being I will probably work off of a fork and implement custom logic in the express server to pass my Bearer token along, but that is not generalizable.


Hi Ben, would it work to set the appropriate header in your HTTP provider?  https://www.promptfoo.dev/docs/providers/http/

For example
```yaml
providers:
  - id: 'https://example.com/generateWithAuth'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
        'Authorization': 'Basic {{encodedAuth}}'
      body:
        prompt: '{{prompt}}...'
```

Would that require the user to copy their token into the config? How is {{encodedAuth}} populated?

You can either hardcode it (or put it in `defaultTest` as a variable), or you _should_ be able to use the `env` global to pass in an environment variable.  e.g. `env.PROMPTFOO_AUTH_TOKEN`
Repository: promptfoo/promptfoo
Issue: Error when using BASIC auth in PROMPTFOO_REMOTE_API_BASE_URL
Root cause: Validation Gap

**Is your feature request related to a problem? Please describe.**
I'm using self-hosted Promptfoo with BASIC authentication on our API server. When setting PROMPTFOO_REMOTE_API_BASE_URL with BASIC auth credentials, I get this error:

```
promptfoo share


...
Request cannot be constructed from a URL that includes credentials: https://XXXXX/health  at node:internal/deps/undici/undici:13185:13 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
```

- Steps to reproduce:
  - Set `PROMPTFOO_REMOTE_API_BASE_URL` with BASIC auth like this:
  - https://username:password@example.com

Is it possible to use promptfoo share with a server that requires BASIC authentication? If not, what authentication method would you recommend?
Environment:

- Promptfoo Version: 0.94
- Self-hosted deployment

### Describe the solution you'd like
I would like to be able to use promptfoo share with a BASIC auth-protected API server. Either by:

- Supporting credentials in the URL
- Or providing an alternative way to specify BASIC auth credentials (perhaps as separate environment variables)

### Describe alternatives you've considered
Setting up a separate non-authenticated endpoint just for promptfoo (adds complexity)


@tan-z-tan We’ve released a fix for this. Please try the latest version and let us know if you’re still experiencing any issues. Feel free to comment or reopen the issue if you need further assistance.

@mldangelo Fixed. Thank you!