zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Bug (?): escaping newline in `prompts`?
Root cause: Documentation Deficiency

With `promptfoo==0.56.0` and the below `promptfooconfig.yaml`:

```yaml
prompts: blah\n\nblah
```

Running `promptfoo eval --verbose` with OpenAI, I see this:

```none
Calling OpenAI API: {...,"messages":[{"role":"user","content":"blah\\n\\nblah"}],...}
```

I am sort of just checking, but when sending `content` to OpenAI, we should be escaping the newline like `\\n`? I would think we don't want to escape the newlines:

```python
>>> print("blah\\n\\nblah")
blah\n\nblah
>>> print("blah\n\nblah")
blah

blah
```

This is actually just YAML working as intended (you can try it on any [yaml->json converter](https://onlineyamltools.com/convert-yaml-to-json)).  If you wrap it in quotes, it will use the actual newline character:

```yaml
prompts: "blah\n\nblah"
```

TIL of this behavior, thanks for sharing! This [Stack Overflow answer](https://stackoverflow.com/a/77090068) helped me understand it too.

Trying out the quoted route as suggested, I confirmed the `\n` isn't escaped:

```yaml
prompts: "blah\n\nblah"
```

```none
Calling OpenAI API: {...,"messages":[{"role":"user","content":"blah\n\nblah"}],...}
```

As this isn't actually a bug after all, going to close this out. Thanks again 👍 

---

An aside, is this is also equivalent to:

```yaml
prompts: >-
  blah


  blah
```
Repository: promptfoo/promptfoo
Issue: Allow separation of 'tests' vs. 'scenarios'
Root cause: Unimplemented Feature Gap

Another feature that we think would be desirable is the ability to run a provided number of tests against 'test cases' or 'test data'.

For example, there may be the following definition:
```yaml
tests:
  - description: Is Valid JSON
    assert
      - type: is-json
  - description: Similarity
    assert:
      - type: similar
```

This means there are two tests that will be run. In our case, there could be 5-10 'scenarios' we would like to run, both with a somewhat different input and different expected output.

Rough idea of what a config with 'scenarios' could look like:

```yaml
prompts: [prompts.txt]
providers: [openai:gpt-3.5-turbo]
scenarios:
  - testData: testData1.txt
    expectedOutput: output1.txt
    expectedSimilarity: 0.8
  
  - testData: testData2.txt
    expectedOutput: output2.txt
    expectedSimilarity: 0.9
  
  - testData: testData3.txt
    expectedOutput: output3.txt
    expectedSimilarity: 0.5

tests:
  - description: Is Valid JSON
    vars:
      testData: {{testData}}
    assert:
      - type: is-json
      - type: javascript
        value: typeof JSON.parse(output) === 'object'

  - description: Meets Expected Output
    vars:
      expectedOutput: {{expectedOutput}}
    assert:      
      - type: similar
        value: {{expectedOutput}}
        threshold: {{expectedSimilarity}}
```

This would result in a total of 6 tests being run: 3 scenarios, 2 tests, 1 provider, 1 prompt.

We also would love to be able to import variables from text files so the yaml test configuration can be cleaner.

Interested in thoughts on this one, and I can help out where needed.

Picking this one up soon - thinking about ways to clean up:
- the number of configs (enabling multiple test suites)
- the number of test cases (enabling better importing/organization for test cases and vars)

Appreciate it! 

Striving to achieve a cleaner config with a lot of tests has been the most challenging aspect so far. We used `$ref` in our config extensively.

I definitely like the idea of being able to run multiple `TestSuite` objects, one for each dataset. All those test suites could use the exact same configuration.

Closing this as complete!  Scenarios are documented here: https://promptfoo.dev/docs/configuration/scenarios
Repository: promptfoo/promptfoo
Issue: Caching concurrent calls in Python provider does not use new caching behaviour
Root cause: Architectural Constraint

The code for handling concurrent caching was updated in https://github.com/promptfoo/promptfoo/pull/703 for any code which used the `fetchWithCache` function. However classes such as the [PythonProvider](https://github.com/promptfoo/promptfoo/blob/main/src/providers/pythonCompletion.ts#L43) implement their own calls to the cache so they don't make use of the fix from #703.

This means that these other classes will still have the bug mentioned in #703.

Hi @JonnySpruce,

Sorry for not closing this sooner - this was fixed by PR #3507 which was merged in March 2025 (version 0.107.6).

The Python provider now properly uses the `getCache()` and `isCacheEnabled()` utilities from the centralized caching system, so it benefits from the same concurrent caching behavior as other providers.

Thanks for reporting this! Feel free to comment or re-open if you have any questions.
Repository: promptfoo/promptfoo
Issue: CLI --first-n only runs the first test case regardless of specified argument when using test CSV
Root cause: Algorithmic Error

I'm using promptfoo 0.56 and testing with `/examples/simple-csv`

1. Clone repository
2. `npm i`
3. `cd examples/simple-csv`
4. `npx promptfoo eval`. Observe there are four output rows.
5. `npx promptfoo eval --first-n 3`. Observe that there is only one output row. I expected the first three.


My fault for only testing this on --n 1. Fix is in #734
Repository: promptfoo/promptfoo
Issue: Request: ability to exclude `Description` from `promptfoo view` and persist UI settings across refresh
Root cause: Unimplemented Feature Gap

I would like to exclude the `Description` column from `promptfoo view`. Is this possible?

I guess if it's not yet implemented, a request could be to exclude columns like `Description` or keys in `vars` from `promptfoo view`'s table

I'd second this - some way to make the web UI configuration persist between page refreshes and server restarts would be great!

Maybe I'm misunderstanding the original request, but this is already possible using the "column selector"

![image](https://github.com/promptfoo/promptfoo/assets/310310/53fcc9d8-c106-4c58-8607-3625b1804055)

It is true that this selection is not persisted though

Oh you're right there per the `Variables`. However, but you can't turn off the `Description` field

<img width="667" alt="screenshot of `promptfoo view`" src="https://github.com/promptfoo/promptfoo/assets/8990777/0d373f24-654a-4ad1-9dce-5a6f55a9021a">

See lower left corner of the above screenshot

Do you think you can make the `Description` toggle-able as well?

Fix coming in #1095!

@typpo thanks for #1095! After pulling in v0.69.2 to try this out, I don't see where in the UI to toggle the description

hey james :) it appears in the `Columns` menu for evals that have Descriptions

<img width="858" alt="image" src="https://github.com/promptfoo/promptfoo/assets/310310/f7e918eb-2d0b-477a-bcaa-4221e5c7311f">


I am not seeing that option in my run today: https://app.promptfoo.dev/eval/f:96a89d44-4ada-4577-9cd2-7bf8bc7f9433/

Maybe I failed to properly pull in the latest `promptfoo`

I confirmed with 0.69.2 locally that what you're seeing (checkbox for description), I am not seeing that same checkbox. Note the web UI I am using was made by `--share`

<img width="432" alt="image" src="https://github.com/promptfoo/promptfoo/assets/8990777/efcbcd0a-79e0-45a1-822b-530217a71f68">
Repository: promptfoo/promptfoo
Issue: Request: support `--max-concurrency=auto`

There is a `pytest` plugin for running tests concurrently using multiprocessing. That plugin supports automagically inferring the number of parallel processes to spawn: `pytest -n auto`.

I know `promptfoo` supports concurrency using `--max-concurrency` (defaulted as of `0.55.0` to 4).
- What type of concurrency does `promptfoo` use? Is it multiprocessing?
- Can we support `--max-concurrency=auto`, and do basic machine-tailored inference for an optimal amount of concurrency?

Evals are I/O bound with respect to the LLM APIs, so in practice it is not
desirable to set it based on your number of CPUs but instead your
network, rate limits, etc.  The concurrency uses node's event loop as
opposed to multiprocessing.

On Mon, Apr 29, 2024 at 10:55 PM James Braza ***@***.***>
wrote:

> There is a pytest plugin for running tests concurrently using
> multiprocessing. That plugin supports automagically inferring the number of
> parallel processes to spawn: pytest -n auto.
>
> I know promptfoo supports concurrency using --max-concurrency (defaulted
> as of 0.55.0 to 4).
>
>    - What type of concurrency does promptfoo use? Is it multiprocessing?
>    - Can we support --max-concurrency=auto, and do basic machine-tailored
>    inference for an optimal amount of concurrency?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/737>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJUAEETTZI6FWRGSS2DY74W4FAVCNFSM6AAAAABG7SRGKKVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3TANJWGU3DCNY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Okay I see. So I guess it's not really feasible to auto-infer a concurrency count.

I will open a docs PR to at least add that the concurrency is managed via a single process event loop
Repository: promptfoo/promptfoo
Issue: Docs request: where does histogram come from?
Root cause: Documentation Deficiency

I have three possible scores: 0, 0.1, and 1 for a Python assertion, and two basic assertions.

```yaml
providers:
  - openai:chat:gpt-4-0613
  - openai:chat:gpt-4-turbo-2024-04-09
  - anthropic:messages:claude-3-sonnet-20240229
defaultTest:
  assert:
    - description: was answered
      type: not-icontains
      value: cannot answer
    - description: has sentences
      type: javascript
      value: output.length > 20
    - description: check value
      type: python
      value: file://assert.py
```

At the top of my `promptfoo view`, I see bins around 0.6 and 0.7, which isn't quite making sense to me:

![screenshot of histogram](https://github.com/promptfoo/promptfoo/assets/8990777/d79d8e29-b5ae-4452-a575-cfcd88219922)

The request is, can we add a little description such that this figure is easy to understand.
- I have three different model providers, is that where Prompt 1 (red), Prompt 2 (blue), and Prompt 3 (green) come from?
- Why does the histogram show scores of `0.6` and `0.7`? Is that like a sum of multiple assertions' scores?

I now understand that I have three assertions:

- Two binary ones: can be score 0 or 1
- One custom assertion: can be score 0, 0.1, 1

I realized the histogram plots mean score: 0.7 = (1 + 1 + 0.1) / 3

That being said, I still think perhaps `promptfoo` can add a little info bubble or hover-over/tooltip that explains this.

Feel free to close this out if uninterested
Repository: promptfoo/promptfoo
Issue: Random sampling of n data rows
Root cause: Unimplemented Feature Gap

The `-n` / `--first-n` flag was recently introduced to enable running fewer than all test cases on a given run. This runs the first n test cases.  Would it be worth considering another flag that enables random sampling of n test cases from the complete set? This would be particularly convenient when testing against very large datasets stored in, e.g., `.csv` files.  At the moment, it's necessary to copy-paste just the cases you want to run into a separate file just for that purpose.

If this feature is a good fit for promptfoo, would it be possible to make it available not just from the CLI, but also in the `.yaml` test config?  Relatedly, would it also make sense to enable use of `--first-n` from the `.taml` config as well?

Thanks!

This is an interesting idea, and wouldn't be too difficult to implement :eyes: 

Tangentially related heads up - `--first-n` is being renamed to `--filter-first-n` in the next release to match other new filtering options.

#2115 adds random sampling with `--filter-sample`!
Repository: promptfoo/promptfoo
Issue: Request: CLI arg to suppress exit code 100 in `promptfoo eval`

I am using `promptfoo` to evaluate a benchmark, not a binary pass/fail outcome. The benchmark is quite hard, so it's basically guaranteed at least one test case will fail.

I can see `promptfoo eval` will return exit code 100 if there are 1+ test case failures. This is actually documented and makes sense ([docs link](https://www.promptfoo.dev/docs/usage/command-line/#promptfoo-eval)).

The request is it would be useful if there was a CLI flag for `promptfoo eval` that would allow test case failures. In other words, even if there's a test case failure, the exit code would be 0. Exit code 1 is used for all other failure modes still.

Possible naming ideas are `--allow-failed-cases` or `--exit-code-ignores-failures`. Thanks for your consideration 👍 

---

I am using `promptfoo==0.55.0`.

A workaround is manually suppressing the exit code 100 using `promptfoo eval || [ $? -eq 100 ]`.

#751 should solve this by allowing you to set `PROMPTFOO_FAILED_TEST_EXIT_CODE=0`
Repository: promptfoo/promptfoo
Issue: CORS error when using self hosted server
Root cause: Configuration Error

I keep running into a cors error if I access my self hosted server that is on a ec2 instance. If I disable cors on my browser however, I am able to reach out to it. I do see the app.use (cors()) in the server.ts, but I'm not sure if this error is coming from that. Does the middleware or nextjs config need to handle cors?


For additional context, this error happens when I use the share button.

Here is the curl response from the header:

HTTP/1.1 200 OK
cache-control: s-maxage=31536000, stale-while-revalidate
content-length: 7257
content-type: text/html; charset=utf-8
date: Tue, 30 Apr 2024 21:37:12 GMT
etag: "162s16q4lu25ll"
Vary: RSC, Next-Router-State-Tree, Next-Router-Prefetch, Next-Url, Accept-Encoding
x-nextjs-cache: HIT
x-powered-by: Next.js
Connection: keep-alive
Keep-Alive: timeout=5

Not sure about this one.  As far as I'm aware there isn't any special cors stuff you'd have to do on your side.  The self-hosted version is meant to be bundled as a single app using the Dockerfile, whereas `server.ts` is the devtool that runs locally.  So you might want to try running the Dockerfile/next.js app instead of running server.ts/promptfoo view.

I'm using the nextui that is built from the DockerFile, I was able to fix this by making the middleware.ts handle the cors and preflight responses. Not sure if this is a use case that you would need here though tbh, if it is happy to contribute here.

I'm using the nextui that is built from the DockerFile, I was able to fix this by making the middleware.ts handle the cors and preflight responses. Not sure if this is a use case that you would need here though tbh, if it is happy to contribute here.

Oh great to hear you found a fix.  I would welcome the contribution if you
have the time to PR the change :)

On Thu, May 2, 2024 at 12:10 PM Karthik Nathan ***@***.***>
wrote:

> I'm using the nextui that is built from the DockerFile, I was able to fix
> this by making the middleware.ts handle the cors and preflight responses.
> Not sure if this is a use case that you would need here though tbh, if it
> is happy to contribute here.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/743#issuecomment-2091322499>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJV5SD576I6W7HKMOFLZAKFRTAVCNFSM6AAAAABHBA42U2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOJRGMZDENBZHE>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: Multiple prompts in a single prompt file when using multiple prompt files
Root cause: Unimplemented Feature Gap

In the documentation it says:

"If you have only one file, you can include multiple prompts in the file, separated by the separator `---`. If you have multiple files, each prompt should be in a separate file."

Multiple prompts in a single file is very convenient, and it's a shame to lose the ability to do this if you have more than one prompt file.  Removing this limitation would allow the use of multiple prompts and multiple prompt files together, meaning you could have, e.g., a `.txt` file containing your string-formatted prompts, and a `.json` file containing JSON-formatted message pairs.

Hi @alexclaydon! Thanks for the great suggestion. This works now! Please give it a try. 

Thank you very much!  We will certainly do so 🙏🏻
Repository: promptfoo/promptfoo
Issue: pass logger as callApi 
Root cause: Unimplemented Feature Gap

In order to have the custom provider fit into the existing logging infrastructure, add the `logger` object in the Api options call.


Great suggestion. I've also included some caching utils in #748, as that is a request I've seen elsewhere.

This change is included in [0.57.0](https://github.com/promptfoo/promptfoo/releases/tag/0.57.0), just published on npm
Repository: promptfoo/promptfoo
Issue: add evalId in generate OutputFile
Root cause: Unimplemented Feature Gap

If we could have the `evalId` of a run in the generated output file (OutputFile), it would be possible to get the VSCode test controller UI to reopen the promptfoo web ui on the evluation page using the `eval?evalId=...` route.

The context is that I have a testcontroller in visual studio code that provide a nice front end to the tests. If I can store the eventId for a run, I can open the web ui directly in vs code for a nice integrated experience.

Your PR is released in 0.62.0!

Great news. Thanks for pushing it through.
Repository: promptfoo/promptfoo
Issue: Feature Request: Assertion Sets
Root cause: Unimplemented Feature Gap

There are meaningful test cases in which at least one of as set of assertions needs to pass, but not all of them need to pass. 

it would be useful to be able to define an assertion that defines sub-assertions and the pass criteria for the assertion set.

Each of the assertions in the set could calculate and produce metrics like other assertions, but would be individually exempted from having a failing result cause a failure for the record.

Example sketch:
```yaml
assert:
    - type: context-faithfulness
      threshold: 0.9
      metric: Groundedness
    - type: assertion-set
      value: >= 1 pass
      assert: 
          - <assertion A>
          - <assertion B>
          - ...
```

I can take a look at implementing this. I haven't worked on assertions yet. Would love to learn that part of the codebase

I would gladly accept a PR!

PR in the works. https://github.com/promptfoo/promptfoo/pull/765

For now I want to ensure that `assert-set` cannot contain `assert-set` so I'm going to add a validation step to validate `TestSuite` assertions.
Repository: promptfoo/promptfoo
Issue: Multiline vars replacement break indentation in config generation
Root cause: Algorithmic Error

## Current situation

If the promptfile contains this:

```yaml
- role: user
  content: |
    The file content is:
    ```
    {{fileContent}}
    ```
```

And the configfile contains this:
```yaml
tests:
  - vars:
      fileContent: |
        // Sample comment
        export const hello = "word"
```

The resulting output is:
```yaml
- role: user
  content: |
    The file content is:
    ```
    // Sample comment
export const hello = "word"
\```
```

Which is invalid

## Expect result

A multiline var value should be indented as well:
```yaml
- role: user
  content: |
    The file content is:
    ```
    // Sample comment
    export const hello = "word"
    ```
```

@yleflour, this is working now! It looks like the issue was resolved some time ago, so I’ll go ahead and mark it as closed. Thanks for your input, and please let me know if you encounter any other issues!
Repository: promptfoo/promptfoo
Issue: Shouldn't "gemini-1.5-pro-latest" be named "gemini-1.5-pro-preview-0409"?
Root cause: Configuration Error

I am new to this awesome tool, so I don't know if the above makes sense. But using "vertex:gemini-1.5-pro-latest" for me results in a an API error while using "vertex:gemini-1.5-pro-preview-0409" shows a (wrong?) warning but works. Please let me know if I am doing something wrong. Otherwise you probably want to rename the following code location:
https://github.com/promptfoo/promptfoo/blob/ac542d1801a819fd8ad267188848814c1ecc6a10/src/providers/vertex.ts#L157

![screenshot](https://github.com/promptfoo/promptfoo/assets/5207269/71415b43-31db-4677-880a-ce29a5b69572)


It looks like maybe Google updated or swapped the model strings (listed [here](https://cloud.google.com/vertex-ai/generative-ai/docs/learn/model-versioning#gemini-model-versions)).  I've added `gemini-1.5-pro-preview-0409`.  The warning is harmless and can be ignored.  Thanks for flagging this!
Repository: promptfoo/promptfoo
Issue: not-equals is not working
Root cause: Algorithmic Error

# Expected Behavior

`not-equals` assertions should behave as the name suggests (negate an equals).

# Current Behavior

The `not-` prefix is getting ignored for `equals` assertions. Therefore it behaves like an `equals`.

# Failure Information (for bugs)
## False negative
```
description: 'Eval'
prompts:
  - prompts/prompt.json
providers:
  - id: openai:gpt-3.5-turbo
    config:
      temperature: 1
      max_tokens: 4096
      best_of: 1
tests:
  - assert:
    - type: not-equals
      value: saddfajdsa
```
![image](https://github.com/promptfoo/promptfoo/assets/167216434/d222160d-448e-49fc-9236-a824ceb25be2)
## False positive
```
description: 'Eval'
prompts:
  - prompts/prompt.json
providers:
  - id: openai:gpt-3.5-turbo
    config:
      temperature: 1
      max_tokens: 4096
      best_of: 1
tests:
  - assert:
    - type: not-equals
      value: Hello
```
![image](https://github.com/promptfoo/promptfoo/assets/167216434/86231e4f-02c7-459e-bb3b-808057f8ea9d)


## Bug in source
https://github.com/promptfoo/promptfoo/blob/ac542d1801a819fd8ad267188848814c1ecc6a10/src/assertions.ts#L276-L286


## Additional information
`not-contains` works as example:
```
description: 'Eval'
prompts:
  - prompts/prompt.json
providers:
  - id: openai:gpt-3.5-turbo
    config:
      temperature: 1
      max_tokens: 4096
      best_of: 1
tests:
  - assert:
    - type: not-contains
      value: saddfajdsa
```
![image](https://github.com/promptfoo/promptfoo/assets/167216434/d3c52e20-8fe1-400e-b9b9-b0631c0cc1fc)