zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Renaming "__expected"s "eval" keyword

I noticed you're using the term "evaluation" where many testing frameworks typically use "assertion".

Personally, I like evaluation better too, however since there's already __expected's use of "eval" (js), I think having different terms would be more appropriate for a clean and intuitive API.

I propose: 
- renaming __expected "eval" keyword to "run", "exec" or my favorite "fnc" indicating an anonymous functions
- The "eval" regex matcher remains in the code for legacy reasons, but the docs only contain the new keyword

If you concur, I can setup a PR.


I'm open to this -- I wonder if `js` might be the most useful prefix, because it indicates which language it actually expects?

Edit: more ideas :stuck_out_tongue:  - `script`, `lambda`, `fn`, `code`

Indeed, I find `js` also visually appealing as a prefix. Ideally though it would be best to use something that conveys the most meaning, and I think, since it's a node module, js as language is somewhat implied.

`lambda` would describe that it is an anonymous function - and would fit if the user can't also put in a named function.

Otherwise I think 'fnc' or 'fn' would be the most useful. Maybe even 'jsFnc' (or 'jsFun' 😁) but that might be too long/descriptive?

Let me know what you think is the best term, and I'm happy to add a PR then. 👨‍💻

Sounds good!  Let's go with `fn` :grin:
Repository: promptfoo/promptfoo
Issue: [Web UI] Increase robustness of markdown rendering for tables
Root cause: Unimplemented Feature Gap

I have enabled the 'Render model outputs as Markdown' option for the UI. However, it does not render tables. e.g. the below is rendered as is:

Here is the action under corporate:

```
| Domain | Function | Activity | Records | Disposal Action |
| --- | --- | --- | --- | --- |
| Corporate | xxx and xxx Relations | Consultation | qqq | yyy |
| Corporate | qq and qq Relations | Consultation | qqq | zzz |
| Corporate | Business qq | Consultation | qq | vvv |
```


Thanks @mannharleen! 

This issue was resolved by PR #1042 (merged June 29, 2024). 

The Web UI now properly renders markdown tables when 'Render model outputs as Markdown' is enabled. Closing as resolved. Please feel free to comment or re-open if you run into any other issues.
Repository: promptfoo/promptfoo
Issue: Use @googleapis/* packages instead of googleapis
Root cause: Architectural Constraint

The googleapis package is over 100MB, installed. You can instead use @googleapis/* packages to pull in the exact packages you need (and google-auth-library where needed for auth). They're the same code as what's in googleapis.

Thanks for flagging, done in https://github.com/promptfoo/promptfoo/pull/1009

This is included in the latest release!
Repository: promptfoo/promptfoo
Issue: Proxy support for AWS SDK
Root cause: Unimplemented Feature Gap

Proxy support was added in https://github.com/promptfoo/promptfoo/pull/299, but not for the AWS SDK, which requires additional configuration.

For the AWS SDK, a NodeHttpHandler (from `@smithy/node-http-handler`) needs to be passed to BedrockRuntime, e.g.:
```
import { ProxyAgent } from 'proxy-agent';
import { NodeHttpHandler } from '@smithy/node-http-handler';
import { BedrockRuntime } from '@aws-sdk/client-bedrock-runtime';
...
const handler = new NodeHttpHandler({httpsAgent: new ProxyAgent()});
const bedrock = new BedrockRuntime({ region: this.getRegion(), requestHandler: handler });
```

Note: NodeHttpHandler supports additional configuration values, but I don't know whether it's necessary to allow customization of those (in my tests simply using the `httpsAgent` was enough).

Affected file: [bedrock.ts:453](https://github.com/promptfoo/promptfoo/blob/main/src/providers/bedrock.ts#L453)

Cheers @mldangelo thanks for taking care of this one so quickly 😄 
https://github.com/promptfoo/promptfoo/pull/1021

I downloaded the code for the commit https://github.com/promptfoo/promptfoo/tree/11ccd73ddaa63ea7142f6ae7269284c3030d9d4d built it and ran it in our pipeline behind a proxy, the issue is fixed for us.

Side note: I checked the PR just now, I think it's not necessary to check for the proxy-related env vars, since ProxyAgent takes care of it internally. In any case, it's working for us now, thanks!

Fix is included in the latest release!
Repository: promptfoo/promptfoo
Issue: Proxy Support for Azure with Client credentials
Root cause: Documentation Deficiency

Hello Team,

Please add proxy support with authentication (proxy with optional user name and password) to your code to consider it from either ENV variable or from YAML file as a parameter.

Our azure onenAI is inside our company and promptfoo unable to connect. It comes back with connection error ETIMEDOUT.

I am able to connect from postman or Java code with proxy. But from command prompt it is not connecting. Please note I am running with "promptfoo eval" command from windows cmd. I tried to give proxy in ENV User variable but not working.
Appreciate your help in this as I am stuck due to this.

Thank you
Chitta

I see an article if it helps to add to your code base as I guess the code using node fetch for the api.

https://stackoverflow.com/questions/60162485/how-to-configure-node-fetch-to-use-the-company-proxy



The requests to Azure are made with [proxy-agent](https://www.npmjs.com/package/proxy-agent), which respects `HTTP_PROXY` and `HTTPS_PROXY` [environment variables](https://www.npmjs.com/package/proxy-from-env#environment-variables).  You should be able to set them accordingly to `[protocol://]<host>[:port]` of your proxy!
Repository: promptfoo/promptfoo
Issue: WebUI Concurrent Editing Issue: Data Overwrites
Root cause: Architectural Constraint

Multiple users concurrently editing evaluations in PromptFoo WebUI causes data overwrites due to the handleRating function sending the entire table for updates. Each save overwrites previous changes. This hinders collaboration and can lead to data loss.

Proposed Solutions:
 * Partial Updates: Send only the modified row to the server.
 * Optimistic Locking: Use versioning to detect and resolve conflicts.

Hey @pchuri, thanks for filing this one - the core issue you described has been addressed. When you filed this, the rating handler was sending the entire table on every save, which meant concurrent edits would stomp on each other. We've since moved to a v4 data model where ratings hit a dedicated endpoint (`POST /eval/:evalId/results/:id/rating`) that only updates the specific result being modified. Two people rating different rows no longer interfere with each other.

The v4 approach has been the default for a while now, so if you're running a recent version you should already have this fix.

There's still a theoretical edge case where two people rate the *exact same* result at the same instant - in that scenario the second write wins. In practice this seems rare enough that we haven't seen reports of it happening, but if it becomes a problem we can add optimistic locking.

Real-time sync (where you'd see other people's ratings appear without refreshing) would be a nice enhancement for heavier collaborative workflows. That's a bigger lift though - would need WebSocket broadcasts and client-side state reconciliation. If there's interest we could track that separately.

Going to close this one out, but feel free to reopen if you're still hitting issues or if the real-time sync idea is something you'd find valuable.
Repository: promptfoo/promptfoo
Issue: http provider do evaluation, response no output!
Root cause: Validation Gap

npx promptfoo@latest --version   0.66.0


/home/xxxx/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/providers/http.js
<img width="395" alt="image" src="https://github.com/promptfoo/promptfoo/assets/12491219/bc588a52-5844-4574-b08f-7afb12f6b5a2">

if I use the http provider, then the response is a string instead of a structure with "output"
So everytime I add assertions to it, it always respond "No output"
https://github.com/promptfoo/promptfoo/blob/main/src/evaluator.ts#L378

For now I have to change this line adhoc to return {output: this.responseParser(response.data)};

https://github.com/promptfoo/promptfoo/blob/main/src/providers/http.ts#L69

Thanks for flagging!  Fix in #1027 

We deployed a fix for this in [0.68.0](https://github.com/promptfoo/promptfoo/releases/tag/0.68.0). Thanks for reporting this @ReverseFelicity!
Repository: promptfoo/promptfoo
Issue: Single test suite with multiple conversation test cases
Root cause: Documentation Deficiency

I want to have multiple different conversations to be part of a single test suite, there's a way to do this?

ie:
```
description: 'Global Test Suite'

prompts: [prompt.json]

providers:
 - openai:gpt-4o

tests:
  - description: "Test Conversation 1"
    - vars:
        question: Who founded Facebook?
    - vars:
        question: Where does he live?
    - vars:
        question: Which state is that in?
  # This test is independent from previous one, with a different prompt/conversation variable altogether
  - description: "Test Conversation 2"
    - vars:
      question: Who founded Twitter?
    - vars:
      question: "Is he into Bitcoin?"
 ```
` prompt.json`:
 ```
 [
  {% for completion in _conversation %}
    {
      "role": "user",
      "content": "{{ completion.input }}"
    },
    {
      "role": "assistant",
      "content": "{{ completion.output }}"
    },
  {% endfor %}
  {
    "role": "user",
    "content": "{{ question }}"
  }
]
 ```
 
Where ideally, each vars could have its own assertion as well.
The idea is that each test would restart/reset the `conversation` array, so no context gets carried over

Thanks for the detailed feature request! This is actually already supported using the `conversationId` metadata field.

Here's your exact use case with the current syntax:

```yaml
# promptfooconfig.yaml
description: 'Global Test Suite'

prompts:
  - file://prompt.json

providers:
  - openai:gpt-4o

tests:
  # Test Conversation 1
  - vars:
      question: Who founded Facebook?
    metadata:
      conversationId: conversation_1
  - vars:
      question: Where does he live?
    metadata:
      conversationId: conversation_1
  - vars:
      question: Which state is that in?
    metadata:
      conversationId: conversation_1
    assert:
      - type: contains
        value: California

  # Test Conversation 2 (independent, no context from conversation 1)
  - vars:
      question: Who founded Twitter?
    metadata:
      conversationId: conversation_2
  - vars:
      question: Is he into Bitcoin?
    metadata:
      conversationId: conversation_2
    assert:
      - type: icontains
        value: yes
```

Each turn can have its own assertions. Tests with the same `conversationId` share conversation history via the `_conversation` variable, while different conversation IDs are completely isolated.

Your prompt template would look like:

```json
[
  {% for completion in _conversation %}
    {
      "role": "user",
      "content": "{{ completion.input }}"
    },
    {
      "role": "assistant",
      "content": "{{ completion.output }}"
    },
  {% endfor %}
  {
    "role": "user",
    "content": "{{ question }}"
  }
]
```

**Resources:**
- [Multi-turn conversation docs](https://www.promptfoo.dev/docs/configuration/chat/#separating-chat-conversations)
- [Full working example](https://github.com/promptfoo/promptfoo/tree/main/examples/multiple-turn-conversation)

Closing as this is already supported. Let us know if you have any questions!
Repository: promptfoo/promptfoo
Issue: Feature: typescript variables
Root cause: Unimplemented Feature Gap

Currently the evaluator supports [javascript and python variables](https://github.com/promptfoo/promptfoo/blob/7f3137d6f48b8f968befb4d199cda957a64d8275/src/evaluator.ts#L122). This issue requests to add typescript variables as a third option.

I think this would be quite difficult to do, given the different ways that typescript can be configured.  Since typescript is compiled, it's probably easiest to point it to the js file in your build or dist directory.

The only exception to this may be some of the alternative js runtimes that have native ts support.

@fvdnabee this is working now! Feel free to re-open if you have any questions. Check out our ts example here: https://github.com/promptfoo/promptfoo/tree/main/examples/ts-config
Repository: promptfoo/promptfoo
Issue: How to compare two separate eval runs in the Web UI
Root cause: Unimplemented Feature Gap

In the config, if I've defined two prompts, or two providers, I see the side-by-side results in the Web UI.

What about the situation if I or someone else have run an eval on a single prompt or provider, and want to combine our two results and see them in the promptfoo Web UI?  Can that be done?

Two thoughts on how to solve this:
- Purely on the UI side, have a way to select existing evals to include in the view, or
- Add a 'promptfoo eval merge` CLI command or similar that lets you select evals to combine together.  This will enable better first-class support for share, export, etc.

> Purely on the UI side, have a way to select existing evals to include in the view, or

I like this UI solution for the use case of multiple collaborators running their own evals, then wanting to do comparisons among the eval runs.

Merging evals together might have its own use cases, but I think would be cumbersome for browsing.

"Compare evals" functionality landed in the UI under `Table Settings`
Repository: promptfoo/promptfoo
Issue: eval failing with `RangeError: Invalid count value: -10`
Root cause: Validation Gap

Just got this in our CI

```
/home/circleci/codriver/tools/promptfoo/node_modules/promptfoo/dist/src/updates.js:34
        const border = '='.repeat(process.stdout.columns - 10);
                           ^

RangeError: Invalid count value: -10
    at String.repeat (<anonymous>)
    at checkForUpdates (/home/circleci/codriver/tools/promptfoo/node_modules/promptfoo/dist/src/updates.js:34:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async main (/home/circleci/codriver/tools/promptfoo/node_modules/promptfoo/dist/src/main.js:185:5)
```

The strangest thing is that:

1) I can't reproduce the same issue at all locally
2) It was working correctly on CI before, started failing a couple of hours ago without _any changes_ to any dependencies (it is using `npx promptfoo`, but we have it installed on `node_packages` at version `0.66.0`) and neither to any of the config/assertions

Thanks for flagging.  This code path is checking for updates and outputting a banner if there is a new version.  0.67.0 was _just_ released, so that's why the problem just started :)

As a temp workaround, set the environment variable `PROMPTFOO_DISABLE_UPDATE=1`.  This will prevent it from checking for updates.

Alternatively, updating CI to use promptfoo 0.67.0 will fix the problem as well.

I will include a fix in the next version.

A fix for this was deployed in [0.68.0](https://github.com/promptfoo/promptfoo/releases/tag/0.68.0) Please let us know if you run into any other issues and thanks for reporting it!
Repository: promptfoo/promptfoo
Issue: Allow configuring which UI columns to show from cli
Root cause: Unimplemented Feature Gap

I have a lot of template variables and want to always hide certain variables.

and in which order IMO

How do I use this feature? the documentation do not mention it. Thank you

This would be huge for me. Similar to #1032. Either one would work.

In progress at https://github.com/promptfoo/promptfoo/pull/3873
Repository: promptfoo/promptfoo
Issue: Provider errors from custom python script not displayed in the webui
Root cause: Validation Gap

I am using a custom python provider that sets the `error` property (https://www.promptfoo.dev/docs/providers/python#python-script). The eval is failed correctly but the error message is not rendering in the UI.

This commit adds those errors to the UI but I'm unclear if this is the right approach:
https://github.com/promptfoo/promptfoo/compare/main...spiderbites:promptfoo:render-errors-in-webui
Repository: promptfoo/promptfoo
Issue: Is there a way to run asserts/evaluations dynamically based on an input prompt ?
Root cause: Unimplemented Feature Gap

So I'm trying to evaluate a llm response ad hoc.

I have multiple asserts like:  

A: Check enum is in results for "Input A" in prompt

B: Check result is sql for Input B

C: Check there is LIMIT keyword

My question is: given a prompt is there a way to only run asserts A and B or only A and C ?



This would be a must feature for an llm response evaluation tool.

Thanks for your awesome project !



Hey @typpo In the attempt of running asserts dynamically first I am trying to input a single prompt via `--prompt-suffix` option

Expected behavior and example:
`!npx promptfoo@latest eval -c promptfooconfig_simple.yaml --no-cache --prompt-suffix question`

promptfooconfig_simple.yaml  contains:
`description: "Promptfoo Eval SQL"

prompts: "system_prompt.txt"

providers:
  - id: "python:Interface.py"
    label: "Development Phase SQL"
    config:
      model: "gpt-4o"
      tokens: 2000
      temperature: 0
tests: test_simple.yaml
`

When I do promptfoo view I expect that my question is appended to the main prompt but it seems not to be the case. 

question: is my input prompt that needs to be appended at the end of the system prompt e.g. : "Show me top 10 results".

Unfortunately it seems --prompt-suffix is not working properly or I am doing something very wrong ?

Can you confirm --prompt-suffix works with an str input or not ? Was it intended to only work with another file ? If so could we make it a feature to also work with a single str.
Repository: promptfoo/promptfoo
Issue: `TypeError: testCases is not iterable`
Root cause: Validation Gap

My `promptfooconfig.yml`:
```yml
description: "verify"

prompts:
  - file://verify.prompt.txt
providers:
  - file://verify.provider.yml
tests: tests/*.test.yml
```
Test file:
```yml
  description: "Validation of the code changes"
  vars:
    preBuildState: file://assets/shared/pre_build.go
    changes: file://assets/valid/changes.md
    postBuildState: file://assets/valid/post_build.go
    diffs: file://assets/valid/diff.txt
  assert:
    - type: is-json
    - type: is-valid-openai-tools-call
    - type: javascript
      value: |
        var args = JSON.parse(output[0].function.arguments)
        return !(
          args.hasSyntaxErrors ||
          args.hasRemovedCodeErrors ||
          args.hasDuplicationErrors ||
          args.hasReferenceErrors            
        )

```

Previous to my update this worked fine. I had all of my tests in my `/tests` folder. After update - this no longer works.

```
Cache is disabled.
Loaded provider openai:gpt-4o from verify.provider.yml
/home/daofficialwizard/.local/share/pnpm/global/5/.pnpm/promptfoo@0.67.0_@aws-sdk+client-bedrock-runtime@3.598.0_@azure+identity@4.2.1_@azure+openai-_o4xgjjjr6zw7j63g3trvuro4vm/node_modules/promptfoo/dist/src/testCases.js:177
                for (const testCase of testCases) {
                                       ^

TypeError: testCases is not iterable
    at loadTestsFromGlob (/home/daofficialwizard/.local/share/pnpm/global/5/.pnpm/promptfoo@0.67.0_@aws-sdk+client-bedrock-runtime@3.598.0_@azure+identity@4.2.1_@azure+openai-_o4xgjjjr6zw7j63g3trvuro4vm/node_modules/promptfoo/dist/src/testCases.js:177:40)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async resolveConfigs (/home/daofficialwizard/.local/share/pnpm/global/5/.pnpm/promptfoo@0.67.0_@aws-sdk+client-bedrock-runtime@3.598.0_@azure+identity@4.2.1_@azure+openai-_o4xgjjjr6zw7j63g3trvuro4vm/node_modules/promptfoo/dist/src/main.js:134:25)
    at async runEvaluation (/home/daofficialwizard/.local/share/pnpm/global/5/.pnpm/promptfoo@0.67.0_@aws-sdk+client-bedrock-runtime@3.598.0_@azure+identity@4.2.1_@azure+openai-_o4xgjjjr6zw7j63g3trvuro4vm/node_modules/promptfoo/dist/src/main.js:545:48)
    at async Command.<anonymous> (/home/daofficialwizard/.local/share/pnpm/global/5/.pnpm/promptfoo@0.67.0_@aws-sdk+client-bedrock-runtime@3.598.0_@azure+identity@4.2.1_@azure+openai-_o4xgjjjr6zw7j63g3trvuro4vm/node_modules/promptfoo/dist/src/main.js:704:9)

Node.js v22.3.0
```

That is the error. 



Seems the issue is actually a missing `-` at the top of the test file. 

We deployed a fix for this in [0.68.0](https://github.com/promptfoo/promptfoo/releases/tag/0.68.0) as well. Thanks for reporting the issue and helping make promptfoo better @ZanzyTHEbar!
Repository: promptfoo/promptfoo
Issue: Allow filtering pre-defined providers (like --filter-pattern but for providers)
Root cause: Unimplemented Feature Gap

I have a promptfooconfig.yaml which defines multiple custom providers. I want to pass cli flag to `promptfoo eval` which will select providers which a regex on the labels field.

I tried using the `-r` flag for this, but it seems to define a new provider to use, not filter on existing providers defined in the config.

Implemented in #1089!

Thanks!
Repository: promptfoo/promptfoo
Issue: Can't specify OpenAI organisation configuration
Root cause: Unimplemented Feature Gap

Currently the OpenAI organisation key is not supported. It would be great if I could configure it, thanks.

Support for `OPENAI_ORGANIZATION` environment variable will be released in 0.19.1.

You will also be able to specify with:
```yaml
providers:
  - openai:chat:gpt-3.5-turbo-0613:
      config:
        organization: org-xyz
```
Repository: promptfoo/promptfoo
Issue: [Question]Does function have specified names for custom python scripts
Root cause: Documentation Deficiency

According to https://www.promptfoo.dev/docs/providers/python and https://github.com/promptfoo/promptfoo/blob/main/examples/python-provider/provider.py

Both scripts have call_api function, Do all custom python scripts have to have this function?
If I have multiple functions in py files, how do promptfoo execute them?

Right now, python providers only support the `call_api` function.

@mldangelo it would be useful if the provider supports the `file.py:function_name` syntax like for prompts.

Implementing support for this in #1053 

For example:
```yaml
providers:
  - python:file.py:my_function
```

A fix for this was deployed in [0.68.1](https://github.com/promptfoo/promptfoo/releases/tag/0.68.1). You can check out the custom python provider example here: https://github.com/promptfoo/promptfoo/tree/0.68.1/examples/python-provider for an example of how to use it. Please let us know if you run into any other issues and thanks again for suggesting this feature!