zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: More eval actions
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I find my self running queries on the database after the evaluation:

1. to have a better understanding of what went wrong.
2. extract some data for analysis

`select count(*), error from eval_results where eval_id='eval-pLQ-2025-05-03T07:07:28' and failure_reason = 1 group by error order by 1 desc`

<img width="358" alt="Image" src="https://github.com/user-attachments/assets/18008d93-baeb-45c4-9652-ee354ce5cd2c" />

or running a query to extract metadata to work on the issues found in the previous query

`select test_case,metadata,error from eval_results where eval_id='eval-1qs-2025-05-02T15:29:41'  and error ='Expected output "naming" to equal "typo"'`

<img width="601" alt="Image" src="https://github.com/user-attachments/assets/c68d674c-7ff6-48f5-aba1-6c8263024775" />


**Describe the solution you'd like**
add a button here that shows counts by error group 

<img width="277" alt="Image" src="https://github.com/user-attachments/assets/a05e5f2a-ac12-4658-b902-8e90ecb4cc5f" />

modify the download page so i can select which columns i want to download
<img width="903" alt="Image" src="https://github.com/user-attachments/assets/c1e6f5c5-e846-40b6-8994-86caf80fcb95" />

**Describe alternatives you've considered**
1. An improved reporting page that i can use to extract data as i need
2. An agent i can ask to fetch the data for me
3. building a query interface where i can save queries as well (important part is the saving), and running on an eval_id so that i do not have to replace those and find the query all the time.

**Additional context**
I have no idea how others use promptfoo and if this is useful widely or if its only a me issue.
Repository: promptfoo/promptfoo
Issue: Use environment variables in "sharing" URLs
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
We use a self hosted Promptfoo server for easy sharing between the team. The instance has a basic auth defined (user name+ password), however Promptfoo doesn't replace the environment variables and we need to write custom scripts to replace them with the correct values, which makes it tedious during development as we might accidentally commint the replaced passwords.
```
sharing:
  apiBaseUrl: 'https://{{env.PROMPTFOO_USER}}:{{env.PROMPTFOO_PASSWORD}}@self-hosted-promptfoo.com'
  appBaseUrl: 'https://{{env.PROMPTFOO_USER}}:{{env.PROMPTFOO_PASSWORD}}@self-hosted-promptfoo.com'
```

**Describe the solution you'd like**
We'd like our environment variables to be recognized in sharing URLs.

**Describe alternatives you've considered**
We're using custom scripts to replace them both locally and in GitLab pipelines, but it's very risky if we forget it and commit the username and password.

**Additional context**
Add any other context or screenshots about the feature request here.
None

Thanks for opening this issue, @vira-s.

Are you able to set the credentials within urls scoped to the environment? [For example](https://www.promptfoo.dev/docs/usage/sharing/#advanced-self-hosted-sharing):

```
export PROMPTFOO_REMOTE_API_BASE_URL=https://user:pass@self-hosted-promptfoo.com
export PROMPTFOO_REMOTE_APP_BASE_URL=https://user:pass@self-hosted-promptfoo.com
```

If that works, I'll ship https://github.com/promptfoo/promptfoo/pull/3903 which will add a new flag to ensure your auth credentials are persisted.

Yes, those 2 environment variables worked perfectly! Thank you.
Repository: promptfoo/promptfoo
Issue: Make it possible to define structured output json schema in an external yaml file
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently, using the OpenAI response format takes up a lot of space, and even with json references, it's hard to make the long and verbose schema reusable. 

**Describe the solution you'd like**
Being able to define the yaml schema in a file, that can be used later with file:// or just by writing the file path/name

**Additional context**
Maybe the OpenAI structured output example in the [examples](https://github.com/promptfoo/promptfoo/tree/main/examples/openai-structured-output) could be changed to show how to do this, if this is already possible.
Repository: promptfoo/promptfoo
Issue: Multiple Turn Conversationi Config/Prompt Doesn't work.  LLM analyzes json it is sent.
Root cause: Configuration Error

**Describe the bug**
Following: https://www.promptfoo.dev/docs/configuration/chat/
Source Code: https://github.com/promptfoo/promptfoo/tree/main/examples/multiple-turn-conversation

Running eval produces this:

> It looks like you're referencing a JSON file that contains information about a multi-turn conversation evaluation for a chatbot. Here's a breakdown of the JSON structure you provided:
> 
> ```json
> {
> ... and so on ...
> 

**To Reproduce**
Changed model provider id.
Used prompt.json and promptfooconfig.yaml as they were.  

promptfooconfig.yaml 
Copy and pasted from here:
https://github.com/promptfoo/promptfoo/blob/main/examples/multiple-turn-conversation/promptfooconfig.yaml

prompts.json
Copy and pasted from here:
https://github.com/promptfoo/promptfoo/blob/main/examples/multiple-turn-conversation/prompt.json


**Expected behavior**
Work as described here:
https://www.promptfoo.dev/docs/configuration/chat/

**System information:**
```
===========================================================================================
Promptfoo Debug Information
===========================================================================================
{
  "version": "0.112.3",
  "platform": {
    "os": "darwin",
    "release": "24.3.0",
    "arch": "x64",
    "nodeVersion": "v22.11.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```

- Promptfoo version:

**Additional context**
Add any other context about the problem here.


@ebsco-rpotter did you manage to resolve this issue on your own?  If you run into anything else or have suggestions on improving the docs, feel free to let us know! 🚀
Repository: promptfoo/promptfoo
Issue: Multiple Turn Conversationi Config/Prompt Doesn't work. LLM analyzes json it is sent.
Root cause: Configuration Error

**Describe the bug**
Following: https://www.promptfoo.dev/docs/configuration/chat/
Source Code: https://github.com/promptfoo/promptfoo/tree/main/examples/multiple-turn-conversation

Running eval produces this:

> It looks like you're referencing a JSON file that contains information about a multi-turn conversation evaluation for a chatbot. Here's a breakdown of the JSON structure you provided:
> 
> ```json
> {
> ... and so on ...
> 

**To Reproduce**
Changed model provider id.
Used prompt.json and promptfooconfig.yaml as they were.  

promptfooconfig.yaml 
Copy and pasted from here:
https://github.com/promptfoo/promptfoo/blob/main/examples/multiple-turn-conversation/promptfooconfig.yaml

prompts.json
Copy and pasted from here:
https://github.com/promptfoo/promptfoo/blob/main/examples/multiple-turn-conversation/prompt.json


**Expected behavior**
Work as described here:
https://www.promptfoo.dev/docs/configuration/chat/

**System information:**
```
===========================================================================================
Promptfoo Debug Information
===========================================================================================
{
  "version": "0.112.3",
  "platform": {
    "os": "darwin",
    "release": "24.3.0",
    "arch": "x64",
    "nodeVersion": "v22.11.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```

- Promptfoo version:
Latest installed today.

**Additional context**
Other non-related evals ran fine.
Repository: promptfoo/promptfoo
Issue: SqliteError: FOREIGN KEY constraint failed
Root cause: Algorithmic Error

**Describe the bug**
After upgrade promptfoo from 0.111.1 to 0.112.3

**To Reproduce**
Run promptfoo eval 

**System information:**
macOS 15.4.1 Apple Silicon
Node v22.15.0

- Promptfoo version:
0.112.3

**Additional context**
```
/Users/kevinzhow/.nvm/versions/node/v22.15.0/lib/node_modules/promptfoo/node_modules/drizzle-orm/better-sqlite3/session.cjs:88
    return this.stmt.run(...params);
                     ^
SqliteError: FOREIGN KEY constraint failed
    at PreparedQuery.run (/Users/kevinzhow/.nvm/versions/node/v22.15.0/lib/node_modules/promptfoo/node_modules/drizzle-orm/better-sqlite3/session.cjs:88:22)
    at QueryPromise.run (/Users/kevinzhow/.nvm/versions/node/v22.15.0/lib/node_modules/promptfoo/node_modules/drizzle-orm/sqlite-core/query-builders/insert.cjs:182:28)
    at Function.<anonymous> (/Users/kevinzhow/.nvm/versions/node/v22.15.0/lib/node_modules/promptfoo/dist/src/models/eval.js:142:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
  code: 'SQLITE_CONSTRAINT_FOREIGNKEY'
}

Node.js v22.15.0
```

I'm getting this as a fresh install too. Not used promptfoo before.  

Even trying an old version 0.111.1, node 22 or even node 24.

`$ promptfoo eval

====================================================================================================
⚠️ The current version of promptfoo 0.111.1 is lower than the latest available version 0.112.3.

Please run npx promptfoo@latest or npm install -g promptfoo@latest to update.
====================================================================================================

/home/dasmith/.nvm/versions/node/v24.0.0/lib/node_modules/promptfoo/node_modules/drizzle-orm/better-sqlite3/session.cjs:88
    return this.stmt.run(...params);
                     ^
SqliteError: FOREIGN KEY constraint failed
    at PreparedQuery.run (/home/dasmith/.nvm/versions/node/v24.0.0/lib/node_modules/promptfoo/node_modules/drizzle-orm/better-sqlite3/session.cjs:88:22)
    at QueryPromise.run (/home/dasmith/.nvm/versions/node/v24.0.0/lib/node_modules/promptfoo/node_modules/drizzle-orm/sqlite-core/query-builders/insert.cjs:182:28)
    at sqliteTransaction.<anonymous> (/home/dasmith/.nvm/versions/node/v24.0.0/lib/node_modules/promptfoo/dist/src/models/eval.js:142:22)
    at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
  code: 'SQLITE_CONSTRAINT_FOREIGNKEY'
}`

Thanks for bringing this to our attention; I'm able to reproduce and am investigating.

Fixed in [0.112.4](https://github.com/promptfoo/promptfoo/releases/tag/0.112.4)
Repository: promptfoo/promptfoo
Issue: openrouter.ai endpoints
Root cause: Documentation Deficiency

Is it possible, and if so how, to use openrouter for my endpoints to test with Promptfoo?
Great work by the way.

Hi @rnehrboss,

Based on my read of their [docs](https://openrouter.ai/models/mistralai/mistral-medium?tab=api) it looks like you should be able to override the existing OpenAI provider:

```yaml
providers:
  - id: openai:chat:mistralai/mistral-medium    # or any other chat model
    config:
      apiBaseUrl: https://openrouter.ai/api
      apiKey: openrouter_api_key
```

You can also use the `OPENAI_API_KEY` and `OPENAI_API_BASE_URL` environment variables if you prefer.

Thanks Ian, will give it a try


On Fri, Jan 12, 2024 at 2:57 PM Ian Webster ***@***.***>
wrote:

> Hi @rnehrboss <https://github.com/rnehrboss>,
>
> Based on my read of their docs
> <https://openrouter.ai/models/mistralai/mistral-medium?tab=api> it looks
> like you should be able to override the existing OpenAI provider:
>
> providers:
>   - id: openai:chat:mistralai/mistral-medium    # or any other chat model
>     config:
>       apiBaseUrl: https://openrouter.ai/api
>       apiKey: openrouter_api_key
>
> You can also use the OPENAI_API_KEY and OPENAI_API_BASE_URL environment
> variables if you prefer.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/394#issuecomment-1889940174>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHU7QE7CMO3MWWM7QQ4K2FDYOGPT5AVCNFSM6AAAAABBYVEOZSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOBZHE2DAMJXGQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: Bedrock provider doesn't use prompt function config
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
When a prompt function returns extra configuration, the bedrock provider doesn't use such config, so things like "tools" and "tool_choice" is not passed to LLM. That means that prompts that use Json structured outputs can't be tested with promptfoo.

things like this wont work
```
prompt: (vars: any) => {
    return {
        prompt: //list of messages,

        // the way structured outputs is configured for anthropic models
        config: {
          tools: [
            {
              name: 'json',
              description: 'Respond with a JSON object',
              input_schema: jsonSchema,
            },
          ],
          tool_choice: {
            name: 'json',
            type: 'tool',
          },
        },
    }
}
```

**Describe the solution you'd like**
`AwsBedrockCompletionProvider.callApi` receives only the prompt string, but not all extra configuration that evaluator pass (a context variable that has the prompt object, with the prompt config).
This config could be use to feed the specific provider `params()` function


Pushed a PR
https://github.com/promptfoo/promptfoo/pull/3970

@mldangelo I see you assigned yourself. Can you take a look at my PR if you haven't done bunch of progress?


This was released in 0.112.5 ([changelog](https://github.com/promptfoo/promptfoo/releases/tag/0.112.5)).
Huge thanks to @afals-qb for the clear report and to @aperezfals for the swift fix in PR #3970
Repository: promptfoo/promptfoo
Issue: Question: Is it possible to pass Pinecone embeddings as context in promptfoo's RAG eval?
Root cause: Documentation Deficiency

Hey there, my LLM application uses Pinecone as RAG context to generate outputs. I was wondering if there's anyway to pass Pinecone embeddings as context in promptfoo so that the score I get gives a more accurate representation of my app's performance as opposed to directly putting text from my documents.

Hello!  I believe we spoke on Discord so I'm pasting the answer here:

Right now you have 2 options:

1) Include the context directly.  In other words, choose some examples that reflect the data in pinecone

2) Use a dynamic variable that calls your code to fetch from pinecone.  For example
```yaml
tests:
  - vars:
      context: file://context_fetcher.py
```
Repository: promptfoo/promptfoo
Issue: Improve datasets feature
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**  
As a developer, my workflow involves building a dataset, running the evaluation, identifying common issues in that evaluation, then iterating on a subset and running the entire evaluation again.

While it is straightforward (though not clear that you are creating datasets when following guides, etc.) to generate the initial dataset, support for the rest of the workflow is nonexistent.

---

**Describe the solution you'd like**  
I would like to select multiple test cases and run an evaluation on those test cases.  

This process would create a new dataset and evaluation. It would be helpful to maintain a link between the two.

---

**Describe alternatives you've considered**  
I have considered rerunning the test cases in place within the evaluation and overriding the previous results:

This approach is useful for a few quick iterations. However, over time, the manipulation of the evaluation becomes too complex to track. There would also be need to be a way to indicate that this evaluation was "corrupted" by having portions of the test results run with another prompt.

Overall i do think that evals should not really mutable.

---

**Additional context**  
Currently, I have been using CSV files and CSV grep to achieve this. For example:  

```bash
csvgrep -c remote_id -r "$REMOTE_ID" "$CSV_PATH" | PROMPTFOO_CACHE_ENABLED=false node evaluate-comment-classification.mjs
Repository: promptfoo/promptfoo
Issue: Add option to turn off Nunjucks engine
Root cause: Unimplemented Feature Gap

Could we add an option to turn off Nunjucks templating? It's been messing me up 🙃 

### Usecase
I have a prompt that accepts fragments of file content - these files may be markdown files, code files, etc. I'm not using any Nunjucks templating - I'm constructing my prompts using JS files.

My problem is that the Nunjucks templating is causing issues - it's detecting all sorts of things in the files, which it thinks are opening tags (for example a line that contains `{# .somevalue }`), but because those aren't tags, it ends up throwing an exception and crashing the run.  It would be great to just have a flag to turn the templating off completely, for when it's not needed.

The only workaround I've found so far has been to just manually `{% raw %}` and `{% endraw %}` tags to the beginning and end of my example inputs, but this if very inconvenient and feels pretty silly to have to do.

You can imagine that even if it wasn't throwing errors, I'd want to be able to process files that _do_ contain Jinja2 templates without Nunjucks trying to process the tables and messing everything up.

**SIDE NOTE**: what made this whole thing way worse is that when this templating exception is thrown, there is no simple way to know which test case is the one that's causing the issue - I had to manually run them one by one to find the offender. Might be nice if exceptions could have some info about the test case they happened in.

#405 adds an environment variable `PROMPTFOO_DISABLE_TEMPLATING` that should solve your immediate need :)

This is included in release 0.37.0
Repository: promptfoo/promptfoo
Issue: I created a kube-promptfoo-controller

This is not a issue, just an advertisement of my OSS [kube-promptfoo-controller](https://github.com/Gekko0114/kube-promptfoo-controller).

[Promptfoo](https://promptfoo.dev/) is a very useful CLI and library for evaluating LLM output quality.
However, when using the openAI API in production, the output may change due to silent updates of openAI API. It can't be monitored by promptfoo.
This repository allows you to run promptfoo as a cronJob. You just need to set up the prompt and schedule on CRD.

I think my repository might useful for promptfoo users as well. I would appreciate it if someone could use it.

Very cool, thank you for sharing!  I'd like to add a community section to the promptfoo website with libraries like this one, once there is enough content.
Repository: promptfoo/promptfoo
Issue: An unexpected error occurred, in the Teams plugins, if I simply set it to owasp:llm
Root cause: Validation Gap

**Describe the bug**

An unexpected error occurred, in the Teams plugins, if I simply set it to owasp:llm.

redteam-config.yaml is below.

```
redteam:
  plugins:
    - owasp:llm
  strategies:
    - prompt-injection
    - jailbreak
  language: Japanese
```

**To Reproduce**

1. Run 'promptfoo redteam generate --config ./redteam-config.yaml --output ./redteam-cases.yml'
2. Display 'An unexpected error occurred,...'

Details is below.


> An unexpected error occurred during red team run: Validation failed for plugin indirect-prompt-injection: Error: Invariant failed: Indirect prompt injection plugin requires `config.indirectInjectionVar` to be set
> Error: Validation failed for plugin indirect-prompt-injection: Error: Invariant failed: Indirect prompt injection plugin requires `config.indirectInjectionVar` to be set
> 

**Expected behavior**

I would like information on how to avoid the error to be displayed.

OR

I want to know if the owasp:llm sugar syntax is inherently not working as intended.


**System information:**

- Promptfoo version: 0.112.7


Hi @delhicurry we'll get this fixed in the next release (tomorrow). In the meantime, I suggest individually specifying plugins. Sorry about the bug! 

Closing as complete. Please give this a try @delhicurry and let us know if you encounter any other issues!

Thank you for dealing with this so quickly.
Repository: promptfoo/promptfoo
Issue: Enable sharing state between hooks and Custom Provider calls
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When calling an API that requires authentication, I want to perform the authentication only once, cache the session/token in a variable, and reuse it across multiple tests.

However, currently there is no mechanism to share variables between hooks and individual Custom Provider calls. This makes it difficult to efficiently reuse state such as authentication sessions across tests.

For example, even if I retrieve authentication information in the beforeAll hook like this:

```python
# hooks.py

def extension_hook(hook_name: str, context: dict) -> None:
    if hook_name == "beforeAll":
        access_token = retrive_access_token()
```

There is no way to refer to that access_token from a PythonProvider call like this:

```python
# custom_provider.py

def call_api(prompt: str, options: Dict[str, Any], context: Dict[str, Any]):
    # I would like to take an access_token retrieved in beforeAll hook but there is no way
    access_token = ........
    res = request(access_token, prompt)
    return { "output": res.message }
```

**Describe the solution you'd like**
Ideally, I would like to have an object or mechanism that allows data (such as authentication information obtained in a beforeAll hook) to be referenced from within a PythonProvider call. This way, shared state can be easily accessed and reused across tests and providers.

**Describe alternatives you've considered**
None

**Additional context**
None


Do you expect your token to expire during the tests, or do you expect it to be valid for the entirety of your test suite? have you tried anything to solve this problem, a very basic way would be to write it to a temporary file for the duration of the test then delete it.

---i was confused, just leaving this here so the notifications don't confuse you
<del> Sorry i notice that your hook is using beforeAll so would be running before all of your tests, why not just call the method in your custom_provider and handle the state there?</del>

Thank you for your response and providing such a wonderful library

> Do you expect your token to expire during the tests, or do you expect it to be valid for the entirety of your test suite? 

Token is valid throughout the entire test suite.

> have you tried anything to solve this problem, a very basic way would be to write it to a temporary file for the duration of the test then delete it.

Currently, I'm just handling this by writing the token to a temporary file, but I think it would be useful to have a context object that can be passed around throughout the test execution as a library feature. So I created this issue as a feature request. 



It is definitely a deficiency with non JavaScript extensions, since they run in the same process as the evaluator itself so they can modify the objects passed in.

But it seems that @will-holley has proposed a pretty nice solution to the problem   👍🏿 

Thank you so much for adding this feature: pr 4197, @.will-holley


@53ningen, you're welcome : )