zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Variables which are objects (i.e. arrays) are not displayed correctly in console output and HTML results
Root cause: Validation Gap

**Describe the bug**

When a variable is i.e. an array of conversations:

```
  - vars:
      conversation:
        - user: How do I work with you?
        - assistant: I can help you understand how to work with me. What specifically would you like to know?
        - user: Tell me more about NDAs
```

then it's not properly displayed in neither console output nor HTML. Here is a comparison

* **Viewer - displayed properly**

<img width="786" height="674" alt="Image" src="https://github.com/user-attachments/assets/fcab4e1c-327a-4271-a87b-f09daa8e3cc2" />

* **Console table output - nothing displayed**

<img width="952" height="915" alt="Image" src="https://github.com/user-attachments/assets/871b7151-81a9-42e3-a8be-0d2d13fc8b13" />

* **HTML output - JS variable displayed**

<img width="606" height="584" alt="Image" src="https://github.com/user-attachments/assets/bc157201-45d4-4568-9b3a-dc7c11aa25d6" />

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

```
description: 'Getting started'
prompts:
  - file://./prompt.js

providers:
  - id: google:gemini-2.5-pro
    config:
      temperature: 0.4

defaultTest:
  vars:
    help_articles: file://./help-articles.txt
    assistant_prompt: file://./customer-support.txt

tests:
  - vars:
      conversation:
        - user: I want my user to sign NDA. How can I do that?
```

```
module.exports = async function ({ vars }) {
  const messages = [
    {
      role: 'system',
      content: vars.assistant_prompt
    }
  ]

  // Convert our concise format to OpenAI format
  if (vars.conversation) {
    for (const turn of vars.conversation) {
      if (turn.user) {
        messages.push({ role: 'user', content: turn.user })
      } else if (turn.assistant) {
        messages.push({ role: 'assistant', content: turn.assistant })
      }
    }
  }

  return messages
}
```


**Expected behavior**

I expect JSON representation of the variable object being outputted in both console and HTML report.

**System information:**

```
{
  "version": "0.118.14",
  "platform": {
    "os": "linux",
    "release": "6.2.0-39-generic",
    "arch": "x64",
    "nodeVersion": "v22.12.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```
Repository: promptfoo/promptfoo
Issue: Importing Promptfoo prevents Node from exiting naturally
Root cause: Resource Mishandling

**Describe the bug**
When importing the npm package, it seems to create a process that prevents node from exiting without even invoking anything.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Create a node project (with package.json "type": "module") and install latest promptfoo
2. Import promptfoo in a JS file like so:
 ```
import promptfoo from 'promptfoo';
```
3. Run the file using node: `node ./test.js`
4. Notice the node process never exits naturally

**Expected behavior**
Node should quickly exit instead of hanging. 
In general, even if promptfoo eval is run, it should cleanup any resources / timeouts and naturally exit.

**System information:**

- Promptfoo version: 0.118.14
- Node version: 22.13.1
- Windows 10, powershell terminal

**Additional context**
Earlier versions of promptfoo did not seem to have this issue. Testing shows that release 0.118.7 has this issue, but 0.118.6 does not.


Hi @justindilen,

Sorry for the bug and for not closing this sooner - this was fixed in **PR #5698** on September 23, 2025 (before you reported it), released in version 0.118.8.

The PostHog client was keeping connections open and preventing Node from exiting naturally. This has been resolved - the process now exits cleanly after commands complete.

Please pull the latest version and the issue should be gone. If you experience any other issues, feel free to comment or re-open.

See the [Command Line documentation](https://www.promptfoo.dev/docs/usage/command-line) for more info.

Thanks for the report!

> Hi [@justindilen](https://github.com/justindilen),
> 
> Sorry for the bug and for not closing this sooner - this was fixed in **PR [#5698](https://github.com/promptfoo/promptfoo/pull/5698)** on September 23, 2025 (before you reported it), released in version 0.118.8.
> 
> The PostHog client was keeping connections open and preventing Node from exiting naturally. This has been resolved - the process now exits cleanly after commands complete.
> 
> Please pull the latest version and the issue should be gone. If you experience any other issues, feel free to comment or re-open.
> 
> See the [Command Line documentation](https://www.promptfoo.dev/docs/usage/command-line) for more info.
> 
> Thanks for the report!

@mldangelo  Appreciate the response. 

Unfortunately, in the original post I was testing using 0.118.14 and was having this issue, so I don't think it was fixed in 0.118.8.

To further confirm, I pulled the latest (0.118.17 as of writing), and this is still an issue.
I also tried your suggested version (0.118.8), and I am also seeing the issue there. 
Seems like only using version  0.118.6 and earlier fixes it.

It looks like I do not have permissions to re-open, but it would be nice if this could still be addressed. We use the promptfoo tool in our CI/CD pipelines, and this issue prevents us from using the latest version as it never exits and times out our pipelines.
Repository: promptfoo/promptfoo
Issue: Promptfoo share link cannot render the json object correctly
Root cause: Configuration Error

Thanks for building the useful tool. I found the following bug. 

- When I create a public link using `promptfood share`. The prompt is displayed as `object object` (https://share.getcloudapp.com/5zuONPnd) . 
- This works okay in the local server. 
- Please see this video for more details: https://app.visla.us/clip/1127803618527932417



Hi @jingconan,

Can you please hard refresh the shared page and try again?  This should be resolved now.  Builds were not happening automatically due to the repo switch from my personal account to `promptfoo` github org.

Yep, just hard refreshed and it works now: https://share.getcloudapp.com/P8uXG7zL . Thank you very much Ian!
Repository: promptfoo/promptfoo
Issue: Add n1n.ai API as a model provider
Root cause: Unimplemented Feature Gap

Hi Promptfoo team,

We're n1n API, and we offer a robust API that provides access to 400+ large language models (LLMs), as well as multimodal generation capabilities (text, images, video, audio, etc.). Our platform makes it easy to integrate these advanced features into your project, empowering developers with powerful AI tools.

With n1n API, you can quickly leverage a comprehensive suite of AI models for diverse applications—everything from text generation to media synthesis—through simple, easy-to-use endpoints.

Adding n1n API would benefit Promptfoo users by:

- Providing access to 400+ models through a single API key configuration
- Supporting all major models (OpenAI, Anthropic, Gemini, Llama, and many more)
- Offering competitive pricing with unified billing (some are lowest to 1/10 of the official price)
- Enabling easy model switching for different assistants without managing multiple API keys

### Describe the solution you'd like

Integrate n1n API as an additional AI model provider in Promptfoo. n1n API is OpenAI-compatible (base URL: https://n1n.ai/v1/) and offers extended multimodal generation capabilities beyond standard text completion, including image, video, and audio generation endpoints.

### Resources

Website: https://n1n.ai/
API Documentation: https://docs.n1n.ai/
Dashboard: https://n1n.ai/console
Support: [help@n1n.ai](mailto:help@n1n.ai)

We are excited to contribute to the Promptfoo ecosystem and provide users with access to the latest AI models through our robust API platform. We look forward to collaborating with the Promptfoo maintainers and community to make this integration successful.

Hi @n1n-api!
Thanks for reaching out! n1n.ai sounds like a great fit for promptfoo. We'd love to have you contribute the integration. You can follow our provider guide here: https://www.promptfoo.dev/docs/contributing/#adding-a-new-provider

Once your PR is merged, we'll send you some promptfoo swag as a thank you! Feel free to reach out if you need any help along the way.
Repository: promptfoo/promptfoo
Issue: Throw an error if config file not found
Root cause: Validation Gap

Currently, if a config file is not found, the error indicates that the test cases are not defined. It could be more apparent if instead a `FileNotFound` error is raised:

```npm
❯ promptfoo eval -c rag_generation.yaml
/opt/homebrew/lib/node_modules/promptfoo/dist/src/testCases.js:130
        throw new Error(`Test case must have either assert, vars, or options property. Instead got ${JSON.stringify(testCase, null, 2)}`
);
              ^

Error: Test case must have either assert, vars, or options property. Instead got {}
    at readTest (/opt/homebrew/lib/node_modules/promptfoo/dist/src/testCases.js:130:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async resolveConfigs (/opt/homebrew/lib/node_modules/promptfoo/dist/src/main.js:131:39)
    at async runEvaluation (/opt/homebrew/lib/node_modules/promptfoo/dist/src/main.js:406:48)
    at async Command.<anonymous> (/opt/homebrew/lib/node_modules/promptfoo/dist/src/main.js:515:9)

Node.js v21.7.1
```

Thanks for the suggestion, done in https://github.com/promptfoo/promptfoo/commit/23f5b98f6322a689c4a113f232a2636cc6eb2d88
Repository: promptfoo/promptfoo
Issue: Licensing Restrictions
Root cause: External Dependency Breakage

Starting with 0.118.15, promptfoo is using a version of claude-agent-sdk that has a proprietary [license](https://github.com/anthropics/claude-agent-sdk-typescript/blob/main/LICENSE.md), which at the time of me creating this issue reads:

> © Anthropic PBC. All rights reserved. Use is subject to Anthropic's [Commercial Terms of Service](https://www.anthropic.com/legal/commercial-terms).

The Commercial Terms of Service link to https://www.anthropic.com/legal/commercial-terms.

While it's true their license has existed since June, it looks like it wasn't being applied to their JavaScript SDK until a recent release.

I am not a lawyer, so I will not speak to the openness of the license when compared to other open-source licenses like MIT, but many enterprises with Legal departments keep an allowlist of licenses for open-source packages and prevent the usage of packages that do not conform. Since claude-agent-sdk is a dependency of promptfoo, it would get blocked by association. While promptfoo cannot dictate licenses to other parties, you should try to minimize the impact to your project by moving the code to an optional plugin, if possible.

Thanks for flagging, @jeffrose.  https://github.com/promptfoo/promptfoo/pull/5935 moves this to an optional dependency.  

This is addressed in the latest version
Repository: promptfoo/promptfoo
Issue: Feature Request: Support pass^N metric for evaluating consistency across repeated test runs
Root cause: Unimplemented Feature Gap

## Feature Request: Support pass^N metric for evaluating consistency

### Background

When evaluating LLM applications, especially agents, consistency and reliability are critical metrics. The **pass^N metric** measures the probability that an agent would succeed on **all N independent attempts**, which is essential for production readiness assessment.

This differs from pass@N:
- **pass@N**: Probability that at least 1 out of N attempts succeeds (optimistic)
- **pass^N**: Probability that all N attempts succeed (consistency/reliability)

### Problem

Currently, promptfoo doesn't seem to support calculating pass^N metrics effectively because:

1. **Automatic caching**: When using `--repeat N`, results are cached, so we're not getting truly independent repeated executions
2. **No built-in pass^N calculation**: Even with derived metrics, there's no straightforward way to calculate the pass rate raised to the power of N across repeated test runs
3. **Test-level consistency tracking**: While #5847 discusses `PROMPTFOO_TEST_REPEAT_PASS_RATE_THRESHOLD`, it focuses on pass@N semantics (at least X% passing), not pass^N (all attempts passing)

### Use Case

For production LLM applications, I need to ensure:
- Each test passes consistently across multiple independent runs
- Calculate pass^4 (or pass^N) to measure reliability: `(individual_test_pass_rate)^4`
- Fail CI/CD if any test shows inconsistent behavior (low pass^N score)

### Proposed Solution

Add support for pass^N metric calculation, which could include:

1. **Disable caching for repeated runs** (or add a flag like `--no-cache-repeats`)
2. **Built-in pass^N derived metric**:
   ```yaml
   derivedMetrics:
     - name: 'pass^4'
       type: 'pass-power'
       power: 4
   ```
3. **Per-test consistency score**: Show pass^N for each individual test in the UI
4. **Threshold support**: Similar to `PROMPTFOO_PASS_RATE_THRESHOLD`, add `PROMPTFOO_PASS_POWER_THRESHOLD`

### Example

```bash
# Run each test 4 times independently (no caching)
promptfoo eval --repeat 4 --no-cache-repeats

# Assert that pass^4 >= 0.8 for production readiness
PROMPTFOO_PASS_POWER=4 PROMPTFOO_PASS_POWER_THRESHOLD=0.8 promptfoo eval
```

### Alternatives Considered

- Using `--repeat` with derived metrics: Doesn't work due to caching
- Manual repeated runs: Loses test correlation and is inefficient
- `PROMPTFOO_PASS_RATE_THRESHOLD` with #5847: Measures pass@N, not pass^N

### Additional Context

The pass^N metric is gaining adoption in the AI evaluation community for measuring agent reliability. See: [Pass@k vs Pass^k: Understanding Agent Reliability](https://www.philschmid.de/agents-pass-at-k-pass-power-k)
Repository: promptfoo/promptfoo
Issue: MCP tool with empty inputSchema causes provider error: 'object' is not of type 'object', 'boolean'
Root cause: Interface Contract Mismatch

**Describe the bug**

When using an MCP tool that has an empty or missing `inputSchema.properties`, Promptfoo transforms it to a JSON schema like `{ type: 'object', properties: {} }` for providers such as OpenAI. While this should be valid, it triggers the following error:

```
{"error":{"message":"Invalid schema for function '<name of tool>': 'object' is not of type 'object', 'boolean'.","type":"invalid_request_error","param":"tools[1].function.parameters","code":"invalid_function_parameters"}}
```

**To Reproduce**
Steps to reproduce the behaviour:
1. Define an MCP tool with no input arguments (missing `inputSchema.properties`).
2. Run a provider (e.g., OpenAI) integration using this tool.
3. Observe the error returned by the provider.

**Expected behavior**
Providers should accept a tool with no input arguments when the schema is an object with empty properties, or Promptfoo should adjust the transformation so that it is always accepted by the provider.

**System information:**
- Promptfoo version: 0.118.11
- Provider: OpenAI

**Additional context**
- The MCP spec appears to allow for the `properties` field to be optional in input schemas.
- It appears that Promptfoo’s transformation code in `src/providers/mcp/transform.ts` may not be handling empty or missing schemas in a way that's compatible with all providers.

Roughly the promptfoo config I've used.
```
providers:
  - id: openai:gpt-4.1
    config:
      mcp:
        enabled: true
        server:
          command: <path to [mcp-golang-sdk](https://github.com/modelcontextprotocol/go-sdk) based mcp server>
          name: my-mcp-server
          verbose: true
          debug: true

prompts: 
  - Please list the MCP tools for you have available.

tests: 
  - assert:
      - type: contains-any
        value: 
          - "<name of tool>"
```
Repository: promptfoo/promptfoo
Issue: Function calling for Ollama models
Root cause: Unimplemented Feature Gap

I tried to test function calling for an Ollama model, but it doesn't seem to work. Did I forget a configuration? For OpenAI it works. Is there a way to see the actual request to the model with logging?

```
 prompts:
  - 'What is the current weather in {{city}}?'
providers:
  - id: 'openai:gpt-4.1-mini'
    config:
      tools:
        - type: function
          function:
            name: get_weather
            description: Get current weather for a location
            parameters:
              type: object
              properties:
                location:
                  type: string
                  description: City and state, e.g. San Francisco, CA
                unit:
                  type: string
                  enum: ['celsius', 'fahrenheit']
              required: ['location']
  - id: 'ollama:completion:qwen3:8b'
    config:
      tools:
        - type: function
          function:
            name: get_weather
            description: Get current weather for a location
            parameters:
              type: object
              properties:
                location:
                  type: string
                  description: City and state, e.g. San Francisco, CA
                unit:
                  type: string
                  enum: ['celsius', 'fahrenheit']
              required: ['location']
tests:
  - vars:
      city: 'Amsterdam'
```
Repository: promptfoo/promptfoo
Issue: [question] Custom Model Name + No Actual run + Annotation
Root cause: Unimplemented Feature Gap

Hi, I want to evaluate different coding agents with different models and create/annotate and evaluate the output manually without any actual model runs or api calls. 

I want to give codex/claude code/droid/opencode, etc... any permutation of models for each agent the same prompt and run them > evaluate manually and annotate if they did what I asked + annotate some subjective outcome ( pretty ui or bad ui, etc) > view reporting results after my experiments with different permutations of agent/model + prompt versions.

Basically I want an option to create a custom model name ( not a full working setup, no working api). For example "codex/gpt-5-codex-mini", "codex/gpt-5-codex-high", "cc/opus-4-5" and give them "prompt20-v-1" and "prompt20-v-2" and without running them through promptfoo annotate the result and even skip the output completely or fill it manually.

Is there a way to do it now without creating any new features?

Thanks

Will use echo provider for this
Repository: promptfoo/promptfoo
Issue: "Value" column should substitute variable values in "Evaluation" tab of "Details"
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
My configuration relies heavily on templating using variables and loop expressions.  I am using `llm-rubric`, but I imagine that this functionality is the same across graders.  I need to understand the actual values that were used for my many tests across many prompts.  Deducing and manually cross-referencing them by switching back-and-forth between the "Evaluation" and "Prompt & Output" tabs is time consuming, frustrating and error-prone – it does not scale.

**Describe the solution you'd like**
Instead of something like this:

```
Did {{ agent_name }} select the expected tools across the conversation?

{% for turn in turns %}
{% if turn.expected_tool %}Turn {{ loop.index }}: Expected tool "{{ turn.expected_tool }}"
{% endif %}
{% endfor %}
```

I would like to see:

```
Did Virtual Assistant select the expected tools across the conversation?

Turn 1: Expected tool "preview_create"
Turn 2: Expected tool "preview_create"
Turn 3: Expected tool "execute_create"
```

**Describe alternatives you've considered**
Manually cross-referencing.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: promptfoo/promptfoo
Issue: Add support for threaded tests
Root cause: Unimplemented Feature Gap

I would like support for threaded tests for OpenAI assistants. I want to be able to run one message, assert the response, and then run another message with a new assertion on the same thread. I have created a draft PR for this feature: #598. I wasn't sure if you would look at a draft PR which is why I'm creating an issue as well.

@typpo, @SiXoS any update on this? I have the same requirement where I want to test few prompts with shared context in a thread. And seems like the `thread` option here: https://www.promptfoo.dev/docs/guides/evaluate-openai-assistants/#comparing-different-versions-of-the-same-assistant does not work?



My priorities at work changed a bit but I might be able to pick it up again, no ETA though.
Repository: promptfoo/promptfoo
Issue: Add subcategories to BeaverTails plugin
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently the BeaverTails promptfoo plugin appears to only have a single plugin `beavertails`. The original dataset however categorises prompts into 14 different subcategories:
* animal_abuse
* child_abuse
* controversial_topics,politics
* discrimination,stereotype,injustice
* drug_abuse,weapons,banned_substance
* financial_crime,property_crime,theft
* hate_speech,offensive_language
* misinformation_regarding_ethics,laws_and_safety
* non_violent_unethical_behavior
* privacy_violation
* self_harm	
* sexually_explicit,adult_content	
* terrorism,organized_crime
* violence,aiding_and_abetting,incitement

Currently, with only a single category, the results of our promptfoo evaluations are one-dimensional.

**Describe the solution you'd like**
Add support to this plugin for subcategories (e.g. `beavertails:self-harm`) based on the classifications in the original dataset.
This will allow us to have two-dimension evaluation results, reflecting the effectiveness of a content moderation filter against specific types of prompts.

Other users may also find this useful for narrowing down their testing to focus on specific kinds of prompts.

**Describe alternatives you've considered**
N/A

**Additional context**
Once again, I much appreciate the receptiveness and responsiveness of the promptfoo team so far :) 

This request is addressed in PR #5994
Repository: promptfoo/promptfoo
Issue: Is there a way to increase the concurrency of requests made to different models / API's?
Root cause: Documentation Deficiency

It would be great to be able to make many calls concurrently for the different prompts/models. Is this possible today? Making all the calls in serial makes many tests prohibitively time consuming.

Thanks!

@typpo apologies if this is already supported, just let me know :)

This is supported through the `evaluateOptions.maxConcurrency` config, or `--max-concurrency` or `-j` on the command line! It defaults to `4` if not set.

For example:
```
promptfoo eval -j 8
```

ah perfect! thanks @typpo.