zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Potential bug with `answer-relevance` assertion calculation
Root cause: Algorithmic Error

I was doing some cross-checks with my own test data when I notice that there might be several bugs within the `answer-relevance` assertion calculation. I am not sure if these are intentional or not:

1. ANSWER_RELEVANCY_GENERATE prompt only uses single curly brackets for the `answer` variable. This means that even though we try to render the value, the answer value won't be populated to the {answer} part of the string
Link: https://github.com/promptfoo/promptfoo/blob/e42a24c00ce1ce6d757c6256c4d53119f79f97ef/src/external/ragas.ts#L10

2. ANSWER_RELEVANCY_GENERATE is being passed to callApi without first being rendered through nunjucks.renderString, meaning all the prompts used for this part of the calculation flow would be the same no matter what is the `answer` value for the given row unit.
Link:
https://github.com/promptfoo/promptfoo/blob/e42a24c00ce1ce6d757c6256c4d53119f79f97ef/src/matchers.ts#L528C9-L528C34

3. Shouldn't the value passed to the matchesAnswerRelevance `input` arg be `test.vars.query` instead of `prompt`?
Link: https://github.com/promptfoo/promptfoo/blob/e42a24c00ce1ce6d757c6256c4d53119f79f97ef/src/assertions.ts#L884

Closing the issue as the PR for the fix is merged
Repository: promptfoo/promptfoo
Issue: Prompt Optimization
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Not related to a problem, but another solution to optimize prompts with objective score datasets. I see PromptWizard released by Microsoft which is designed mostly for OpenAI and I need an application that is capable of doing prompt improvement in an evolutionary branching style that's traceable. I really like Promptfoo based on what I've seen so far.

**Describe the solution you'd like**
 A git style evolutionary branching strategy that runs on quantifiable data and comes up with hypotheses to try and see if it improves the prompt to get better results using a Prompt Template to 

**Describe alternatives you've considered**
I've started implementing something like this for Amazon Bedrock but it's tedious to code myself and it's nice that your framework has a lot of this logic automatically. 

I have a custom job runner for Converse API that runs data in batches and then uses the results to calculate bad, medium, and good examples based on parameters provided in the request. The AI gives you the reasoning for its score for a certain quantifiable metric as part of the output as a separate field as a list of string. Then the examples as specified in the request are included with the AI score compared to the objective score and the reason why the AI gave the result with hypotheses included in this template prompt for improving the system prompt, user prompt, and tool definition. 

The template prompt predicts on an OpenAPI Bedrock compatible PromptOptimization which has fields system_prompt, user_prompt, and tool_setup configuration. 

**Additional context**
The reason I don't just implement this myself is that you have such a wide arrange of support for different platforms and options that I can't support myself, it would make it easier to work with my colleagues and base our work around an external standard. 

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1354/prompt-optimization">ENG-1354</a></p>

This is quite useless linear bot. Absolutely useless to me.
Repository: promptfoo/promptfoo
Issue: Gemini provider fails with MCP tools due to unsupported `additionalProperties` in JSON Schema
Root cause: Interface Contract Mismatch

**Describe the bug**

When using an MCP server with the Google Gemini provider, API calls fail because MCP tool schemas (generated via the MCP SDK / Zod) include JSON Schema properties that Gemini's function calling API doesn't support, specifically `additionalProperties`.

The same MCP server works correctly with Anthropic and OpenAI providers.

Error from Gemini API:
```
Invalid JSON payload received. Unknown name "additionalProperties" at 'tools[0].function_declarations[0].parameters': Cannot find field.
```

**To Reproduce**

1. Clone the reproduction repo: https://github.com/vedmakk/promptfoo-test
2. Run `cd mcp-server && bun install`
3. Add API keys to `.env` (see `.env.example`)
4. Run `npx promptfoo eval -v`
5. Observe: Anthropic and OpenAI succeed, Google Gemini fails with the above error

**Expected behavior**

The Gemini provider should sanitize MCP tool schemas to remove unsupported JSON Schema properties (`additionalProperties`, `$schema`, `default`, `title`, `pattern`, etc.) before sending them to the Gemini API, similar to how it already strips `$schema`.

**Screenshots**

N/A

**System information:**

- Promptfoo version: 0.120.8

**Additional context**

The fix should be in `src/providers/mcp/transform.ts` in the `transformMCPToolsToGoogle()` function. Currently it only removes `$schema`, but needs to recursively strip all unsupported properties.

Gemini's API only supports these schema properties: `type`, `format`, `description`, `nullable`, `enum`, `maxItems`, `minItems`, `properties`, `required`, `propertyOrdering`, `items`.

Reference: [Gemini API Function Calling Schema](https://ai.google.dev/api/caching#Schema)

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1358/gemini-provider-fails-with-mcp-tools-due-to-unsupported">ENG-1358</a></p>

**Additional finding after debugging:**

The issue is more nuanced than "all MCP tools fail with Gemini". It specifically affects **tools that define input parameters using Zod schemas**.

| Test Case | Gemini Result |
|-----------|---------------|
| Tool with no parameters | ✅ Works |
| Tool with Zod `inputSchema` | ❌ `additionalProperties` error |

**Reason:** The MCP SDK uses `zod-to-json-schema` to convert Zod schemas to JSON Schema. Zod objects are strict by default (reject unknown keys), so the converter adds `additionalProperties: false` to reflect this behavior. Gemini's API doesn't recognize this property.

This explains why some MCP servers work fine with Gemini (tools without parameters or without Zod schemas) while others fail.
Repository: promptfoo/promptfoo
Issue: The web frontend always shows one more test failure than was present in the results
Root cause: Algorithmic Error

**Describe the bug**
I run tests against the results of my custom provider. Whenever one or more assertions fail for a given test, I see one more failure that I can click through, and it's duplicated.

<img width="619" height="207" alt="Image" src="https://github.com/user-attachments/assets/12f5e4a1-f340-4c50-8b98-36916eae2485" />
<img width="612" height="171" alt="Image" src="https://github.com/user-attachments/assets/48e7a39f-02e5-4dee-897c-0d63fedcb3ad" />

All my failures are generated via custom Javascript functions returning a `GradingResult` with `pass` being `true/false`, `score` being `0/1` and `reason` being empty or set to the failure reason. 

The number of failures matches my expectations (1 / 5), only the UI display seems to be wrong.

**To Reproduce**
Unsure if it's related to using custom providers and assertions or whether this happens with regular providers too, so I can't give an exact repro. What I'm basically doing in my assertions is:

```
type GradingResult = {
  pass: boolean;
  score: number;
  reason: string;
};

function pass(): GradingResult {
  return { pass: true, score: 1, reason: "" };
}

function fail(reason: string): GradingResult {
  return { pass: false, score: 0, reason };
}
```

And then return either `pass()` or `fail(...)`.

**Expected behavior**
The web frontend only shows the exact number of failures that occurred.

**System information:**
```
================================================================================
Promptfoo Debug Information
================================================================================
{
  "version": "0.120.8",
  "platform": {
    "os": "win32",
    "release": "10.0.26100",
    "arch": "x64",
    "nodeVersion": "v25.2.1"
  },
  "env": {
    "telemetryDisabled": false,
    "telemetryDebug": false
  },
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```


<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1361/the-web-frontend-always-shows-one-more-test-failure-than-was-present">ENG-1361</a></p>
Repository: promptfoo/promptfoo
Issue: The Bedrock Converse provider inserts unnecessary sleeps even when local cache hits
Root cause: Validation Gap

**Describe the bug**
When inserting a sleep using the `delay` option to avoid exceeding the LLM API rate limit, the Bedrock Converse provider inserts unnecessary sleeps even when promptfoo hits the local cache.

**To Reproduce**
1. Configure AWS credentials and Bedrock.
2. Save the following configuration file as `tmp/promptfooconfig.yaml`

```yaml
evaluateOptions:
  maxConcurrency: 1
providers:
  - id: bedrock:converse:amazon.nova-lite-v1:0
    delay: 5000
prompts:
  - 'Why is the sky blue?'
  - 'Why is the sea blue?'
tests:
  - assert:
      - type: icontains
        value: 'blue'
```

3. Run the command twice.

```bash
$ PROMPTFOO_CONFIG_DIR=tmp/.promptfoo npm run local -- eval -c tmp/promptfooconfig.yaml -v
```

The second execution should hit the local cache, but you can see that a sleep is inserted.

**Expected behavior**
When a local cache hit occurs, unnecessary sleep should not be inserted.

**Screenshots**
The following is an excerpt from the command output.

```
[logger.ts:1] Provider type: bedrock:converse:amazon.nova-lite-v1:0
[logger.ts:1] [EvaluatorTracing] isTracingEnabled check: metadata=false, yamlConfig=false, env=false, result=false
[logger.ts:1] Calling AWS Bedrock Converse API
{
  "modelId": "amazon.nova-lite-v1:0",
  "messageCount": 1,
  "hasSystem": false,
  "hasTools": false,
  "hasThinking": false
}
[logger.ts:1] Returning cached response
[logger.ts:1] Provider response properties: output, tokenUsage, cost, metadata
[logger.ts:1] Provider response cached property explicitly: undefined
[logger.ts:1] Evaluator response = {"output":"The sky appears blue due to a phenomenon called Rayleigh scattering. Here's a step-by-ste...
[logger.ts:1] Evaluator checking cached flag: response.cached = false, provider.delay = 5000
[logger.ts:1] Sleeping for 5000ms
[logger.ts:1] Tracked token usage for bedrock:converse:amazon.nova-lite-v1:0 (AwsBedrockConverseProvider): total=224, cached=0
[logger.ts:1] Eval #1 complete (1 of 2)
```

**System information:**

- Promptfoo version: 0.120.9

**Additional context**
The cache itself appears to be functioning correctly. This is confirmed by tests that succeed even when AWS authentication is disabled and the cache is present, indicating that no unnecessary API calls are being made.

<!-- linear-linkback -->
<p><a href="https://linear.app/promptfooo/issue/ENG-1363/the-bedrock-converse-provider-inserts-unnecessary-sleeps-even-when">ENG-1363</a></p>
Repository: promptfoo/promptfoo
Issue: Feature Request: Extend Comment options in UI

The matrix view in the UI allows the user to leave a comment on each model output, which is really helpful! 

For some tasks outputs can be rather long & it'd be very helpful if we could leave multiple comments linked to different lines of the output. 

Alternatively, even having a bigger box to leave a comment in would allow for easier editing/review.

Thank you!

Hi @magdalena-mladenova-veeva, https://github.com/promptfoo/promptfoo/pull/713 adds an improved comment view. It's still not specific to individual lines of output, but it's a bigger box for you to use.  Hope this helps

amazing, thank you so much!

This change is included in release [0.56.0](https://github.com/promptfoo/promptfoo/releases/tag/0.56.0)
Repository: promptfoo/promptfoo
Issue: support for different provider for llm-rubric and embeddings
Root cause: Documentation Deficiency

We have separate deployments for each model. Would it be possible to have a "model-provider" and a "embedding-provider" separate in the defaultTest section?

Hi @pelikhan, this is already supported!  Have a look at the example here: https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/#multiple-graders

Thanks, all green now.
Repository: promptfoo/promptfoo
Issue: add header with provider information
Root cause: Unimplemented Feature Gap

We have a test scenario with multiple providers and a few tests. Would it be possible to add a header cell on the output column with the id/label of the provider?

Hi @pelikhan, I may not be understanding this ask correctly.  Currently the id/label should appear at the top of the table in the header.

![image](https://github.com/promptfoo/promptfoo/assets/310310/6c4e4151-8aba-480b-8eb2-07b9fc362ded)


Ah, I don't see it.
![image](https://github.com/promptfoo/promptfoo/assets/4175913/c00b1443-104e-4714-a626-77ed8114a283)

Could this be because I use a custom provider?

```yaml
description: summarize all files
prompts:
  - summarize
providers:
  - id: provider.mjs
    label: gpt-4
    config:
      model: gpt-4
tests:
  - vars:
      files: src/rag/markdown.md
    asserts:
      - type: llm-rubric
        value: is a summary
      - type: factuality
        value: Markdown is a text-based syntax to generate documents
      - type: icontains
        value: Markdown
```

(accidental closing)

Ah, this is happening because the provider label is intentionally hidden when all providers are the same in the eval.  Potentially could remove this behavior if people find it confusing or inconsistent https://github.com/promptfoo/promptfoo/blob/main/src/web/nextui/src/app/eval/ResultsTable.tsx#L756-L758

Great, that makes sense.

The provider is always the same JavaScript file, the interesting one is "label".

(Is label something I hallucinated?) It could de displayed on the UI of present always.

Just made #721 - I think we can just show the provider always, for consistency :)  Thanks for flagging this

This change is included in release [0.56.0](https://github.com/promptfoo/promptfoo/releases/tag/0.56.0)

We also need a way to display a label instead of the provider; since the provider id is always going to be the file name (provider.mjs in this case)

![image](https://github.com/promptfoo/promptfoo/assets/4175913/03c3ca8b-6e1e-4290-9146-c6d8c5e45f85)
Repository: promptfoo/promptfoo
Issue: Feature Request: Version Release LLM-Judged Metrics To Enable Pinning
Root cause: Unimplemented Feature Gap

I want to ensure that I can use a specific version of a specific metric (using the current prompt and generation method, for example) and be guaranteed of continuing to use that same setting going forward, regardless of whether the default changes in this project.

Updated metric functionality would create a new version. Version could default to "latest" to use the latest version.

Desired state:
```yaml
  assert:
    - type: context-faithfulness
      version: 1.0
```

closing as not relevant
Repository: promptfoo/promptfoo
Issue: OpenAI base URL environment variable: OPENAI_API_BASE_URL vs OPENAI_BASE_URL
Root cause: Interface Contract Mismatch

OpenAI's API clients support environment variables for specifying the base URL, typically used to connect to OpenAI's servers through a proxy server.

In both their Node API and their Python API, the name of the environment variable is `OPENAI_BASE_URL`:

- Node API [reference](https://github.com/openai/openai-node/blob/c6629c37fcbc5223c258169d49a30bbb668cb07d/src/index.ts#L29)
- Python API [reference](https://github.com/openai/openai-python/blob/e9724398d2bdd87ba41f199c3577303f1b80f2c7/src/openai/_client.py#L118)

Can promptfoo be updated to support the same variable? 

There's currently an `OPENAI_API_BASE_URL` added in #177, which may have been a typo? Or perhaps OpenAI added that variable later?

Agreed, let's add support for OPENAI_BASE_URL. I'll do this when I get a
moment or feel free to PR it yourself!

On Thu, Apr 25, 2024, 6:24 PM Eric Fung ***@***.***> wrote:

> OpenAI's API clients support environment variables for specifying the base
> URL, typically used to connect to OpenAI's servers through a proxy server.
>
> In both their Node API and their Python API, the name of the environment
> variable is OPENAI_BASE_URL:
>
>    - Node API reference
>    <https://github.com/openai/openai-node/blob/c6629c37fcbc5223c258169d49a30bbb668cb07d/src/index.ts#L29>
>    - Python API reference
>    <https://github.com/openai/openai-python/blob/e9724398d2bdd87ba41f199c3577303f1b80f2c7/src/openai/_client.py#L118>
>
> Can promptfoo be updated to support the same variable?
>
> There's currently an OPENAI_API_BASE_URL added in #177
> <https://github.com/promptfoo/promptfoo/issues/177>, which may have been
> a typo? Or perhaps OpenAI added that variable later?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/716>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJW4FVVW64QJTBJLC53Y7GUETAVCNFSM6AAAAABGZ4RXNGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGI3DINZWHE4DOOI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: allow provider to report updated prompt?
Root cause: Unimplemented Feature Gap

We have a framework that generates prompts on the fly (https://github.com/microsoft/genaiscript) which means that the prompt files built on the fly and sent to the LLM (This works great with the custom javascript provider model and we were able to integrate nicely with promptfoo)

This is problematic for the prompt source because it always ends up being the javascript source instead of the generated prompt, which disables a number of assertions that rely on the input.

Would there be a way to also provide the opportunity for the api provider to return the prompt it actually as part of the response? With a couple pointers, I would be happy to produce a PR.

This request definitely makes sense - I'm thinking about the best way to implement and will provide some pointers.
Repository: promptfoo/promptfoo
Issue: Adding yaml prompt support to other models
Root cause: Architectural Constraint

Hey,

I was trying to use the new `yaml` prompt file feature but realized that it's currently only supported with OpenAI.

I see this is the code that does it:

```ts
 const trimmedPrompt = prompt.trim();
    if (trimmedPrompt.startsWith('- role:')) {
      try {
        // Try YAML
        messages = yaml.load(prompt) as { role: string; content: string }[];
      } catch (err) {
        throw new Error(
          `OpenAI Chat Completion prompt is not a valid YAML string: ${err}\n\n${prompt}`,
        );
      }
    } else {
      try {
        // Try JSON
        messages = JSON.parse(prompt) as { role: string; content: string }[];
      } catch (err) {
        if (
          process.env.PROMPTFOO_REQUIRE_JSON_PROMPTS ||
          trimmedPrompt.startsWith('{') ||
          trimmedPrompt.startsWith('[')
        ) {
          throw new Error(
            `OpenAI Chat Completion prompt is not a valid JSON string: ${err}\n\n${prompt}`,
          );
        }

        // Fall back to wrapping the prompt in a user message
        messages = [{ role: 'user', content: prompt }];
      }
    }
```

Wondering if this code should be moved out to a preprocessing layer before the provider is called, that way it would be called regardless of the provider. Thoughts?


Thanks Paul - this went out in version 0.17.6!
Repository: promptfoo/promptfoo
Issue: Error running database migrations: Cannot open database because the directory does not exist
Root cause: Documentation Deficiency

I have been facing a puzzling error that was occurring only on the github actions runs. I wonder if anyone has faced before?

![image](https://github.com/promptfoo/promptfoo/assets/4175913/ea081ce8-c604-4ad5-95cc-dc04c581d219)

- workflow file:

```yaml
name: nightly test

on:
    workflow_dispatch:
jobs:
    build:
        runs-on: ubuntu-latest
        env:
            OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
            OPENAI_API_TYPE: ${{ secrets.OPENAI_API_TYPE }}
            OPENAI_API_BASE: ${{ secrets.OPENAI_API_BASE }}
            AZURE_OPENAI_API_BASE: ${{ secrets.OPENAI_API_BASE }}
            AZURE_OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
            PROMPTFOO_CACHE_PATH: .genaiscript/cache/promptfoo
            PROMPTFOO_DISABLE_UPDATE: 1
        steps:
            - name: Set up promptfoo cache
              uses: actions/cache@v2
              with:
                  path: .genaiscript/cache/promptfoo
                  key: ${{ runner.os }}-promptfoo-v1
                  restore-keys: |
                      ${{ runner.os }}-promptfoo-
            - uses: actions/checkout@v3
              with:
                  submodules: "recursive"
                  fetch-depth: 0
            - uses: actions/setup-node@v3
              with:
                  node-version: "20"
                  cache: yarn
            - run: yarn install --frozen-lockfile
            - name: typecheck
              run: yarn typecheck
            - name: compile
              run: yarn compile
            - name: run test within scripts
              run: yarn test:scripts
```

Hi @pelikhan, I think you need to set `PROMPTFOO_CONFIG_DIR` and make sure that directory is created.  Thanks for flagging - I'll make sure the documentation is updated to make this point clear.

does the github action (https://www.promptfoo.dev/docs/integrations/github-action/) support wildcards for the config files? I could definitely be switching to using that one.

> Hi @pelikhan, I think you need to set `PROMPTFOO_CONFIG_DIR` and make sure that directory is created. Thanks for flagging - I'll make sure the documentation is updated to make this point clear.

Yup, that fixed it.
Repository: promptfoo/promptfoo
Issue: Request: JSON schema for `promptfooconfig.yaml`
Root cause: Unimplemented Feature Gap

It would be cool to publish a JSON schema for `promptfooconfig.yaml`. Then within an IDE one can:

- Hover over fields and get a description of them
- Rely on IDE warnings to iterate on mistakes in a `promptfooconfig.yaml`

For example PyCharm natively supports JSON schema integrations: https://www.jetbrains.com/guide/dotnet/tips/json-schema/

Then for bonus points the JSON schema could be registered in the JSON Schema Store: https://www.schemastore.org/json/ --> https://github.com/SchemaStore/schemastore/tree/master/src/schemas/json

---

For reference, here's how I personally make JSON schemas:

1. Express schema as [Pydantic `BaseModel`](https://docs.pydantic.dev/latest/api/base_model/)
2. Export a JSON schema via [`BaseModel.model_json_schema()`](https://docs.pydantic.dev/latest/api/base_model/#pydantic.BaseModel.model_json_schema)
3. Add the JSON schema to PyCharm for the file name (e.g. `promptfooconfig.yaml`)

This is now available at https://promptfoo.dev/config-schema.json

You can put this at the top of your YAML: 
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
```