zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Req: Allow specifying typescript functions that return prompts
Root cause: Unimplemented Feature Gap

The templating works nicely up until prompts get more complicated. Currently I'm constructing the prompt and history of messages in code, where there's stages of extracting data from objects and encoding them in various ways. This doesn't play so nicely with (e.g. put a json encoded string inside  a). Even long multi-line prompts are awkward in a json file with no newlines.

If prompts could (as well as being a file or list of files) include functions that take as an argument the named variables and return a string, I think that could more neatly integrate with existing typescript projects.

This could probably fit in the yaml and be callable from the cli, but a first pass could be to add this functionality to evaluate - so it's available within the node package. Prompts could have a renderer on them which defaults to renderPrompt if not defined.

Would you be interested in this being added? I think the change could be relatively small for the node package change, I'm less sure of dynamic imports for adding it to the yaml. Perhaps it would work more easily for js config.



Hi @IanCal,

I see what you're saying.  Something similar is supported by [custom providers](https://github.com/promptfoo/promptfoo/blob/main/examples/node-package/index.js#L6-L14), but this functionality doesn't extend to the prompts themselves.

I'm definitely open to adding this support for prompts as well.
Repository: promptfoo/promptfoo
Issue: Importing provider tools from a file does not work even after #1342
Root cause: Validation Gap

**Describe the bug**

The file is taken as a string instead of an object. (tried with json, yml and txt)

![image](https://github.com/user-attachments/assets/915fae12-8760-44b6-b524-3b82ffb7a36d)

**To Reproduce**

Here is the provider code : 
```yml
  - id: openai:chat:gpt-4o-mini
    config:
      label: gpt-4o-mini
      temperature: 0
      max_tokens: 1024
      tool_choice: 'auto'
      tools:
        - file://./tool-event-api.json
```

tool-event-api.json
```
{
  "type": "function",
  "function": {
    "name": "event_api",
    ...
  }
}
```

**Expected behavior**

The provider should receive the tool configuration.

**System information (please complete the following information):**

- Promptfoo version: 0.74.0

**Additional context**

Discussion on Discord where first discussed : https://discord.com/channels/1153767083381903389/1268551965357641728
When looking up the value from a python script, here is what i get : `'tools': ['file://./tool-event-api.txt']` Instead of the file content


I've also tried some less conventional stuff like this :

```yaml
  - id: openai:chat:gpt-4o-mini
    config:
      label: gpt-4o-mini
      temperature: 0
      max_tokens: 1024
      tool_choice: 'auto'
      tools:
        - $ref: "#/defaultTest/vars/toolEventAPIFile"
```

```yaml
defaultTest:
  vars:
    toolEventAPIFile: file://./tool-event-api.json
```

In this case, toolEventAPIFile is correctly loaded and can be used. However, `tools` contains ['file://./tool-event-api.json']...

Thanks for flagging - #1357 fixes the case where `tools` is set to a filepath.  Note that right now only a single filepath is supported, e.g.

```yaml
  - id: openai:chat:gpt-4o-mini
    config:
      tools: file://external_tools.yaml
```

This should be good in the latest release
Repository: promptfoo/promptfoo
Issue: promptfoo doesn't detect false or 0 outputs, I get '[FAIL] No output'
Root cause: Validation Gap

**Describe the bug**
I use an http provider, my endpoint returns a json like:
```json
{
    "blocked": true
}
```
When the value is `true` it works as expected, but when it is `false`, promptfoo doesn't detect the value, and I get `[FAIL] No output` message. The behaviour is similar if I use 0 and 1 values (promptfoo doesn't detect 0).

**To Reproduce**
1. Save this python code to `test_api.py`:
```python
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class RequestModel(BaseModel):
    message: str

class ResponseModel(BaseModel):
    blocked: bool

@app.post("/status", response_model=ResponseModel)
def post_status(request: RequestModel):
    if request.message == 'this is true':
        return_value = True
    else:
        return_value = False
    
    return ResponseModel(blocked=return_value)

```
2. Run the API with uvicorn:
```bash
uvicorn test_api:app
```
3. Save this configuration to a `promptfooconfig.yml` file:
```yaml
description: "bug test"

prompts:
  - "{{question}}"

providers:
  - id: 'http://127.0.0.1:8000/status'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        message: '{{prompt}}'
      responseParser: 'json.blocked'
      responseType:

defaultTest:
  assert:
    - type: equals
      value: "{{is_blocked}}"
tests:
  - vars:
      question: "this is true"
      is_blocked: true
  - vars:
      question: "this is not true"
      is_blocked: false

```
3. Run promptfoo with:
```bash
promptfoo eval
```
**Expected behavior**
Both test pass, as the API return `true` for the first call and `false` for the second call. 

**System information (please complete the following information):**

- Promptfoo version: 0.74.0


Thanks for the report! #1356 should fix

This should be good in the latest release
Repository: promptfoo/promptfoo
Issue: TestSuite Schema Validation Error when I use boolean values
Root cause: Validation Gap

**Describe the bug**
I get `TestSuite Schema Validation Error` whenever I set a boolean value as `tests\vars`

**To Reproduce**
Run promptfoo with the following configuration:
```yaml
# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
description: "bug test"

prompts:
  - "{{question}}"

providers:
  - id: 'http://127.0.0.1:8000/status'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        message: '{{prompt}}'
      responseParser: 'json.blocked'
      responseType:

defaultTest:
  assert:
    - type: equals
      value: "{{is_blocked}}"
tests:
  - vars:
      question: "this is true"
      is_blocked: true
  - vars:
      question: "this is not true"
      is_blocked: false
```

**Expected behavior**
It should be possible to set a value to boolean.

**Screenshots**
The error screen:
<img width="600" alt="image" src="https://github.com/user-attachments/assets/9819b3e5-cc4c-4d9e-8de1-50397aead944">
**System information (please complete the following information):**

- Promptfoo version: 0.74.0



Released in 0.76.0
Repository: promptfoo/promptfoo
Issue: Output to google-sheets failed
Root cause: Validation Gap

**Describe the bug**
Writing to the sheet causes an error and cannot be executed.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. OpenAI API Key and Google Cloud Credential write to .env
2. Edit 'examples/google-sheets/promptfooconfig.yaml' Writing output to a sheet
3. Run 'npx promptfoo@latest eval'
4. See error

```
Error: Invariant failed: Unsupported output file format undefined. Please use one of: csv, html, json, txt, yaml, yml.
    at invariant (C:\Users\tkakizaki\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\tiny-invariant\dist\tiny-invariant.cjs.js:14:11)
    at writeOutput (C:\Users\tkakizaki\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\util\index.js:93:34)
    at runEvaluation (C:\Users\tkakizaki\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\commands\eval.js:140:46)
    at async doEval (C:\Users\tkakizaki\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\commands\eval.js:240:5)
```

**Expected behavior**
Output result to google sheets.

**Screenshots**
If applicable, add screenshots to help explain your problem.

![screeenshot02](https://github.com/user-attachments/assets/0e70f803-06db-4297-ae80-5fbd2b670169)


**System information (please complete the following information):**

- Windows 10
- nodejs 20.16.0
- Promptfoo version: 0.74.0

**Additional context**
Add any other context about the problem here.


Thanks for the report. #1367 should fix this.

This should be good in the latest release
Repository: promptfoo/promptfoo
Issue: Possible to combine scenarios with Google Sheets test cases?
GitHub labels: Feature Request

I'm trying to combine scenarios with google sheets for the test cases. Is this supported? If so, is there something wrong with my syntax? The below causes the following error:

```yaml
scenarios:
  - config:
      - vars:
          csvFilePath: industries.csv
      - vars:
          csvFilePath: industries-with-descriptions.csv
    tests: https://docs.google.com/spreadsheets/d/1OsJDtkyshJVzOp-iw_ZAEbM2ZflNHEqIbpo/edit?usp=sharing
```

```bash
npx promptfoo@latest eval
TestSuite Schema Validation Error:

  {"_errors":[],"scenarios":{"0":{"_errors":[],"tests":{"_errors":["Expected array, received string"]}},"_errors":[]}}

Please review your promptfooconfig.yaml configuration.
/Users/jedidiah/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/evaluator.js:388
                    ]).map((test) => {
                       ^

TypeError: (scenario.tests || [{}]).map is not a function
    at Evaluator.evaluate (/Users/jedidiah/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/evaluator.js:388:24)
    at evaluate (/Users/jedidiah/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/evaluator.js:806:15)
    at runEvaluation (/Users/jedidiah/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/commands/eval.js:114:56)
    at async doEval (/Users/jedidiah/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/commands/eval.js:240:5)

Node.js v20.10.0
```

Thanks for flagging - This is unsupported right now

Got it. Thank you 🙏🏻

Hey! We looked into this but decided not to add support for it - scenarios are a bit of a legacy feature and we're moving away from them.

The good news is there's a straightforward workaround: just add `csvFilePath` as a column directly in your Google Sheet. So instead of two scenario configs, you'd duplicate your test rows - one set with `csvFilePath: industries.csv` and another with `csvFilePath: industries-with-descriptions.csv`.

Your config becomes:

```yaml
prompts:
  - "Your prompt here with {{csvFilePath}} and {{input}}"
providers:
  - openai:gpt-5.2
tests: https://docs.google.com/spreadsheets/d/1OsJDtkyshJVzOp-iw_ZAEbM2ZflNHEqIbpo/edit
```

It's a bit more repetitive in the spreadsheet, but honestly it ends up being easier to work with since each row is fully self-describing.

If you really want to keep the configs separate, you can also just make two config files and run `promptfoo eval -c config1.yaml -c config2.yaml`.

Closing this out, but let us know if the workaround doesn't cover your use case.
Repository: promptfoo/promptfoo
Issue: Possible to reuse test assertions from the __expected column of a Google Sheet?
Root cause: Unimplemented Feature Gap

I'm trying to re-use a complex `is-json` assertion from a Google Sheets-supplied test list. Re-using the assertion works properly when directly referencing it from the yaml config, but when I put the same string in the Google Sheets __expected column, it appears to be evaluated as an `equals` against the literal string. Is reusing assertions from external test files just not supported? I imagine as an alternative approach I could make it work with the `defaultTest` feature instead in this particular case…

<img width="1500" alt="image" src="https://github.com/user-attachments/assets/47a1b6c1-3914-4069-b031-4f0aa2d1c11b">
<img width="283" alt="image" src="https://github.com/user-attachments/assets/aea9b210-985a-4659-983a-fec02883f97d">


Unfortunately not - `$ref` is a feature of json schema, so it's not supported in Google Sheets which is a non-json/non-yaml format.

Got it. Thanks for the quick reply. I'll go with the `defaultTest` approach for now.

p.s. Promptfoo is a great project—it'll have a big impact on our ability to ship reliable prompts. Thanks for making and sharing and supporting it! 🙏🏻
Repository: promptfoo/promptfoo
Issue: Eval grouping 
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Right now we can see individual evals under the evals tab. But a functionality to group the evals together will be nice.

**Describe the solution you'd like**
clear grouping of evals will help to identify no of evals for a certain use-case and will create a logical grouping.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: promptfoo/promptfoo
Issue: Google Gemini provider as LLM grader 400 error
Root cause: Interface Contract Mismatch

**Describe the bug**
When using `google:gemini-pro` as LLM grader, I get the following error:
```
"error":"API did not return any candidate responses: {\"error\":{\"code\":400,\"message\":\"Invalid JSON payload received. Unknown name \\\"content\\\" at 'contents[0]': Cannot find field.
```
If I understand it correctly, the API was updated and expects a different format now: https://ai.google.dev/api/generate-content#method:-models.generatecontent

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Use `promptfoo init` to create the default `promptfooconfig.yaml`
2. (Optional if you don't have an OpenAI API Key set) Replace the providers with: `"google:gemini-pro"` so you only need to set the `GOOGLE_API_KEY` in the following step
4. [Get a Google AI API Key](https://aistudio.google.com/app/apikey) and run `export GOOGLE_API_KEY={your api key from step 1}`
5. Set `google:gemini-pro` as the LLM grader as described [here](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#overriding-the-llm-grader) e.g. by adding the following to your `promptfooconfig.yaml`: 
```yaml
defaultTest:
  options:
    provider: google:gemini-pro
```
6. Run `promptfoo eval`

**Expected behavior**
The `llm-rubric` assertion on `new york city` topic should be successful.

**System information (please complete the following information):**

- Promptfoo version: 0.75.2

**Additional context**
Add any other context about the problem here.


Hi @Peetee06, this was fixed in #450 (v0.100.0). Thanks for reporting!

The 400 error with the `content` field was due to the Gemini provider not converting OpenAI chat format to the correct Gemini format (which uses `parts` instead of `content`). That format conversion was added for `llm-rubric` and other model-graded assertions.

To use Gemini as a grader now:
```yaml
defaultTest:
  options:
    provider: google:gemini-2.0-flash  # or google:gemini-2.5-pro or google:gemini-3.0-preview 
```
Repository: promptfoo/promptfoo
Issue: Asserting on stop reason or finish reason
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
The problem is that the current prompt evaluation library, `promptfoo`, does not provide a way to easily assert on the `finish_reason` (OpenAI) or `stop_reason` (Anthropic) parameters returned by the language model API. These parameters indicate why the model stopped producing the completion, which is valuable information for testing and evaluating prompts.

**Describe the solution you'd like**
I would like to see `promptfoo` add a new assertion type that allows checking the `finish_reason` or `stop_reason` value. This could be done by adding a new assertion type like this:

```json
{
   "type": "is-stop-reason",
  "value": "stop",
}

```

This would allow writing assertions like `{"type": "is-stop-reason", "value": "stop"}` or `{"type": "is-stop-reason", "value": "max_tokens"}`.

The possible values should be standardized across different model providers.

**Describe alternatives you've considered**
An alternative solution is to include the raw LLM API response in the context for the `"javascript"` assertion type. This would allow writing custom assertions like:

```json
{
  "type": "javascript",
  "value": "context.rawOutput.stop_reason == 'stop'"
}

```

However, this would require a different assertion for each model provider, as the response format may vary.


Hi @tomquist,

Sorry for not closing this sooner - this has been working since PR #3879 was merged in July 2025 (version 0.116.7).

You can now use the `finish-reason` assertion type:

```yaml
assert:
  - type: finish-reason
    value: stop
```

This works with OpenAI and Anthropic providers, with normalized values: `stop`, `length`, `content_filter`, `tool_calls`.

See the [Deterministic Assertions documentation](https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic#finish-reason) for more details.

Thanks for the suggestion! Feel free to comment or re-open if you have any questions.
Repository: promptfoo/promptfoo
Issue: Support `gpt-4o-2024-08-06` model
Root cause: Documentation Deficiency

When I try to run OpenAI latest model I get this error:
```
Error: Unknown OpenAI model type: gpt-4o-2024-08-06. Use one of the following providers: openai:chat:<model name>, openai:completion:<model name>, openai:embeddings:<model name>, openai:image:<model name>
```

The issue occurs because the latest model wasn't mapped yet in promptfoo source code. You can see it here: https://github.com/promptfoo/promptfoo/blob/main/src/providers/openai.ts#L50-L56

I would suggest to not throw an error when a model isn't found. It blocks us, the clients of this package, from using latest models until promptfoo authors update the mapping. A warning might be better.


I assume you are trying `openai:gpt-4o-2024-08-06`?  You should be able to use `openai:chat:gpt-4o-2024-08-06` without waiting for an update.  

The reason this is necessary is that it wouldn't know whether to use the `chat` or `completion` API.  Though nowadays it's a pretty safe bet that an unknown model is a chat model.

@typpo thanks for the quick answer!

Landed in 0.76.0
Repository: promptfoo/promptfoo
Issue: Allow disabling Nunjucks
Root cause: Validation Gap


Good news! This is already possible by setting the environment
variable PROMPTFOO_DISABLE_TEMPLATING=1

On Fri, Aug 9, 2024, 6:16 PM Matan Yadaev ***@***.***> wrote:

> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/1405>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJUKPLT2BL2C7XFGJNLZQVSXJAVCNFSM6AAAAABMJLSTQ6VHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ2TQOBTGU2DGNY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


@typpo brilliant, thanks!

@typpo Disabling Nunjucks causes `llm-rubric` to fail. I'm not using templating in my assertion. Is that a bug?
![CleanShot 2024-08-10 at 12 42 40](https://github.com/user-attachments/assets/a4876161-1fb6-4f3a-8c8c-c106319ed7a6)


llm-rubric itself is a nunjucks template. May I ask why you want to disable
templating? That could help for suggesting alternatives.

On Sat, Aug 10, 2024, 2:44 AM Matan Yadaev ***@***.***> wrote:

> @typpo <https://github.com/typpo> Disabling Nunjucks causes llm-rubric to
> fail. I'm not using templating in my assertion. Is that a bug?
> CleanShot.2024-08-10.at.12.42.40.png (view on web)
> <https://github.com/user-attachments/assets/a4876161-1fb6-4f3a-8c8c-c106319ed7a6>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/1405#issuecomment-2280668059>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJSGHSIZDMMGG2CVTGTZQXOHTAVCNFSM6AAAAABMJLSTQ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBQGY3DQMBVHE>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


I run an evaluation against tens of websites content (HTML), there are plenty of websites that probably using Angular.js and have `{{variable}}` in their HTML, and it causes the evaluation to crash because it doesn't find the variable in the context, obviously.
Currently I fixed by removing these brackets from the HTML(`html.replaceAll(/({{|}}/g, '')`), but ideally, I wouldn't like to run Nunjucks at all as I'm using Promptfoo in Node.js and I can run expressions in the code instead.

Hi @MatanYadaev, We’ve just released [0.85.0](https://github.com/promptfoo/promptfoo/releases/tag/0.85.0), which includes changes to how Nunjucks templating is handled only in the grader context (overriding the disable option just for those cases).  Please give it a try and let us know if this resolves the issue you're facing. If you encounter any further problems, feel free to report back!
Repository: promptfoo/promptfoo
Issue: Document why OPENAI_API_KEY in getting started
Root cause: Documentation Deficiency

I am looking into adopting `promptfoo`, and in the getting started step 3, randomly it asks for an OpenAI API key.

https://promptfoo.dev/docs/getting-started

Why is the OpenAI API key needed?  Can we document this (or add a link to the explanation) into the Getting Started instructions?

---

Is there some way we can have the "getting started" instructions not require OpenAI being set up?

Yes, if you aren't using GPT then you don't need to set your OpenAI API key!  `promptfoo init` creates a config that uses GPT by default, hence its inclusion in Getting Started.

Also worth noting that if you use any model-graded assert, you will have to [override the LLM grader](https://promptfoo.dev/docs/configuration/expected-outputs#overriding-the-llm-grader) (and potentially the rubric prompt) to select a model other than GPT.

Thanks, I can contribute a docs PR for this shortly

I was in the docs anyway so I made an edit - feel free to add further clarification if you see a need :)

Just to confirm, is the default Getting Started supposed to use OpenAI's chat endpoint or completion endpoint?

It uses OpenAI's chat endpoint by default.

Thanks, I opened https://github.com/promptfoo/promptfoo-docs/pull/6 to upstream my learnings, thank you!
Repository: promptfoo/promptfoo
Issue: Allow to configure Ajv
Root cause: Configuration Error

AJV is used in the `is-json` assertion: https://github.com/promptfoo/promptfoo/blob/35a33c4a86f5bad01b1427d1fcae7e08762b1df8/src/assertions.ts#L69

AJV defaults are strict, and because no config is passed to it, the created instance is strict.

The issue I face is with passing an `example` property to the JSON schema. The `example` property isn't mentioned in the JSON Schema spec, and therefore, when AJV sees it, it throws an error on strict mode.

It can be resolved by making AJV not strict (`new Ajv({strict: false})`) or by setting allowed keywords (`new Ajv({keywords: ['example']})`).

Being able to set a default Ajv configuration, or using an instance that I set, could help me resolve this issue.

![CleanShot 2024-08-12 at 00 30 19@2x](https://github.com/user-attachments/assets/a4808ecd-bad0-42b3-a6ae-3457b93be380)


@MatanYadaev https://github.com/promptfoo/promptfoo/pull/1415 should solve your issue. Let me know if it looks good to you. 

@mldangelo Thanks, it works well!
Although the ideal solution IMO would be to being able to pass my own instance of AJV, because there are tons of configurations and plugins, but your PR solves my issue so it's great. :)