zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Use promptfoo for output evaluation/visualisation but not inferencing?
Root cause: Documentation Deficiency

Thank you for the fantastic tool! 

If I understand it correctly, promptfoo sends an API call to a model, then evaluates/visualises model outputs. Is it possible to de-couple the two steps & only employ it for evaluation, given model output?


Hi Magdalena,

You can try the standalone assertions feature: https://promptfoo.dev/docs/configuration/expected-outputs/#running-assertions-directly-on-outputs.  This lets you run outputs directly against a list of assertions and view the results.

Let me know if this fits what you're looking for!

This is great, thank you! 

In the above set-up, a list of outputs are provided & subsequently shown in separate rows in the resulting evaluation table. Would there by any way to specify metadata about each output (e.g. model/prompt it was created with) & show these side-by-side rather than on separate rows?



#581 implements this feature in a very basic way.  Right now it allows you to specify your output JSON like this:

```json
[
  {"output": "Hello world", "tags": ["foo", "bar"]},
  {"output": "Greetings, planet", "tags": ["baz", "abc"]},
  {"output": "Salutations, Earth", "tags": ["def", "ghi"]}
]
```

And the tags are shown in the web UI as a comma-separated list.

In the future this feature can certainly be improved, but hope it helps for now.  It will be included in the next published release.

Published in [0.49.0](https://github.com/promptfoo/promptfoo/releases/tag/0.49.0)!

Thank you so much!

Was missing the matrix view in this setting, and I just realised that one can use a custom script (e.g. following https://promptfoo.dev/docs/providers/python), where instead of calling their app API, they use the function to open up pre-existing model results. Just adding it here in case others might find it helpful as well :)
Repository: promptfoo/promptfoo
Issue: Security Vulnerability Report

Hello,

Thank you for your time and attention.

I have submitted a security-related issue in the "Security" tab. I would be grateful if you could please review it at your earliest convenience.

Best regards,
Kim SooHyun.

<img width="336" height="22" alt="Image" src="https://github.com/user-attachments/assets/cf7857f3-400e-499d-964c-48124115371c" />

<img width="337" height="21" alt="Image" src="https://github.com/user-attachments/assets/08882beb-4432-4a0c-b0a6-0e04f5a283ea" />



@mldangelo @vsauter 

I am writing to inquire about the status of the vulnerability I recently reported.

Hi @im-soohyun, thank you again for the responsible report and for your patience. We followed up with you privately through our security reporting process and have addressed the issue you reported. For anyone else following along, this was handled via a private report in the Security tab and verified there. I will close this issue out for now. Please feel free to re-open it or follow up if you have any additional questions or notice anything else.
Repository: promptfoo/promptfoo
Issue: JSON Prettify option does not work when Markdown rendering is enabled in WebUI
Root cause: Algorithmic Error

**Describe the bug**
JSON Prettify option does not work when Markdown rendering is enabled simultaneously in the WebUI settings. When both "Render Markdown content" and "Prettify JSON outputs" options are enabled, JSON outputs are rendered as plain text through Markdown processing instead of being properly prettified with syntax highlighting and formatting.

Additionally, if JSON content contains parts that can be interpreted as Markdown, those parts get unintentionally rendered as Markdown markup, corrupting the JSON display.

**To Reproduce**
1. 
2. Set up a provider (Python/JS/TS) that returns object outputs
3. Configure transform functions to extract relevant parts for evaluation (content can be either Markdown text or JSON)
4. Open the Promptfoo WebUI and navigate to Settings
5. Enable both "Render Markdown content" and "Prettify JSON outputs" options
6. Run an evaluation and view results in the evaluation table
7. Observe that JSON output is displayed as plain text instead of being prettified

**Expected behavior**

When both options are enabled, the system should intelligently handle different content types:
- Valid JSON content should be prettified with proper formatting and syntax highlighting
- Non-JSON text content should be rendered as Markdown
- Both display options should work rather than conflicting with each other
- I think JSON formatting should take priority over Markdown rendering for valid JSON content


**Screenshots**

<img width="1175" height="540" alt="Image" src="https://github.com/user-attachments/assets/3c715bed-1447-4b4d-8124-276b7f5da06f" />

https://gist.github.com/pokutuna/8fc497e03a2e81259cd95d75cba50927

**System information:**

- Promptfoo version: 0.118.8 (latest when this issue created)

**Additional context**

The issue is in `src/app/src/pages/eval/components/EvalOutputCell.tsx`. The rendering priority checks Markdown before JSON prettifying, so when `renderMarkdown` is enabled, it processes ALL text including valid JSON, and the `prettifyJson` condition is never reached.

Reorder the conditional checks to prioritize JSON detection over Markdown rendering - check for valid JSON first, then fallback to Markdown for non-JSON content.
Repository: promptfoo/promptfoo
Issue: Azure OpenAI Vision expect data:image/jpeg;base64, prefix for local image input
Root cause: Documentation Deficiency

**Describe the bug**
In the documenation https://www.promptfoo.dev/docs/providers/azure/#using-vision-models the local path to a file uses auto base64 encoding but does not pass the _data:image/jpeg;base64,_ prefix which is necessary to pass the image to Azure OpenAI.

**To Reproduce**
Run _promptoo eval_ with the example in the documenation with a local file and you get a "_BadRequest Invalid image URL. The URL must be a valid HTTP or HTTPS URL, or a data URL with base64 encoding._" error from Azure OpenAI 

**Fix**
Use the data:image prefix in the prompt template in the yaml like this
```yaml
prompts:
  - |
    [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What do you see in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "data:image/jpeg;base64,{{image_url}}"
            }
          }
        ]
      }
    ]

tests:
  - vars:
      image_url: file://assets/image.jpg # Local file (auto base64)
```
and now _promptfoo eval_ runs successfully

**Addtional information**
Tested with Azure OpenAI GPT-4.1 and GPT-5-Chat
promptfoo version 0.118.8



Thank you! 🙏
Repository: promptfoo/promptfoo
Issue: OpenRouter integration errors
Root cause: Documentation Deficiency

Hey!

Seems like the openrouter provider isn't working.

Using this test file:

```
description: openrouter test
prompts:
  - Write a tweet about bananas
providers:
  - id: openai:chat:gpt-3.5-turbo
    config:
      apiBaseUrl: https://openrouter.ai/api
      apiKey: [real key]
tests:
  - vars:
    assert:
      - type: icontains
        value: banana
```

returns this error :

![image](https://github.com/promptfoo/promptfoo/assets/13338950/170c53e9-1c0e-48f9-9b6e-e75ecbec1d18)




Hey @dorperetz, it should be:
```
apiBaseUrl: https://openrouter.ai/api/v1
```
Let me know how that works. I'll update the doc!
Repository: promptfoo/promptfoo
Issue: Feature Request: CLI command to retrieve results after manual labelling in csv file
Root cause: Unimplemented Feature Gap

Hello, would it be possible to add a CLI to download the results from an experiment in the web UI, ideally in a csv file (rather than manually fetching the json files). Ideally the columns would be:
- the prompt
- the variables
- the score
- the result (pass / fail)
- Whether the result is a manual label or not (this one is very important)

Yes, this should be fairly straightforward.  I can add this to https://github.com/promptfoo/promptfoo/pull/482

Change was released in [0.50.0](https://github.com/promptfoo/promptfoo/releases/tag/0.50.0)
Repository: promptfoo/promptfoo
Issue: Enhance Reporting Dashboard
Root cause: Documentation Deficiency

The reporting dashboard in Promptfoo needs enhancements to improve usability and visualisation. The scope includes:
- Updating the dashboard theme to provide a cleaner, modern, and more user-friendly interface.
- Reviewing and updating the underlying data structure to better support flexible reporting, scalability, and improved analytics.

The goal is to create a more intuitive dashboard experience while ensuring the data structure can accommodate future enhancements.

I’m looking for guidance, could you please suggest the best approach or point me to any available resources?

i am working on using grafana for a nice promptfoo dashboard but I am not sure which of the underlying db tables the current promptfoo dashboard is using to display the very nice stats. But we need just a slightly different look that can be shared with stakeholders

Now sure which table is used to visualize the below.

<img width="1738" height="303" alt="Image" src="https://github.com/user-attachments/assets/e804b7d8-5765-4e1d-b6c2-1db9550a5b58" />

ok, sorted it out.

<img width="1261" height="523" alt="Image" src="https://github.com/user-attachments/assets/84df8b04-45d0-4361-8636-562341d3e40e" />

Hello @yosiasz , I’m also working on displaying the results in Grafana. Did you use the existing SQLite database as the source to move data into Grafana? Personally, I think SQLite may not be the best option for this. I tried storing the test results in a MySQL database and then connecting it with Grafana. Could you explain a bit about your process?

yes I am using the out of the box SQLite db for promptfoo. SQLite indeed may not be the best option, but MySQL is a more robust enterprise option.

Then it is just a matter of querying the `eval_results` table in your grafana dashboard. The data in that table is in json format so you will need to use json extraction functions to get what you need out of it.


Hi @yosiasz , Which DB did you use for this process?
Repository: promptfoo/promptfoo
Issue: Integration of Google VertexAI gcloud auth
Root cause: Unimplemented Feature Gap

Hi!

So far for VertexAI users, the `VERTEX_API_KEY` must be provided as described here: https://www.promptfoo.dev/docs/providers/vertex

However its not really following best practices for GCP users and hard to implement for CI/CD
Would there be a way to use the google client client to authenticate with gcp projects without providing any API key? I see there are some google cloud node.js libraries: https://github.com/googleapis/google-cloud-node but I am not sure what would be the best way to implement this

Thanks for the suggestion.  Based on my understanding, we'd need to switch over to a client library for vertex AI and/or REST authentication with application default credentials ([docs](https://cloud.google.com/vertex-ai/docs/workbench/reference/authentication#client-libraries)).  That probably means using [`google-auth-library`](https://github.com/googleapis/google-auth-library-nodejs)

#580 updates the provider to use Application Default Credentials

This means a machine that is properly configured with a service account, or logged into the gcloud CLI, should "just work"

Will be included in the next published release!

amazing, thanks a lot!

Published in [0.49.0](https://github.com/promptfoo/promptfoo/releases/tag/0.49.0)!
Repository: promptfoo/promptfoo
Issue: Requesting for help - Rate limiting Error
Root cause: Configuration Error

Hello Team,

Firstly thanks for this awesome tool. I am using openapi and get the rate-limiting error from promptoo tasks endpoint. I would like app to use my own  endpoint not could api. My configuration involves providers, export openapi key (terminal), REST endpoint of target. I am running plugins redteam. Could you please help me?

What do you have concurrency set at?

On Sat, Sep 27, 2025, 6:31 PM Vikram ***@***.***> wrote:

> *mikemicky4321* created an issue (promptfoo/promptfoo#5752)
> <https://github.com/promptfoo/promptfoo/issues/5752>
>
> Hello Team,
>
> Firstly thanks for this awesome tool. I am using openapi and get the
> rate-limiting error from promptoo tasks endpoint. I would like app to use
> my own endpoint not could api. My configuration involves providers, export
> openapi key (terminal), REST endpoint of target. I am running plugins
> redteam. Could you please help me?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/5752>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJXQGZMGJR6FND73MHT3U427PAVCNFSM6AAAAACHV67Z52VHI2DSMVQWIX3LMV43ASLTON2WKOZTGQ3DAOJTGAZTGNY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Hello @typpo,
I didn't set any concurrency. I am running using cli. I guess it is using default parameters. Please find the promptfooconfig.yaml file below.

[promptfooconfig.yaml](https://github.com/user-attachments/files/22583067/promptfooconfig.yaml)

I would like promptfoo to use my providers not promptfoo cloud api. 


Hello @typpo,
Could you please support me? I am waiting for your response.

Hi @mikemicky4321, thanks again for using Promptfoo and for the detailed report. Since this issue was opened we have made a number of improvements to rate limiting and provider configuration on the tasks endpoint, and I am not able to reproduce the behavior you described with the current version.

I am going to close this out for now to keep the issue tracker tidy. If you are still seeing rate limiting problems or having trouble ensuring requests go only to your own providers, please either re-open this issue, open a new one with your config, or email me directly at michael@promptfoo.dev and I will be happy to take a closer look.
Repository: promptfoo/promptfoo
Issue: Save evals directly in self-hosted machine

How to save evals run locally in a self hosted machine?

The documentation only indicates how to use the self hosted machine with promptfoo share: https://www.promptfoo.dev/docs/usage/self-hosting

Evals are actually saved directly on the self-hosted machine already.  They are written to `/root/.promptfoo` on the docker image.  You can persist the volume by passing an argument like `-v /path/to/local_promptfoo:/root/.promptfoo` to `docker run`.

I've updated the documentation to explain this better in [2a57f4d](https://github.com/promptfoo/promptfoo/commit/2a57f4d45160f14194c33f359959d62e421a2f77).

Let me know if you have any other qs!

Hi, but what if promptfoo is deployed in another machine?

My team and myself want to be able to run the eval in their own machine and push the results to a machine which serves the promptfoo server.

Also for security reasons we cannot directly ssh to the other machine to manually copy the results and we can only reach predefined ports.
Repository: promptfoo/promptfoo
Issue: Update VLGuard plugin to use the original VLGuard dataset
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
The new VLGuard plugin added in https://github.com/promptfoo/promptfoo/issues/5231 / https://github.com/promptfoo/promptfoo/pull/5243 uses the dataset at https://huggingface.co/datasets/kirito011024/vlguard_unsafes

This dataset is derived from the original at https://huggingface.co/datasets/ys-zong/VLGuard by extracting only unsafe records, and transforming the data format.

Problems with the derivative dataset:
1. Unlike the original which is licensed under MIT, the derivative dataset has no licence attached to it
2. The absence of safe records (which are present in the original set) reduces its utility for benchmarking evaluations

**Describe the solution you'd like**
If it's not too much to ask, we'd like the VLGuard plugin for promptfoo to be updated to use the original https://huggingface.co/datasets/ys-zong/VLGuard dataset

**Describe alternatives you've considered**
A more general plugin that can flexibly take a link to an arbitrary dataset and other information about the dataset's structure. However, there don't appear to be many other datasets of this nature that are publicly available, and the few that are have more restrictive or dubious licensing that would not work for our use case (but could be suitable for others). For example, https://huggingface.co/datasets/AIML-TUDA/LlavaGuard / https://huggingface.co/datasets/AIML-TUDA/smid says "All reasonable efforts have been made to ensure that the images in the SMID are in the public domain or licensed under Creative Commons or similar open licensing schemes", but this makes no guarantees about whether any of those are licensed under CC-NC (which would prohibit commercial use).

That's a long-winded way to say "we don't need the ability to pull arbitrary datasets right now, but other folks might find it helpful in the future" 🙂 

**Additional context**
I work with @sudosammy-atlassian , on behalf of our team I wanted to say thank you very much for your receptiveness and responsiveness to the previous feature request (https://github.com/promptfoo/promptfoo/issues/5231) 🙏
Repository: promptfoo/promptfoo
Issue: Nunjucks Custom Filters not working
Root cause: Documentation Deficiency

Example template seem to be not working anymore: https://github.com/promptfoo/promptfoo/tree/main/examples/nunjucks-custom-filters

When viewing YAML via the `promptfoo view`, the `nunjucksFilters` is missing.

Thanks.

Hi @christiankyle-ching,

Can you elaborate on "not working"?  It does look like the `allcaps` filter is working as intended in the example:

![llm eval with nunjucks filter](https://github.com/promptfoo/promptfoo/assets/310310/63a1bbb4-5e8b-4a27-a3d0-7209500917f8)

I agree that it should appear in the "View YAML" modal, though.
Repository: promptfoo/promptfoo
Issue: Cache behavior unpredictable in latest release
Root cause: Algorithmic Error

**Describe the bug**

I updated promptfoo to the latest release  `0.118.11` and the cache behavior seems unpredictable now. On re-runs, I would have expected promptfoo to retry error or failed cases, as it did in previous versions. Instead, it re-runs the entire suite. At times, it seems to re-run only a few tests, but not sure which ones. It definitely behaves completely different from before the update.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run a test suite
2. Re-run the test suite

**Expected behavior**
For step (2) the cache will kick-in and only the failed or error tests are re-run.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

```
===================================================================================================================================
Promptfoo Debug Information
===================================================================================================================================
{
  "version": "0.118.11",
  "platform": {
    "os": "linux",
    "release": "6.16.7-200.fc42.x86_64",
    "arch": "x64",
    "nodeVersion": "v24.6.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
===================================================================================================================================
Please include this output when reporting issues on GitHub: https://github.com/promptfoo/promptfoo/issues
```

- Promptfoo version: `0.118.11`

**Additional context**
The cache worked as [documented](https://www.promptfoo.dev/docs/configuration/caching/) in `0.118.10`

Thanks for reporting this @rvernica! I think this was caused by #5770 introduced a bug in resultIsForTestCase() that breaks the default cache/resume behavior. I am working on a fix.
Repository: promptfoo/promptfoo
Issue: Save evals in self hosted machine when running it from another machine
Root cause: Unimplemented Feature Gap

[Continuation of past issue as I do not seem to be able to reopen it: promptfoo/promptfoo/issues/578]

My team and myself want to be able to run the eval in their own machine and push the results to a machine which serves the promptfoo server.

Also for security reasons we cannot directly ssh to the other machine to manually copy the results and we can only reach predefined ports.

I understand what you're saying.  Right now `promptfoo share` is the best way to do this.  Are you saying that you'd prefer something that automatically shares/pushes to the remote machine?

Actually I thought share was not working (it was a config error on the server). It is ok then thanks
Repository: promptfoo/promptfoo
Issue: VertexAI 0.49 release causing issues for VertexAI provider
Root cause: Configuration Error

Hello,

I am using `vertex:gemini-pro` as a provider with the following rubric prompt:
```

# The LLM output will be graded by gemini-pro
defaultTest:
  options:
    provider: vertex:gemini-pro
    config:
      temperature: 0
      maxOutputTokens: 1024
    rubricPrompt:
      - role: system
        content: >-
          You are evaluating the answer...
```

However I've installed the latest promptfoo version (0.49) and I am getting totally new results from the grader, most of them being `llm-rubric produced malformed response`. My two cents is that the config for the llm is no longer working but is taking the default parameters from Vertex which is for example `temperature = 0.9` (see https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini)


0.49.0 included a different way of obtaining the Vertex API auth key, but it didn't change the actual request.  Two questions:
-  If you set `LOG_LEVEL=debug`, for example `LOG_LEVEL=debug promptfoo eval`, what is in the API response?
- Does switching to 0.48.0 (`npx promptfoo@0.48.0 eval`) solve the problem and if so, how does the response differ?

Passthrough of generationConfig options should be fixed in 0.49.2
Repository: promptfoo/promptfoo
Issue: Option for `contains: False` in assertions
Root cause: Unimplemented Feature Gap

Sometimes LLMs return a response in html/markdown format when not asked for it specifically. It will be helpful to ascertain that any markup formatting characters are not present in the response.

Perhaps you could do a [regex check](https://promptfoo.dev/docs/configuration/expected-outputs/deterministic/#regex) for lines starting with `#` or links of the form `[...](...)`?

Brilliant idea. I myself ended up using `llm-rubric` to check for absence of any formatting. 

Excellent!  Let me know if you run into any other cases where it would be useful to add a new assertion type.