zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Posthog errors even after disabling telemtry
Root cause: Validation Gap

**Describe the bug**
Due to our firewall settings telemtry calls to posthog cause errors and early exits.
After disabling telemetry, there are errors shown during running.

**To Reproduce**
I'm unable to give exact steps since I don't know what our firewall setup is like but for us
1. DIsable telemtery with `$env:PROMPTFOO_DISABLE_TELEMETRY=1`
2. Run promptfoo eval

**Expected behavior**
A clean run with not logging of errors

**Screenshots**
Due to security policy unable to provide screenshots but the error reads

`Error while flushing PostHog PostHogFetchNetworkError: Network error while fetching PostHog`
the error seems to be coming from `node-modules\posthog-core\src\index.ts:1130:17` which is fun since I cant even see that module
The "cause" is `DOMException [TimeoutError]: The operation was aborted due to timeout`


- Promptfoo version:
promptfoo version: 0.116.2

**Additional context**
The error I see here is different to the one without setting the telemtry promptfoo env variable, the other one crashes the run.


This was resolved in the latest release. Thanks for flagging @QuarkNerd!
Repository: promptfoo/promptfoo
Issue: There's a duplicate entry in the config-schema.json
Root cause: Validation Gap

**Describe the bug**
A clear and concise description of what the bug is.
In the config-schema.json, the bias plugin is duplicated, which leads to a big red error in the editor when referencing the schema

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

https://github.com/promptfoo/promptfoo/blob/e2b7d127d76c1433b62e0e28362cd6b86cea0611/site/static/config-schema.json#L1096
lines 1096 and 1097 are the same.

If you include the following line in your promptfooconfig.yaml, it'll display the error that "items 5 and 6 are identical):
```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
```

**Expected behavior**
A clear and concise description of what you expected to happen.
 No error should be displayed in the first line of the yaml config file.
**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version:

**Additional context**
Add any other context about the problem here.


This was resolved in the latest release! Thanks for flagging. Out of curiosity, how did you notice it? Did it affect any of your tooling? 

I just saw the error in VS Code, and was curious why all of my config yamls are marked as having issues, even though they were fine last week. So I just dug around in the config schema to see what's up there
Repository: promptfoo/promptfoo
Issue: Can I generate a new sessionId for each iteration of the Jailbreak strategy?
Root cause: Documentation Deficiency

Hello!

I'm currently configuring red teaming for our project, an I noticed that Jailbreak is described as a single-shot strategy, which (if I understand correctly) should send independent prompts to our LLM. However as I see, the sessionId is the same throughout the iterations, I'm generating the sessionIds as described in the docs:
```yaml
defaultTests:
  options:
    transformVars: '{ ...vars, sessionId: context.uuid}'
```

We're storing the conversations for each sessionId, which means that this strategy isn't single-shot for us, since the LLM knows the conversation history in that session.
Is there a way to force new sessionId generation for each iteration in the Jailbreak or other strategies?

Thanks!
Repository: promptfoo/promptfoo
Issue: Unable to run: I m trying to ran our custom model test , but not getting option to run the same.. every time it replaces with sample Yaml files .. please suggest step to choose run type for HTTP request
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


my request type 
providers:
  - id: https
    config:
      url: '...'
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        'ai_agent_id': '...', 
        'user_query': '{{prompt}}', 
        'configuration environment': 'DEV'
      transformResponse: 'json'

try to run cmd 
promptfoo  init
- Not sure 
- Not sure 
- ask to replace readme file and promptfoo.ymal  ( this is the issue , i want run my ymal for 1 prompt test to custom LLM model)
Pls suggest


Hi @dvis0777, thanks for jumping on a call with us to troubleshoot. For other people following along, we got the http provider working correctly for this endpoint and went over how to se up a config. Closing the issue for now.
Repository: promptfoo/promptfoo
Issue: basePath is undocumented as config option useful for Custom Python Providers
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
Searching the PromptFoo documentation website for "basePath" turns up "No results for "basePath""

But some usage patterns probably do need it.

**Describe the solution you'd like**
I'd like basePath to be made official and documented.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**

<img width="556" height="214" alt="Image" src="https://github.com/user-attachments/assets/dc3e01ce-e98d-4fda-8b4b-53461b5d6201" />


Thanks, this is now documented
Repository: promptfoo/promptfoo
Issue: Allow fetching langfuse prompts by label
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
It would be nice to iterate on the prompts without affecting production. To test the latest version of a prompt, one needs to fetch it by the `latest` label instead of a hardcoded version number.

**Describe the solution you'd like**
Langfuse already [supports](https://api.reference.langfuse.com/#tag/prompts/get/api/public/v2/prompts/{promptName}) fetching prompts by label. So this should be an easy add.
Repository: promptfoo/promptfoo
Issue: Image support in prompt for gemini model
Root cause: Documentation Deficiency


I'm trying to test whether Gemini can reliably detect certain objects in images I provide. However, I noticed that it doesn't seem to process the images at all—it just returns what looks like a random list of objects, even when they aren't present.

I also couldn't find much specific documentation about Gemini's vision/image input format or capabilities (unlike OpenAI, which has more detailed guides). Has anyone here successfully used Gemini to analyze images and detect specific objects or even description of the image? !

Hi @mahtaz! Thanks for reporting this issue. Good news - image support for Gemini models is fully implemented and working in promptfoo! This feature was added in PR #4767 and supports all Gemini vision-capable models.

You can find detailed documentation here:
- [Google Provider Docs - Multimodal Inputs section](https://www.promptfoo.dev/docs/providers/google#configuration) 
- [Working example in the repo](https://github.com/promptfoo/promptfoo/tree/main/examples/google-aistudio-gemini)

If you're still experiencing issues, try update to the latest version. `npm install -g promptfoo@latest`. The feature definitely works - I just tested it myself with object detection and it correctly identified objects in test images. If you're still having trouble after trying these steps, feel free to throw some time on my calendar.  Happy to help debug further!

Hi Micheal
Thanks so much for the quick update and the detailed info — really
appreciate it:)

That’s great to hear! I’ll update to the latest version of promptfoo and
give it another shot with the Gemini model. I’ll also check out the
examples and docs you linked. If I still run into any hiccups, I’ll
definitely take you up on your offer to debug together.

Thanks again for your help!

Best,
Mahta

On Fri, Jul 25, 2025 at 7:02 AM Michael ***@***.***> wrote:

> Closed #4849 <https://github.com/promptfoo/promptfoo/issues/4849> as
> completed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/4849#event-18807045519>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AIJJX47C4QKAYGNDQRIYLWL3KGQNNAVCNFSM6AAAAACBGIVWUCVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJYHAYDOMBUGU2TCOI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: error with `promptfoo view`
Root cause: Validation Gap

I am trying to run `promptfoo view` but I am getting the following error
<img width="969" alt="image" src="https://github.com/promptfoo/promptfoo/assets/64668750/fcecd59b-1f50-4d6c-8b51-cf9143462e68">

versions I am using:
- Node.js `v21.6.1` (tried with `v20.11.0`)
- promptfoo `0.42.0`

Can you check if you have another instance of `promptfoo view` running in the background?

Otherwise, you'll have to find what's running on port 15500.  Depending on your operating system, usually this can be done with some variation of the `netstat` command.

ah good catch! I had promptfoo view opened in one of my million tabs, thank you!
Repository: promptfoo/promptfoo
Issue: Do you know of any packages or frameworks similar to promptfoo?

I was looking for something just like promptfoo. Do you know of any packages or frameworks similar to this? I would like to consider other comparisons.

I also would like to know.

Unfortunately not really.  I built this because there wasn't anything else out there that did what I needed it to do.  OpenAI does have an [Evals](https://github.com/openai/evals) framework you can take a look at.  Its focus is on testing OpenAI models with heavier test cases, and some of the more advanced test cases require Python implementation.

The main thing I need is this but in python with langchain compatibility. It might be worth cloning and converting.

As far as I know, `QAEvalChain` in `langchain` module might be useful to me. I'm still looking to see if there are other alternatives.

@typpo Thanks for the link reference.

For those of you working in Python, have a look at the [end-to-end LLM chain testing documentation](https://www.promptfoo.dev/docs/configuration/testing-llm-chains#end-to-end-testing-for-llm-chains).

Specifically, I've created [an example](https://github.com/promptfoo/promptfoo/tree/main/examples/langchain-python) that shows how to evaluate a Python LangChain implementation.

The example compares raw GPT-4 with LangChain's LLM-Math plugin by using the `exec` provider to run the LangChain script:
```yaml
# promptfooconfig.yaml
# ...
providers:
  - openai:chat:gpt-4-0613
  - exec:python langchain_example.py
# ...
```

The result is a side-by-side comparison of GPT-4 and LangChain doing math:

![langchain gpt-4 eval](https://www.promptfoo.dev/assets/images/langchain-eval-1c44c280e70607b6c1682aa03ee33074.png)

Hope this helps your use cases.  If not, interested in learning more.

Side note - `QAEvalChain` is similar in approach to the `llm-rubric` assertion type of promptfoo.  It can help evaluate whether a specific answer makes sense for a specific question.

It looks like it was released recently.
[hegelai/prompttools: Open-source tools for prompt testing and experimentation ](https://github.com/hegelai/prompttools/tree/main)


@typpo : First of all congratulations on the great work in building this library. It would be great if we can have some way to directly compare and contrast `promptfoo` with `prompttools` and `evals by OpenAi`. This will make life easier for consumers to pick & choose best among these based on the usecase.
Repository: promptfoo/promptfoo
Issue: HuggingFace api key token configuration
Root cause: Unimplemented Feature Gap

HuggingFace provider allows api key to be set via environment variable.

it would be good if it could be set even using provider configuration in the same way the OpenAi provider do.

Thanks for the suggestion.  #494 adds support for the `apiKey` config param on huggingface
Repository: promptfoo/promptfoo
Issue: fix promptfoo import
Root cause: Validation Gap

I have an eval that I ran on an AWC EC2, because my PC went to sleep and aborted the eval. The EC2 can't reach our self-hosted Promptfoo server, so my only option to move the eval there is to import the eval result to my PC and share it with the self-hosted server.

I've tried running the import command with the eval command's output
```
$ promptfoo import -v output.json
[main.js:86] Verbose mode enabled via --verbose flag
[import.js:58] Importing v3 eval
[eval.js:168] Inserting prompt eff...82
[eval.js:191] Inserting dataset 312....8c
[import.js:89] Failed to import eval: TypeError: Transaction function cannot return a promise
```
Didn't work..
So I thought I'll use the output of the `promptfoo export evalID > export.json` command
```
$ promptfoo import -v export.json
[main.js:86] Verbose mode enabled via --verbose flag
[import.js:68] Importing v2 eval
[import.js:89] Failed to import eval: SqliteError: NOT NULL constraint failed: evals.config
```
Didn't work either..
How can I use the import command properly? 
I'm using version 0.116.7
Thanks!
Repository: promptfoo/promptfoo
Issue: Cannot set environment variables in self-hosted share server
Root cause: Configuration Error

**Issue**

I cannot seem to get the environment variables to be recognized when I build and run a self-hosted share server. Specifically, I am trying to use the store type: filesystem. 

**Steps to Reproduce:**

1. Build server container image 
2. `docker run -d -p 3000:3000 -e PROMPTFOO_SHARE_STORE_TYPE=filesystem -v $(pwd)/share-store:/app/share-store promptfoo-ui:latest node server.js`
3. `PROMPTFOO_REMOTE_API_BASE_URL=http://localhost:3000/api PROMPTFOO_REMOTE_APP_BASE_URL=http://localhost:3000 npx promptfoo share -y`

**Expected Behavior:**

The share file is written to the volume mounted: `$(pwd)/share-store`

**Actual Behavior:**

I'm able to view the share link, but the application isn't using the env `PROMPTFOO_SHARE_STORE_TYPE=filesystem` and defaults to type memory instead.

I would really appreciate any assistance on this! 


Thanks for raising this issue.  Please give it a try with 0.43.1, which has the fix #498 

Thanks for the quick turnaround!  
Tested and confirm it works on my end too 👍🏽
Repository: promptfoo/promptfoo
Issue: Use {{metric}} variable in defaultTests.assert
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I'd like to use the `metric` property for better filtering of the eval results.  My current config looks similar to this one:
**promptfooconfig.yaml**:
```
providers:
  - file://my_provider.yaml
defaultTest:
  assert:
    - type: llm-rubric
      threshold: 0.1
      metric: '{{metric}}'
prompts:
  - '{{userPrompt}}'
tests:
  - file://my_tests.yaml
```
**my_tests.yaml** (contains at least a few hundred prompts):
```
- vars:
  userPrompt: "Some prompt"
  metric: "metric1"
- vars:
  userPrompt: "Some other prompt"
  metric: "other"
```

**Describe the solution you'd like**
I'd expect to see the metrics on the UI where I could filter for them, however what I see is just `{{metric}}: 50.00% (1.00/2.00)`.
What I'd like to see is: `metric1: 50.00% (1.00/2.00) | other: 50.00% (1.00/2.00)`

**Describe alternatives you've considered**
I don't want to expand the `assert` for each prompt in the `my_tests.yaml`, because I like how it's compact in the `defaultTests.assert`

**Additional context**
Promptfoo version: 0.116.7

![Image](https://github.com/user-attachments/assets/484fc2cd-f07c-4a51-9f67-3b320b71fb39)
Repository: promptfoo/promptfoo
Issue: Problem using is-valid-openai-tools-call assertion
Root cause: Validation Gap

When using the `is-valid-openai-tools-call` assertion, if gpt returns plain text, there is no response object. It therefore not possible to determine the llms actual output.

Currently working around by using `is-json` + custom javascript assertions.

If I understand your issue correctly, #541 will fix this problem by allowing other assertions when GPT returns plaintext instead of a tools call.  Fix will be in the next release.

Released in 0.46.0
Repository: promptfoo/promptfoo
Issue: Failed to evaluate ollama model
Root cause: Interface Contract Mismatch

When evaluate ollama model response failed with `{"error":"invalid options: basePath"}`
```
Calling Ollama API: {"model":"openchat:7b","messages":[{"role":"user","content":"Answer the question: What's your name?"}],"options":{"basePath":""}}
Calling Ollama API: {"model":"openchat:7b","messages":[{"role":"user","content":"Answer the question: Hello"}],"options":{"basePath":""}}
        Ollama generate API response: {"error":"invalid options: basePath"}
Eval #1 complete (1 of 2)
        Ollama generate API response: {"error":"invalid options: basePath"}
Eval #2 complete (2 of 2)
```

I have tried both 0.43.1 and installing from github, and also found this PR that tried to fix this issue: https://github.com/promptfoo/promptfoo/pull/480. But for some reason the problem is still there?

My config is like the following
```yaml
description: 'My first eval'

prompts:
  # - file://prompt1.json
  - "Answer the question: {{question}}"

providers:
  - ollama:chat:openchat:7b

tests:
  - vars:
      question: What's your name?
    assert:
      - type: equals
        value: Hello, How may I assist you today?
  
  - vars:
      question: Hello```

Can you share the output of `promptfoo --version`?

@typpo Yes sure.
```bash
$ promptfoo --version
(node:3222913) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
0.43.1
```

Ah I see the problem.  Thanks for flagging, will get a fix in

On Wed, Feb 28, 2024 at 9:13 PM Long Chen ***@***.***> wrote:

> @typpo <https://github.com/typpo> Yes sure.
>
> $ promptfoo --version
> (node:3222913) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
> (Use `node --trace-deprecation ...` to show where the warning was created)
> 0.43.1
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/501#issuecomment-1970414374>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJUXOBBSS4PAGLWDBVDYV24HBAVCNFSM6AAAAABD3WDYPSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZQGQYTIMZXGQ>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


#506 should fix, although I have not yet had the chance to test locally yet.  The fix in #480 affected the text completion code path, not the chat completion path.

@typpo Thank you so much! How can I test it with the latest version? I have tried to uninstall promptfoo and reinstall it from github with `npm install github:promptfoo/promptfoo` but the problem is still there.

Update: the latest version 0.44.0 worked.

Same here. I have 

```
$ promptfoo --version
0.96.0
$ ollama --version
ollama version is 0.3.9
```

Error I get:
```
API call error: Error: Request failed after 4 retries: fetch failed. Output:
undefined
```

@palladius can you run with `--verbose` and share the debug output?
Repository: promptfoo/promptfoo
Issue: Feature Request: How to evaluate Azure Open AI Assistant
Root cause: Unimplemented Feature Gap

Currently in config we can provide only Open AI assistant ID. 

It would be good if we can also provide Azure Open AI assistant ID and evaluate it,

Example - 

providers: 
- id: azureopenai:assistant:asst_fEhNN3MClMamLfKLkIaoIpgB
     config:
       apiHost: 'xxxxxxxx.openai.azure.com'

This does not work and gives below error -

>npx promptfoo@latest eval
C:\Users\Jagrati.Modi\AppData\Local\npm-cache\_npx\81bbc6515d992ace\node_modules\promptfoo\dist\src\providers.js:173
            throw new Error(`Unknown Azure OpenAI model type: ${modelType}. Use one of the following providers: azureopenai:chat:<model name>, azureopenai:completion:<model name>`);

Try with this

  - id: azureopenai:chat:csdep-gpt4
    config:
      apiHost: cs-oi-yyyyyy-testing.openai.azure.com
      apiKey: 984681cXXXXXXXc20d22YYdf2a7

> Try with this
> 
> * id: azureopenai:chat:csdep-gpt4
>   config:
>   apiHost: cs-oi-yyyyyy-testing.openai.azure.com
>   apiKey: 984681cXXXXXXXc20d22YYdf2a7

Isn't this only for a model deployment, not an assistant? 

#577 adds support for OpenAI assistants on Azure!

Published in [0.49.0](https://github.com/promptfoo/promptfoo/releases/tag/0.49.0)

https://promptfoo.dev/docs/providers/azure/#assistants
Repository: promptfoo/promptfoo
Issue: Test multiple stages of a conversation
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
We have a chatbot and want to test something with history, I'm aware we can just pass in a history but this makes it difficult to mantain when functinality changes.

**Describe the solution you'd like**
It would be good to be able to have a way to create a group of tests where we can assert on the output of a test and then use that output in another test.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


just found the documentation for this https://www.promptfoo.dev/docs/configuration/chat/
Repository: promptfoo/promptfoo
Issue: Failure to parse response from factuality checker results in mysterious `Invalid option: )` test failure
Root cause: Algorithmic Error

We get test failures like this:

    expected
    ```
    Foo is bar.
    ```
     to match factuality with "Foo is baz", but it did not.
    Reason:
    Invalid option: )

With [additional logging](https://github.com/promptfoo/promptfoo/pull/503), we see the full response from the model is:

```
E) The answers differ, but these differences don't matter from the perspective of factuality.
```

[promptfoo tries to extract the letter `A`...`E` here](https://github.com/jameshfisher/promptfoo/blob/7a1ef8afb1343ec7e84a849cfeed5b90baa5fac0/src/matchers.ts#L365) with

```
    const option = resp.output.trim().charAt(1).toUpperCase();
```

But this does not work with this response:

```
> const response = "E) The answers differ, but these differences don't matter from the perspective of factuality."
undefined
> response.trim().charAt(1).toUpperCase()
')'
```

I don't understand the `charAt(1)`. At a guess, I suppose the code is expecting the response to begin with an open paren, like `(E) The answers differ ...`.