zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: GitHub Action workflow stuck since version 0.118.7
Root cause: External Dependency Breakage

**Describe the bug**
Since version `0.118.7`, the GitHub Actions workflow proposed in the docs [here](https://www.promptfoo.dev/docs/integrations/ci-cd/#github-actions) no longer works - it's stuck on the evaluation step.

**To Reproduce**
Run the workflow linked above on versions:
* 0.118.7 (doesn't work)
* 0.118.6 (works correctly)

**Expected behavior**
Version `0.118.7` should continue to work correctly as previous versions when running the evaluations using the proposed workflow in the docs.

**Screenshots**
Running on version `0.118.7` (the evaluation step eventually timed out):
<img width="4064" height="1810" alt="Image" src="https://github.com/user-attachments/assets/3ae4c306-a2ef-44d8-81ef-908d1f5162d3" />

Running on version `0.118.6` (finished successfuly):
<img width="4054" height="1828" alt="Image" src="https://github.com/user-attachments/assets/45d773e5-2010-4808-9569-97b6547fff80" />

**Additional context**
The workflow file I run in the screenshots above is slightly different than what's in the docs. In particular, here's the evaluation step:

```yaml
- name: Run promptfoo evaluation
  env:
    AZURE_API_KEY: ${{ secrets.AZURE_API_KEY }}
  run: |
    npx promptfoo@0.118.6 eval \
      --config myconfig.yaml \
      --env-file myenv/.env \
      --no-cache \
      --no-table \
      --output report.html
```

but I think it shouldn't make a difference.

It seems that version `0.119.0` fixed the issue.

Thanks so much for reporting this and Im glad to hear version `0.119.0` fixed the issue.
Please don’t hesitate to reach out if you run into any other problems or notice anything unexpected.
Repository: promptfoo/promptfoo
Issue: bug: `init --example` shows success message and creates empty directory after failed download
Root cause: Validation Gap

**Describe the bug**
When `promptfoo init --example <name>` fails to download an example (e.g., non-existent example or network error), and the user declines to retry, the command:

1. Leaves an empty directory on the filesystem
2. Shows a success message with instructions to use the (empty) example

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run 'promptfoo init --example nonexistent-example'
2. Observe error `Failed to download example: Failed to download example: Failed to fetch directory contents: Not Found`
3. Enter `n` on 'Would you like to try downloading a different example?' prompt
4. See success message and empty `nonexistent-example` directory
```
View the README at nonexistent-example/README.md or run:

`cd nonexistent-example && promptfoo eval`

to get started!
```

**Expected behavior**

At a minimum:
- Silent exit (no success message)
- No empty directory left on filesystem

Alternatively:
- Show a helpful error message with next steps
- No empty directory left on filesystem
- Optional: Offer fallback to default `init` or context-aware suggestions based on error type (network failure vs 404)

Example of helpful message:
```bash
No project initialized. To get started, try:

  promptfoo init --example           (browse and select an example)
  promptfoo init                      (create a basic project)

View examples at: https://github.com/promptfoo/promptfoo/tree/main/examples
```

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: 0.118.17

**Additional context**

The primary codepaths related to this issue:

Problem 1: Directory is created optimistically before knowing if the download is a success: 

https://github.com/promptfoo/promptfoo/blob/a982ddf50ea80aae71dcbe91c90c943ea6d3ae71/src/commands/init.ts#L66-L75

Problem 2: If the user decides not to choose another example upon failure, we exit the `attemptDownload` block but the `exampleName` still exists which means we don’t return here: 

https://github.com/promptfoo/promptfoo/blob/a982ddf50ea80aae71dcbe91c90c943ea6d3ae71/src/commands/init.ts#L150-L152

**Possible Solutions**

There's a couple of different ways to go about this but some ideas include:

Empty Directory:
* Move directory creation until after the download succeeds

Erroneous Success Message:
* Track success separately from the `exampleName`
or
* Clear `exampleName` when the user aborts

I'm happy to implement any or all of these fixes. Let me know which direction you prefer!

Hi @LizzHale! This is a good find. We would love help on this. Please feel free to take them and reach out if you would like even more promptfoo swag!
Repository: promptfoo/promptfoo
Issue: Add support for picking up variables from .env file
Root cause: Unimplemented Feature Gap

It would be great to have support for picking up environment variables from the `.env` file (I tried it's not working).
Also, I sourced the env variables in my terminal session but they are also not getting picked. I'm not sure what am I doing wrong. 

I don't want to add env variables in the `yaml` file to avoid committing it to `git`

<img width="924" alt="image" src="https://github.com/promptfoo/promptfoo/assets/36589645/9c459761-b4b9-448f-9333-7f20cb4bbdcd">

___

This is my `yaml`

```yml
description: "My first eval"

prompts:
  - "Write a tweet about {{topic}}"
  - "Write a very concise, funny tweet about {{topic}}"

providers:
  - id: azureopenai:chat:gpt-35-16k
    config:
      apiHost: https://<endpoint>.openai.azure.com
      temperature: 0.4
      apiVersion: 2024-02-15-preview
  - id: openai:gpt-4
    config:
      temperature: 0.4

tests:
  - vars:
      topic: bananas

  - vars:
      topic: avocado toast
    assert:
      # For more information on assertions, see https://promptfoo.dev/docs/configuration/expected-outputs
      - type: icontains
        value: avocado
      - type: javascript
        value: 1 / (output.length + 1) # prefer shorter outputs

  - vars:
      topic: new york city
    assert:
      # For more information on model-graded evals, see https://promptfoo.dev/docs/configuration/expected-outputs/model-graded
      - type: llm-rubric
        value: ensure that the output is funny

```

I have seen mentions of `export OPENAI_API_KEY=your_api_key_here` in the docs, so it should work. 
I'm using Turbo repo and this is my path `/Documents/projects/cool-ai-project/packages/prompts`. If it helps investigating the issue. 

For now, I'll hardcode API keys in `yaml` but not commit anything.

This was actually just added in #602!  It'll be in the next published release (probably published within the next 12 hours)

This change is released in [0.50.0](https://github.com/promptfoo/promptfoo/releases/tag/0.50.0).  Thanks!

Hello, thanks for adding this.  Is it possible to pass a non-standard path to an `.env` file to the CLI?  While our npm scripts are run from the repo root, we prefer not to have any number of `.env` files cluttering it up, so they tend to go in package-specific sub-directories along with config files (such as `promptfooconfig.yaml`).

@alexclaydon Made a PR here that allows you to override the `.env` path: https://github.com/promptfoo/promptfoo/pull/615
Repository: promptfoo/promptfoo
Issue: Run External File from `tests.csv`
Root cause: Unimplemented Feature Gap

Thanks to #238 , `tests.csv` supports Python function calls for assertions.
However, `__expected` doesn't seem to provide the ability to run external files (not just Python). Assert can execute external files using the `file://` prefix to `value`, which would be nice to have for `__expected` as well.

Thanks for flagging this.  #608 fixes handling for external assertion files that are not Javascript or Python.

Tested with the following:

promptfooconfig.yaml:
```yaml
prompts:
  - "Write a tweet about {{topic}}"

providers:
  - openai:gpt-3.5-turbo-0613

tests: tests.csv
```
tests.csv:
```
topic,__expected1,__expected2,__expected3
bananas,javascript:file://assert.js,contains:file://contains_text.txt,contains:file://contains_json.json
```
assert.js:
```js
module.exports = (output, { vars }) => {
  console.log(`Received ${output} using variables ${JSON.stringify(vars)}`);
  return {
    pass: true,
    score: 0.5,
    reason: 'Some custom reason',
  };
};
```
contains_text.txt:
```
banana
```

Fix is released in [0.50.0](https://github.com/promptfoo/promptfoo/releases/tag/0.50.0). Thanks!

Wow, I didn't think it'd be fixed so quickly, thank you so much!!
Repository: promptfoo/promptfoo
Issue: modelaudit source code

Model audit is still available on PyPi https://pypi.org/project/modelaudit/ where it shows that it is released under a MIT license. However, when clicking on the links to view the source they 404? https://github.com/promptfoo/modelaudit Has the source been moved somewhere else, or is it no longer open source?

Hi @Zoobdude! We haven't released the source code yet but will do so in the near future (it's a brand new project). I'll update the license to reflect this. If you would like to chat about modelaudit please feel free to send me an email or put time on my calendar - https://cal.com/michael-dangelo/30min I would love to chat and understand your requirements / collect product feedback. 

Oh nice! I came across this when looking for an alternative to https://github.com/protectai/modelscan and it looked promising. I'll get something in your calendar.
Repository: promptfoo/promptfoo
Issue: [FAIL] Error: Unknown assertion type: contains-some
Root cause: Documentation Deficiency

The full error:

[FAIL] Error: Unknown assertion type: contains-some Error: Unknown assertion type: contains-some at runAssertion (/Users/me/.npm/_npx/1ddbeaccabfe00a4/node_modules/promptfoo/dist/src/assertions.js:256:11) at runAssertions (/Users/me/.npm/_npx/1ddbeaccabfe00a4/node_modules/promptfoo/dist/src/assertions.js:43:30) at Evaluator.runEval (/Users/me/.npm/_npx/1ddbeaccabfe00a4/node_modules/promptfoo/dist/src/evaluator.js:101:74) at async /Users/me/.npm/_npx/1ddbeaccabfe00a4/node_modules/promptfoo/dist/src/evaluator.js:277:25

I was just changing the initial test YAML


```
# This configuration runs each prompt through a series of example inputs and checks if they meet requirements.
#removed prompts.txt, 
prompts: [prompts-v1.txt]
providers: [openai:gpt-3.5-turbo-0613]
tests:
  - description: Test
    vars:
      var1: first variable's value
      var2: another value
      var3: some other value
    assert:
      - type: contains-some
        value:
          - "something "
          - "other "
          - "The other"
      - type: javascript
        value: 1 / (output.length + 1)  # prefer shorter outputs
```


... everything else is the same

I think this is an example of (human) hallucination on my part.  The correct type is `contains-any`, not `contains-some`.  Give that a try and see how it works.  

I'll get the docs/example fixed :sweat_smile:
Repository: promptfoo/promptfoo
Issue: model-graded evaluation with python script
Root cause: Validation Gap

We're using an internal proxy to forward requests to chatgpt. I want to use for grading the outputs. I've tried to configure it in the following way:
```
defaultTest:
  options:
    provider: 
      - id: python:scripts/compass.py
        config:
          model: gpt-4
```
with calling the internal api inside python.
However, I face this issue:
![image](https://github.com/promptfoo/promptfoo/assets/22130831/fec11cc6-4998-4d37-bd67-574c8985dab4)


It looks like you are creating an array when you don't mean to.  It should look like this:

```yaml
defaultTest:
  options:
    provider: 
      id: python:scripts/compass.py
      config:
        model: gpt-4
```

@typpo  thanks, i've tried the way you suggested, however got another error:
```
TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at Object.join (node:path:1175:7)
    at PythonProvider.callApi (/opt/homebrew/lib/node_modules/promptfoo/dist/src/providers/pythonCompletion.js:22:63)
    at matchesLlmRubric (/opt/homebrew/lib/node_modules/promptfoo/dist/src/matchers.js:227:38)
    at async runAssertion (/opt/homebrew/lib/node_modules/promptfoo/dist/src/assertions.js:663:17)
    at async /opt/homebrew/lib/node_modules/promptfoo/dist/src/assertions.js:82:24
```

When I use it in a providers list (not for grading) it works fine though

Hmm.. can you try this and let me know how it works?
```yaml
defaultTest:
  options:
    provider: 
      id: python:scripts/compass.py
      config:
        model: gpt-4
        basePath: ''
```

unfortunately, same error.

I see you've committed a fix, have you tested it?

Fix is released in [0.51.0](https://github.com/promptfoo/promptfoo/releases/tag/0.51.0)!
Repository: promptfoo/promptfoo
Issue: Document content error about Alibaba

In the file: `site/docs/providers/alibaba.md`, there is a section `Configuration`:

```
providers:
  - qwen-max
  - id: alibaba:qwen-plus # Aliases: alicloud:, aliyun:, dashscope:
    config:
      temperature: 0.7
      apiKeyEnvar: 01234567890123456789012345678901 # or set `DASHSCOPE_API_KEY` to your Alibaba Cloud API key
      API_BASE_URL: https://dashscope-intl.aliyuncs.com/compatible-mode/v1 # Optional: Override default API base URL
```
`API_BASE_URL` should be changed to `apiBaseUrl`.

And if it is possible, I recommend add a note that if using the Alibaba in Beijing, we should use `https://dashscope.aliyuncs.com/compatible-mode/v1`.

Thank you for you patience and time!



Also, in this section, if we run ```- qwen-max```, there will be an error, we should write this as ```- alibaba:qwen-max```
Repository: promptfoo/promptfoo
Issue: Using variables in initialMessages
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When using initialMessages it is not possible to use variable placeholders. Because of this I can't use it in defaultTest and need to set it manually for every test case. For "instructions" it is possible to use it as a template with variable placeholders. I would like to have the same functionality with initialMessages.

**Describe the solution you'd like**
Variable placeholders like {{payment_method}} in the content sections of initialMessages get filled with defined variables.
This should work in the following cases:
- at the vars level
- at the provider config level
- in defaultTest
- per unique test case
- if the initialMessages are defined in a separate file
- combinations of the above cases

Example (in defaultTest at the provider level):
provider:
  id: 'promptfoo:simulated-user'
    config:
      maxTurns: 3
      initialMessages:
        - role: user
          content: I've selected my flight and I'm ready to book
        - role: assistant
          content: Great! How would you like to pay?
        - role: user
          content: I want to pay via {{payment_method}}
        - role: assistant
          content: Okay, then please tell me your credentials for {{payment_method}}
-------------
This example could allow to use this as a template and fill it with the payment_method variable for every test case. This way it is possible to analyse the ongoing conversation flow for different payment methods without the need to override the complete initialMessages variable for every test case.

**Describe alternatives you've considered**
At the moment, I need to write down the same initialMessages for every test case and must change the unique parts by hand. This is redundant and more complex than defining an initialMessages "template" with variable placeholders in defaultTest.

**Additional context**
Thank you again for adding the initialMessages feature. This is a real gamechanger. I hope this feature request helps to make the simulated-user provider even better :D
Repository: promptfoo/promptfoo
Issue: Docker images not updated since v0.118.14
Root cause: External Dependency Breakage

**Describe the bug**

The Docker release workflow has been failing consistently. As a result, the Docker image at https://ghcr.io/promptfoo/promptfoo has not been updated since version 0.118.14 (released October 9, 2025).

The workflow fails during `npm ci` with a segmentation fault (SIGSEGV) in the `@swc/core` postinstall script on Alpine Linux.

**To Reproduce**

Check these links
- https://ghcr.io/promptfoo/promptfoo
- https://github.com/promptfoo/promptfoo/actions/workflows/docker.yml

**Expected behavior**

The Docker workflow should successfully build and publish images to ghcr.io/promptfoo/promptfoo for each release.

**Screenshots**

N/A

**System information:**

- Promptfoo version: 0.119.4

**Additional context**

Updating `@swc/core` to a newer version may resolve this issue.
- https://github.com/swc-project/swc/issues/11126 (Alpine musl SIGSEGV)
- https://github.com/swc-project/swc/issues/11177 (ARM64 segfault)
Repository: promptfoo/promptfoo
Issue: CommandLineOptions: maxConcurrency not working
Root cause: Configuration Error

**Describe the bug**
The commandLineOptions: maxConcurrency is ignored. When setting a value for maxConcurrency, the default concurrency of 4 is used and also displayed in the command line. Setting it in the command line directly using -j or --max-concurrency works as expected.

**To Reproduce**

1. Set commandLineOptions: maxConcurrency: 5
2. run promptfoo eval
3. Look at command line output
4. See "Running X test cases (up to 4 at a time)..."

**Expected behavior**
Setting commandLineOptions: maxConcurrency should successfully set the concurrency of API requests

- Promptfoo version:
0.119.3
Repository: promptfoo/promptfoo
Issue: Encountering API Call Error When Switching Provider from OpenAI to Azure OpenAI
Root cause: Documentation Deficiency

Hello everyone,

I'm in the process of transitioning the Promptfoo provider from OpenAI to Azure OpenAI, as outlined in the [official documentation](https://www.promptfoo.dev/docs/providers/azure). However, despite adhering closely to the specified steps, I've encountered an issue during the API calls to the Azure OpenAI service.

Error encountered:

`[FAIL] API call error: Error: Request failed after 4 retries: request to https://undefined/openai/deployments/gpt-35-turbo-16k/chat/completions?api-version=2023-12-01-preview failed, reason: getaddrinfo ENOTFOUND undefined`

Steps I've taken:

I meticulously followed the guide to change the API provider settings from OpenAI to Azure OpenAI, setting the environment variable in my terminal as `export AZURE_OPENAI_API_KEY={key}`, where {key} is the value obtained from KEY 2 under "Keys and Endpoint" in the "Resource Management" section of the Azure dashboard.
Executed the evaluation tests using `promptfoo eval --no-cache` from the tests directory. 

Using the latest version 0.50.0 and nvn 20.10.

The error message above was the result of these steps. It seems that the `provider.config.apiHost` is not read for some reason.

Test case setup:

```
prompts: [../../prompts/general_promptjson]
providers: [azureopenai:chat:gpt-35-turbo-16k]
defaultTest:
  options:
    provider:
      id: azureopenai:chat:gpt-35-turbo-16k
      config:
        apiHost: 'ai-qa-svc.openai.azure.com'
tests:
  - description: Conversation?
    vars:
      user_request: who are you?
    assert:
      - type: similar
      threshold: 0.8
      value: I'm an AI assistant.
```

Any insights or guidance on how to resolve this error would be greatly appreciated.

Can you try setting it up so that you provide the `apiHost` in the `providers` list:

```yaml
prompts: [../../prompts/general_promptjson]
providers: 
  - id: azureopenai:chat:gpt-35-turbo-16k
    config:
      apiHost: 'ai-qa-svc.openai.azure.com'

defaultTest:
  options:
    provider:
      # This provider is used for grading only. You'll have to switch this to an embeddings model if you want to use it for the `similar` assertion.
      id: azureopenai:chat:gpt-35-turbo-16k
      config:
        apiHost: 'ai-qa-svc.openai.azure.com'

tests:
  - description: Conversation?
    vars:
      user_request: who are you?
    assert:
      - type: similar
      threshold: 0.8
      value: I'm an AI assistant.
```


That did it, I was indeed missing the config in the `providers` list. 
Thanks @typpo! 🙌
Repository: promptfoo/promptfoo
Issue: Is there any way to add custom API provider to the list of API providers?
Root cause: Unimplemented Feature Gap

I'm using the promptfoo for a customized LLM. I would like to use the options on its webserver (Using the New Eval tab to run the evaluation). 
Is there any solution to add my customised provider to the list of available provider?

yeah, I think this is needed

Hey @shahriardn! 👋

We've shipped this feature! You can now customize the provider list in the web UI for self-hosted deployments.

Create a `ui-providers.yaml` file in your `.promptfoo` directory to configure which providers appear in the eval creator. When this config exists, the UI will show only your specified providers instead of the full default list.

This was implemented in #6124 and includes:
- Server-side provider list customization via `ui-providers.yaml`
- New `/api/providers` endpoint that serves your custom config
- Updated eval creator UI that respects the custom provider list

Check out our [self-hosting documentation](https://www.promptfoo.dev/docs/configuration/self-hosting/) for setup details including Docker and Kubernetes examples.

Thanks for opening this issue and helping shape the direction of promptfoo! And thanks to everyone else who showed interest 

Feel free to open a new issue if you run into any questions or have feedback on the implementation.
Repository: promptfoo/promptfoo
Issue: Web viewer: env-var controlled table settings
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
When using the web viewer to communicate test results, it would be nice to be able to control default table settings. In my case especially the ["prettify json"-option](https://github.com/promptfoo/promptfoo/blob/b2147486cc1f7ab8ca86f05869c00656b4d3e907/src/app/src/pages/eval/components/store.ts#L392) 

**Describe the solution you'd like**
The possibility to set environment variables like:
```sh
export PROMPTFOO_WEB_VIEWER_TABLE_SETTING_PRETTIIFY_JSON=1
export PROMPTFOO_WEB_VIEWER_TABLE_SETTING_SHOW_PASS_FAIL=0
```
when I spin up the promptfoo service, such that, when the variables are set, they will overwrite the hardcoded defaults for [`useResultsViewSettingsStore `](https://github.com/promptfoo/promptfoo/blob/b2147486cc1f7ab8ca86f05869c00656b4d3e907/src/app/src/pages/eval/components/store.ts#L380)

I guess the environment variables should be declared in [src/envars.ts](https://github.com/promptfoo/promptfoo/blob/main/src/envars.ts) and imported from that file.

**Describe alternatives you've considered**
The current workaround is just to instruct people to adjust the table settings to get a favorable overview of the results.

**Additional context**
I consider this a minor feature and hopefully an easy first issue. I will try to find the time to implement it myself, but would very much appreciate feedback on the suggestion. 


@DKAndreasen would be happy to accept a PR with this change and send you some swag as a thank you. Please feel free to reach out if you have questions or get stuck!
Repository: promptfoo/promptfoo
Issue: How to integrate n8n agent prompt's
GitHub labels: Feature Request

help-wanted

good-first-issue

Hi,

so I've been trying to integrate n8n prompts by first extracting them and then converting the variables to a promptfoo compatible format. However, I got stuck with an output format note on how the prompt would look like?

Any hints how to integrate promptfoo with n8n agents? Especially when output prompts or any other tool calls are involved?