zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: parsing wrong path when test by list of path
Root cause: Validation Gap

I was running npx promptfoo@latest eval --no-cache 
I got error when I test by using list of path.  When I did
```
tests: 
  - test/test1.csv
```
or
```
tests: ['test/test1.csv']
```  
I got 
<img width="1261" alt="Screenshot 2024-03-01 at 9 25 30 PM" src="https://github.com/promptfoo/promptfoo/assets/9929375/fa6668f3-6bac-44d3-9b54-3a4bdaf2389f">

if I use
```
tests: test/test1.csv
```
then it worked perfectly.

here is my all config
```
description: 'My first eval'

prompts:
  prompt/one_prompt_test.txt: onePromptTest
  prompt/some_prompt_test.txt: somePromptTest
  prompt/four_prompt_test.txt: fourPromptTest

providers:
  - id: 'python:provider/my_script.py'
    config:
      additionalOption: 123
    prompts: fourPromptTest
  - id: 'python:provider/new_script.py'
    config:
      additionalOption: 123
    prompts: [onePromptTest, somePromptTest]

tests: ['test/test1.csv']
```

I am running it on macOS 
and node.js v21.6.2
python:3.11

#520 should resolve this issue.  It will be in the next release

The problem was that `csv` files were not supported in the `tests` array.

Fix released in 0.44.0
Repository: promptfoo/promptfoo
Issue: How to test multi-turn conversations with tool calls?
Root cause: Documentation Deficiency

I need my model to *DO* something with data returned by a tool call, but I can't seem to get promptfoo to initiate another round-trip to the model. Below is a simple example that asks the model to (1) add two numbers, then (2) write a poem about the resulting sum.

I expect to get a poem, but the final output is simply the result of the tool call: `"11"`.

How do I return the tool result to the model for another round of reasoning?

promptfooconfig.js:
```
module.exports = /** @type {import('promptfoo').TestSuiteConfig} */ ({
  prompts: [
    'Add the following numbers together: {{a}} and {{b}}. Then, write a poem about the resulting sum.',
  ],
  providers: [
    {
      id: 'openai:gpt-4o',
      config: {
        model: 'gpt-4o',
        temperature: 0,
        tools: [
          {
            type: 'function',
            name: 'addNumbers',
            function: {
              description: 'Add two numbers together',
              parameters: {
                type: 'object',
                properties: {
                  a: { type: 'number' },
                  b: { type: 'number' },
                },
                required: ['a', 'b'],
              },
            },
          },
        ],
        tool_choice: 'auto',
        functionToolCallbacks: {
          addNumbers: (parametersJsonString) => {
            const { a, b } = JSON.parse(parametersJsonString);
            return JSON.stringify(a + b);
          },
        },
      },
    },
  ],
  tests: [
    {
      vars: { a: 5, b: 6 },
    },
  ],
});
```

Hey @collinhundley 

You can achieve this with a custom provider that implements the conversation loop. We actually have a great example of this pattern in the [`examples/redteam-medical-agent`](https://github.com/promptfoo/promptfoo/tree/main/examples/redteam-medical-agent) - check out the `src/llm.js` file which shows exactly how to handle multi-turn tool conversations.

Then reference your custom provider in your config:
```yaml
providers:
  - id: http
    config:
      url: http://localhost:3090/api/chat
      method: POST
      headers:
        'Content-Type': 'application/json'
      body:
        'message': '{{prompt}}'
        'sessionId': '{{sessionId}}'
      transformResponse: 'json.message'
```

This pattern gives you full control over the conversation flow.

See also https://github.com/promptfoo/promptfoo/issues/3462

@collinhundley - this is a duplicate of #3462 which has more detailed discussion on the same topic.

As @AISimplyExplained mentioned, the current approach is to use a custom provider that implements the full conversation loop. The `examples/redteam-medical-agent` example shows this pattern.

Closing as duplicate - see #3462 for the full discussion on workarounds and any future updates on native support for multi-turn tool call testing.
Repository: promptfoo/promptfoo
Issue: Command line usage with -r --providers
Root cause: Documentation Deficiency

The documentation offers two types for using the `-r --providers` flag.
Firstly, by name, this works as expected, however it's unclear how to pass options via a provider's `config` entry.
Second, by path. The source suggests that a .yaml file is expected, as such I prepared the following command.

```
npx promptfoo@latest eval -c experiment.yaml -r file://provider.yaml
```

Where `provider.yaml` is of the form

```
providers:
  - model-name:
      id: unique-identifier
      config:
        temperature: 0.0
        # other options
```

Promptfoo expects an ID for the provider file. Adding one as follows, leads to promptfoo searching for a module with the name `provider-id`.

```
id: provider-id
providers:
  ...
```

Thus, how does one use the path option, correctly?
Lastly, I'd like to suggest for multiple paths and names to be a valid entry to the commandline. For my purpose, I'd like to specify a range of provider.yaml files with configurations set, and the ability to select one or multiple by command-line.

Congrats on 2k stars!


Hi @Wehzie,

Thanks for the question about the `-r --providers` flag!

After reviewing this issue, I can clarify that the `-r` flag is designed for loading custom provider **implementations** (JavaScript or Python files), not provider **configuration** files.

**What works:**
```bash
# Use provider names
promptfoo eval -r openai:gpt-4 anthropic:claude-3

# Use custom provider implementations
promptfoo eval -r file://my_custom_provider.py
```

**What doesn't work:**
```bash
# Loading provider configs from YAML - NOT SUPPORTED
promptfoo eval -r file://provider.yaml
```

**How to organize provider configurations:**

Define all your providers in your main `promptfooconfig.yaml`:

```yaml
providers:
  - id: openai:gpt-4
    config:
      temperature: 0.7
  - id: anthropic:claude-3
    config:
      max_tokens: 1000
```

PR #652 (which referenced this issue) fixed a Windows-specific path issue but didn't implement loading provider configs from separate files.

If you'd like the ability to load provider configurations from external YAML files via command line, please open a new feature request describing your use case.

Since this issue is from March 2024 with no follow-up activity, I'm closing it now. Feel free to comment or re-open if you need clarification.

Thanks!
Repository: promptfoo/promptfoo
Issue: Disable thinking for ollama provider
Root cause: Unimplemented Feature Gap

Some Ollama models support thinking output (DeepSeek R1, Qwen 3). 

Ollama has the ability to enable or disable thinking. In the CLI, thinking is disabled using /set nothink followed by the prompt.

I'd like to be able to pass this configuration for ollama provider in promptfooconfig.yaml

When using `ollama:qwen3:4b`, the following disabled the "thinking".

```
defaultTest:
  vars:
    system_message: file://prompts/system_message.txt
```

`prompts/system_message.txt`
```
/no_think

Answer very concisely.
```

I can't figure out how to know apply `/no_think` to specific providers though:

Eg. For `qwen`, but not `gemma`

```
providers:
  - id: ollama:qwen3:0.6b
  - id: ollama:gemma3:1b
```

Thanks for you reply. I'd expect to have some configuration parameter which is passed to ollama and not a system prompt. This blog post describes how it can be passed when using the platform https://ollama.com/blog/thinking

Thanks for flagging!  #5341 should address this
Repository: promptfoo/promptfoo
Issue: type: llm-rubric with threshhold

greetings

with the following csv

```
source,expected,targetLanguage
```

I want the output translation by the provider to be compared to the expected sentence. Is my below approach well designed?

```
tests: 
  - file://score.csv   
  - description: Translation with threshold
    assert:
      - type: llm-rubric
        value: |
          Rate how similar the following output is to the reference.
          Reference: "{{ expected }}"
          Score from 0 (not similar) to 1 (identical).
        threshold: 0.2   
```

Thanks!

Resolved it thanks!

how did you solve it?

sorry @donatoaz I went with this approach, not sure how solid it is but a visual inspection seems to indicate that the scoring is spot on.

I added the following to my prompt.json

```
  {
    "role": "user",
    "content": "Translate {{sourceSentence}} into {{targetLanguage}}. Compare the model's translation against the target reference: {{targetSentence}}. Score from 1 (poor) to 10 (perfect), considering accuracy, fluency, and meaning."
  }
```


 



this was what I wanted to do but did not do what I needed it to do, show me scores

```
prompts:
  - file: translate.prompt

providers:
  - openai:gpt-4o-mini

datasets:
  - file: translations.csv

tests:
  - vars:
      source: "{{source}}"
      targetSentence: "{{targetSentence}}"
      lang: "{{lang}}"
    assert:
      - type: similarity
        value: "{{targetSentence}}"
        threshold: 0.75
        weight: 1
      - type: llm-rubric
        rubric: |
          You are a translation evaluator.
          Compare the model output with the reference "{{targetSentence}}".
          Score from 1 (poor) to 10 (perfect).

```
Repository: promptfoo/promptfoo
Issue: Chat Models
Root cause: Documentation Deficiency

Hi, how would you evaluate chat models with this tool? I see gpt-3.5-turbo being used in completion mode, but say I want to evaluate:

```
{"role": "system", "content": "You are a chatbot"}
{"role": "assistant", "content": "Hello, I am a chatbot! How can I help you"}
{"role": "human", "content": "What is the capital of France?"}
{"role": "assistant", "content": "<ANSWER>"}
```

and assert ANSWER is Paris?

NVM, found it in the docs https://promptfoo.dev/docs/configuration/parameters
Repository: promptfoo/promptfoo
Issue: Update OpenAI client code to support GPT-5
Root cause: Validation Gap

**Describe the bug**
For GPT-5 models routed through `openai/chat.js`, the `max_tokens` parameter is computed but never included in the request body. Additionally, GPT-5 should be handled by the Responses API with `max_output_tokens`, not the Chat Completions API.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Configure a provider for GPT-5 in Promptfoo:
   ```yaml
   providers:
     - id: openai:gpt-5
       config:
         max_output_tokens: 50
   ```
2. Run `promptfoo eval -v` (or any command that triggers a call)
3. Observe the debug log of the request body: it lacks `max_tokens` parameter
4. See that token limits are not enforced as expected

**Expected behavior**
- GPT-5 requests should be sent to the OpenAI Responses API (`/v1/responses`) using `max_output_tokens`
- If routed through Chat Completions API, `max_tokens` should not be included in the request body

**Screenshots**
N/A

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: `0.117.4`

**Additional context**
- Root issues:
  - `openai/chat.js` never adds `max_output_tokens` to the request body (lines 168-174)
  - Temperature is correctly not supported for GPT-5 (reasoning models) as per `supportsTemperature()` method

- Code evidence:
  ```168:174:dist/src/providers/openai/chat.js
  const maxTokens = isReasoningModel
    ? undefined
    : config.max_tokens ?? getEnvInt('OPENAI_MAX_TOKENS', 1024);
  ```

  ```186:196:dist/src/providers/openai/chat.js
  const body = {
    model: this.modelName,
    messages,
    seed: config.seed,
            ...(maxTokens ? { max_tokens: maxTokens } : {}),
            ...(maxCompletionTokens ? { max_completion_tokens: maxCompletionTokens } : {}),
            ...(reasoningEffort ? { reasoning_effort: reasoningEffort } : {}),
            ...(temperature ? { temperature } : {}),
  };
  ```

- Suggested fixes:
  - Route GPT-5 to the Responses provider and use `max_output_tokens` (or map `max_tokens` → `max_output_tokens` for compatibility)
  - Remove temperature for GPT-5

- Impact: API errors when using GPT-5

- Workaround: Monkey patch library

- **Changes made**: Removed incorrect temperature-related issues since GPT-5 correctly doesn't support temperature as per the `supportsTemperature()` method in the responses provider.

Hi @ICarpenter,

- To use the Responses API, please use the `openai:responses` provider i.e. `openai:responses:<model name>`.
- From the [Chat Completion docs](https://platform.openai.com/docs/api-reference/chat/create#chat_create-max_tokens), "[`max_tokens`] is now deprecated in favor of `max_completion_tokens`, and is not compatible with [o-series models](https://platform.openai.com/docs/guides/reasoning)." I believe that "o-series-models" is stated in error since it links out to the reasoning documentation.  Therefore, since GPT-5 is a reasoning model: use `max_completion_tokens` instead of `max_tokens`.
Repository: promptfoo/promptfoo
Issue: HTTP 500 error upon clicking "Run Evaluation" on the UI
Root cause: Configuration Error

Hi all,

when deploying promptfoo as docker container **without** Supabase, I get a HTTP 500 upon clicking "Run Evaluation" on the web UI.

![image](https://github.com/promptfoo/promptfoo/assets/158186135/423a8b6b-df84-4c10-87d2-ef86b20a7476)

I assume it's related to the Dockerfile and the configuration related to Supabase, as promptfoo tries to access the placeholder content which can be seen in the docker container log.

![image](https://github.com/promptfoo/promptfoo/assets/158186135/01fe2bf2-dc14-485f-99be-2cb91f6d3f3c)

How should the Dockerfile correctly look like when Supabase should not be used?

Thanks in advance!

Support for running evals in self-hosted webui is added in #540.  This will be in the next published release!

Released in 0.46.0

Thanks a lot @typpo ! 👍
Repository: promptfoo/promptfoo
Issue: Azure OpenAI provider uses outdated data sources API
Root cause: External Dependency Breakage

**Describe the bug**
Starting from API version 2024-02-15-preview, Azure OpenAI introduced breaking changes that affect applications using Azure AI Search as a data source. These changes include:

The API path has changed from /extensions/chat/completions to /chat/completions, even when data sources are used.
The request body parameter dataSources has been renamed to data_sources.
The data source type AzureCognitiveSearch has been renamed to azure_search.
As a result, any Promptfoo application built in the past year that integrates with Azure AI Search will no longer function correctly with newer API versions.

**To Reproduce**

1. Create a modern Azure AI Search index.
2. Connect Promptfoo to the index as a data source.
3. Run an evaluation.
4. Observe the error.

**Expected behavior**

Promptfoo should support data sources as documented by Microsoft:
- [Azure OpenAI API reference ](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/references/on-your-data?tabs=rest)
- [Azure OpenAI Data source - Azure AI Search Reference](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/references/azure-search?tabs=rest)

The following Promptfoo configuration should work:

```yaml
providers:
  - id: azure:chat:gpt-4o
    config:
      apiHost: 'https://{{azue openai name}}.openai.azure.com'
      dataSources: # or data_sources, in line with the new Azure Open AI API spec
        - type: "azure_search"
          parameters:
            endpoint: "https://{azure ai search name}.search.windows.net"
            indexName: '{{index name}}'
            authentication:
              type: "api_key"
              key: "{{key}}"
```

**System information:**

- Promptfoo version: 0.117.5

**Additional context**

suggested solution:


- In [Azure OpenAI Chat provider (lines 170–173)](https://github.com/promptfoo/promptfoo/blob/5ce7197833437c26cc08cded360593e189785dc2/src/providers/azure/chat.ts#L171-L173), the code currently checks if data sources are used and falls back to the old URL. This is incorrect. The correct behavior is to use the default URL (/chat/completions) for all API versions from 2024-02-15-preview onward, regardless of data source usage.



- Additionally, if the API version is 2024-02-15-preview or later, the request body should use data_sources instead of dataSources [on line 138](https://github.com/promptfoo/promptfoo/blob/5ce7197833437c26cc08cded360593e189785dc2/src/providers/azure/chat.ts#L138).
Repository: promptfoo/promptfoo
Issue: Claude3 Support
Root cause: External Dependency Breakage

Claude3 is released, seems like they changed the API:
![image](https://github.com/promptfoo/promptfoo/assets/1726527/a08e8a97-11fb-4ad1-8286-72c6578988d2)


Wondering if @streichsbaer, who wrote our mistral provider, has any interest in this one :)

I'd love to make this change but am in the hospital with a family member right now.  Otherwise, will probably get to it in the coming weeks.

@typpo hope all is well soon!

Funny you should say that. I just came here to offer my help with this and saw the message :)

On it!

And sorry to hear about you being at the hospital with a family member, hope all is well. 

Alrighty, have just opened a PR that adds support for Claude 3.
@typpo, let me know if there is anything else I can help with 👍 .

@zhlmmc, you can test the changes locally by following these steps:
1. Clone the repo
2. Check out the PR branch
3. Add the new provider to your `promptfooconfig.yaml`
```
provider:
- id: anthropic:messages:claude-3-opus-20240229
    config:
      temperature: 0.0
```
4. You can use the OpenAI prompt template, e.g. `prompts: [prompt.json]`. System prompts are optional.
```
[
  {
    "role": "system",
    "content": "{{ system_message }}"
  },
  {
    "role": "user",
    "content": "{{ question }}"
  }
]
```
5. Then just run `npx /path/to/promptfoo eval`

Let me know if there are any issues!

Thanks @streichsbaer!  This is released in 0.45.0
Repository: promptfoo/promptfoo
Issue: UnsafeBench plugin has an overly-restrictive license

The authors of UnsafeBench: https://huggingface.co/datasets/yiting/UnsafeBench state:

> The dataset is for **non-commercial academic research only**. 

Unfortunately, this prevents us from using it in our organization. Please consider adding support for another dataset with a more permissive licenses :)

I found this one as an example: https://huggingface.co/datasets/kirito011024/vlguard_unsafes

Thanks!
Repository: promptfoo/promptfoo
Issue: How to generate attack sample data using redteam.plugins in an offline environment
Root cause: Configuration Error

There is an exception when using redteam.plugins to generate attack sample data in an offline environment, as follows:
An unexpected error occurred during red team run: Unable to proceed with test generation: Network error: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
URL:  https://api.promptfoo.app/health
Please check your API configuration or try again later.
Error: Unable to proceed with test generation: Network error: fetch failed (Cause: Error: getaddrinfo ENOTFOUND api.promptfoo.app)
URL:  https://api.promptfoo.app/health
Please check your API configuration or try again later.
at synthesize (D:\Program Files\nodejs\node_global\node_modules\promptfoo\dist\src\redteam\index.js:547:23)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
The following environment variables have already been configured:
PROMPTFOO_DISABLE_TELEMETRY=1
PROMPTFOO_DISABLE_UPDATE=1
PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true
PROMPTFOO_DISABLE_SHARING=1
PROMPTFOO_SELF_HOSTED=1
The redteam configuration is as follows:
redteam:
provider:
id: ollama:qwen3:latest
config:
model: ollama:qwen3:latest
baseUrl:  http://xxx.xxx.xx.xx:11434  
temperature: 0.9
max_tokens: 1024
timeout: 30000
timeout: 300000  
threshold: 0.75   
failFastThreshold: 0.3 
  
numTests: 1
plugins:
- harmful:chemical-biological-weapons  

Hi @nanhua96712-glitch,

Thanks for asking about offline environment support.

You can now disable remote generation for red team plugins:

```bash
export PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true
promptfoo redteam run
```

This gives you a subset of offline capabilities by generating attack samples locally without calling external APIs.

If you need completely offline/on-prem deployment for your environment, please reach out about our enterprise offering.

Please pull the latest version and give it a try. See the [Red Team Configuration documentation](https://www.promptfoo.dev/docs/red-team/configuration) for more details.

If you have questions, feel free to comment or re-open.

Thanks!
Repository: promptfoo/promptfoo
Issue: Huggingface providers does not need apiKey to get the response
Root cause: Documentation Deficiency

While using the node package, i'm able to run tests without setting the apiKey with some Hugginface providers:

- tinyllama
- openchat
- mistral
- bloom

I'm running the service in k8s and i'm sure that the `HF_API_TOKEN` env var is not set.
The same behaviour while running tests using the cli.

I suppose this is not an issue of the package itself. If so, please forgive me about this issue and just close it.

This is expected. Huggingface allows unauthenticated use of some of its
APIs. For reliability, I recommend always using an API key.

On Tue, Mar 5, 2024, 12:31 AM Lorenzo ***@***.***> wrote:

> While using the node package, i'm able to run tests without setting the
> apiKey with some Hugginface providers:
>
>    - tinyllama
>    - openchat
>    - mistral
>    - bloom
>
> I'm running the service in k8s and i'm sure that the HF_API_TOKEN env var
> is not set.
> The same behaviour while running tests using the cli.
>
> I suppose this is not an issue of the package itself. If so, please
> forgive me about this issue.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/525>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJRSLTTUYTMS7HBHPVLYWV7FJAVCNFSM6AAAAABEGVK32GVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE3DQNRQHAZTCNI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: Creating an eval in the self-hosted docker image returns 500 error
Root cause: Validation Gap

**Describe the bug**
Trying to create a new eval using a self-hosted docker instance returns a 500 error from server

**To Reproduce**
Set up docker instance using the docker-compose from the [instructions](https://www.promptfoo.dev/docs/usage/self-hosting/#method-2-using-docker-compose) 
1. Clicked Create Eval
2. Filled in the fields
3. Then clicked "run eval"
4. Get 500 error in browser (Windows Chrome, also replicated in Firefox)

**Expected behavior**
The eval should run correctly

**Screenshots**
If applicable, add screenshots to help explain your problem.

<img width="1863" height="990" alt="Image" src="https://github.com/user-attachments/assets/b7eb2a7a-d2f8-4ccd-b947-2694b92b5241" />

**System information:**
- Promptfoo version: latest docker image from ghcr
- Error log from Promptfoo 
```
TypeError: Cannot destructure property 'evaluateOptions' of 'req.body' as it is undefined.
at /app/src/server/routes/eval.ts:31:11
```

Snippet of JSON posted to server
```
  "evaluateOptions": {
    "maxConcurrency": 4,
    "repeat": 1
  },
```



Hi!

Could you share the request body in dev tools when you click "run eval"? the exact request should be to `/api/eval/job`. 

Apologies - our Infrastructure team were blocking requests in the web application firewall.
All good now