zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: "Copy Contents" Button in Assertion details / metadata WebView Window
Root cause: Unimplemented Feature Gap

A clear and concise description of what you want to happen.

- When viewing detailed results in an individual output and test details (by clicking hour glass), it would be helpful to have a quick "copy field" button for each of the many fields, particularly in the metadata.
- This is desirable to pull out larger prompts, raw blocks of metadata, etc

Other thoughts
- potentially add the button only if output is large enough to reduce clutter?

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
- copying + pasting works ok, but not if the output is large enough and is therefore hidden. You need to expand the "value", then slowly drag to highlight the entire block before you can copy.

Better copy functionality within that dialog is coming in #3379 !

This is now released

Just tested, this is great!
Repository: promptfoo/promptfoo
Issue: Security threat. /bin/sh: 1: Syntax error due to improper character handling
Root cause: Validation Gap

When using exec provider with variable with special characters. eg:
```
prompts:
  - "{{violation}}"


providers: 
  - exec:python3 provider.py:

tests:
  - description: exploit
    vars:
      violation: "`('"
``` 

Whenever a variable contains special shell characters we might get an error like:
```
Error: Command failed: python3 provider.py "[{\"role\":\"user\",\"content\":\"`('\"}]" "{\"id\":\"exec:python3 provider.py\",\"config\":{\"basePath\":\"\"}}" "{\"vars\":{\"violation\":\"`('\"}}"
/bin/sh: 1: Syntax error: Unterminated quoted string
```

**Using shell when executing script providers is also a security threat! The provider should be executed directly (via exec call) rather that via system call  through shell.** Adding more escaping rules is not the right path as it can quickly get convoluted and either brake the passed values or leave a gap for another escaping trick.
For example if the variable is:
```
      violation: "`rm -rf /*`"
```

It will execute the `rm -rf /*` command rather than passing that to provider. This behavior is not only insecure but also inconsistent with other providers.

I believe in JS it's `spawn()` that does the proper `exec` syscall (and JS's `exec` is actually a `system` call):

https://stackabuse.com/executing-shell-commands-with-node-js/

Thanks for raising this.  #343 rewrites exec calls to not use a shell via `execFile` and `spawn`.  You're right that this solves a lot of headaches and potential side effects with escaping.

There is a backwards compatibility issue for anyone who relied on shell behaviors.  I'm considering retaining support with a `useShellUnsafe` or similar config param for the `exec` provider type.

Resolved in version 0.33.0
Repository: promptfoo/promptfoo
Issue: FR: Ollama Chat API
Root cause: Unimplemented Feature Gap

# Feature Request

## Ollama Chat API

Starting v0.1.14 Ollama implemented chat API (akin to OpenAI's):
https://github.com/jmorganca/ollama/releases/tag/v0.1.14

```
curl http://localhost:11434/api/chat -d '{
  "model": "mistral",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant that answers concisely." },
    { "role": "user", "content": "why is the sky blue?" }
  ]
}'
```

This allows for more  elaborate prompting techniques, like example answers, etc. without using raw prompt templates.
It'd be great if promptfoo implements that API provider. 

Thanks for pointing out this new API.  This implementation is in progress: PR #342 

Resolved in version 0.33.0
Repository: promptfoo/promptfoo
Issue: Question: Is it possible to take an external LLM response(output) and promptfoo just runs the metrics on the output?
Root cause: Documentation Deficiency

- So I am working on an application that employs LLM to generate a response. It takes in a sample.csv file and it generates a response for each of the examples. 

- Ideally what I would want is to use the application to generate these responses and export them as` response.csv`. And this response.csv will be taken up by promptfoo and does the metrics evaluation.
- Right now, promptfoo does both the tasks of generating the response as well as evaluating it. Is it possible to just use promptfoo for the evaluation metrics?

Hi @ananthanandanan, yes this is possible.  Two ways:
1. promptfoo can call an [external script](https://promptfoo.dev/docs/providers/custom-script/) that reads the CSV file and generates the response.  
2. If you're building in Javascript, you may prefer to implement a [custom API provider](https://promptfoo.dev/docs/providers/custom-api/) to call your application natively.

Thanks, @typpo . The first one definitely would work for my use-case. My app is a python, streamlit app, where the user can upload a CSV and get back a response and then is exported as response.csv. 

I was just going through the examples and found [this](https://github.com/promptfoo/promptfoo/blob/main/examples/langchain-python/langchain_example.py), so I will tweak my script to take the csv file as args and see how it goes.

I had one follow-up question, for the same application. Would it be possible to aggregate the scores of multiple runs? For eg
![image](https://github.com/promptfoo/promptfoo/assets/53400471/37104d99-f2fe-4221-b796-b3f05b14a619)


Not quite like that, but you can run multiple configs at the same time:

```
promptfoo eval -c config1.yaml config2.yaml config3.yaml
```

Which will aggregate the results all in one view.
Repository: promptfoo/promptfoo
Issue: TogetherAI provider do not support max_tokens config
Root cause: Configuration Error

**Describe the bug**
I'm using TogherAI provider but the response tokens are limited to 1024. I tried to config `max_tokens` but it has no effect and it is supported in their API https://docs.together.ai/reference/chat-completions-1:
```yaml
  - id: togetherai:chat:Qwen/Qwen2.5-Coder-32B-Instruct
    config:
      temperature: 0.0
      apiKeyEnvar: TOGETHER_API_KEY
      cache: false
      max_tokens: 4096
```

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Configure togetherai provider with max_tokens greater than 1024
2. Make a call where you know the response will be greater.
3. See the response, it is cut by 1024 tokens.

**Expected behavior**
`max_tokens` config passthrough the together API

**Screenshots**
I can add it if needed.

**System information:**

```
====================================================================================================================================================================
Promptfoo Debug Information
====================================================================================================================================================================
{
  "version": "0.107.3",
  "platform": {
    "os": "darwin",
    "release": "24.3.0",
    "arch": "arm64",
    "nodeVersion": "v23.10.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```

- Promptfoo version:
0.107.3

**Additional context**
As a workaround, if I set `OPENAI_MAX_TOKENS=4096` it works.


same issue here

Fixed by https://github.com/promptfoo/promptfoo/pull/3468
Repository: promptfoo/promptfoo
Issue: Prompfoo giving false-positives?
Root cause: Validation Gap

Hello,

Here is a simple example with two obvious errors.
Yet Promptfoo claims `Failures: 0`

Demonstration: `npx promptfoo@latest eval`:
![image](https://github.com/promptfoo/promptfoo/assets/2452725/cf4cffc9-3a57-441d-9a01-625b921daca7)


Code:
prompt.txt
```
{{question}}
```

promptfooconfig.yaml
```
prompts: prompt.txt # prompt.txt is just a file that contains {{question}}, since we're passing the question directly through to the provider.
providers:
  - exec:python -c "print('[FAIL] 9691')"

# These test properties are applied to every test
defaultTest:
  assert:
    - type: not-contains
      value: returned status code
    - type: not-contains
      value: "Error: "
    - type: not-contains
      value: "[FAIL]"
    - type: not-contains
      value: "Error: "

tests:
  - vars:
      question: 19382/2?
    assert:
      - type: contains
        value: 9691
  - vars:
      question: Hello. Please tell me 'Hello'.
    assert:
      - type: contains
        value: Hello
```

Did I miss or misunderstand something?
My end goal is to test a python script in CI/CD, and rely on `Failures: 0` to validate tests. But since it's always `Failures: 0`...

Thanks for your answers,

Hi @matthieudelaro,

Thanks for the catch.  The script invocation is failing due to a syntax error (which incidentally is related to the escaping issue solved in #343).  The failure counter isn't being incremented when the shell command itself throws an error.

This is fixed in 61d1b068f26c63f3234dc49c9d5f5104b9cf1cda and will be resolved in the next published release.

Resolved in version 0.33.0

Perfect, thanks for solving that so quickly 🙏
I ran the same test : works like a charm.
Repository: promptfoo/promptfoo
Issue: Bedrock model identifier for Claude 2.1 is `anthropic.claude-v2:1`; Code splits : so it hits v2
Root cause: Algorithmic Error

![image](https://github.com/promptfoo/promptfoo/assets/3356134/3204fb2b-c3e8-4a73-ae12-dc49dc276067)

It looks that in code, it takes only the modelName without `:1`

This is the full provider name `bedrock:completion:anthropic.claude-v2:1`

![image](https://github.com/promptfoo/promptfoo/assets/3356134/d212e297-9210-4715-ba56-42b165cc5d39)

Thanks!

Thanks for the catch.  4469b693993934192fee2e84cc27c21e31267e5f solves this issue, and will be included in the next published release.

Should be fixed in 0.33.1.  Let me know if you encounter any issues
Repository: promptfoo/promptfoo
Issue: Adding AZURE_OPENAI_API_KEY in the  config.yaml file. Is it possible?
Root cause: Configuration Error

Is there any way to run promntfoo eval adding the Azure and OpenAI API keys in the config YAML file? i dont want to add the Keys in my environment variables. What are the option we have for this? -- Thanks  a lot

promt.txt
`
Answer the following concisely: {{message}}
`

Here my "promptfooconfig.yaml" --> addind the keys as follow since is the way used when running "promtfoo view". but is not running:
```
env:
  AZURE_OPENAI_API_KEY: XXXXXXXXXXXXXXXX
  OPENAI_API_KEY: YYYYYYYYYYYYYYYYYY
description: ''
providers:
  - id: openai:chat:gpt-3.5-turbo
    config:
      temperature: 0
      max_tokens: 128
  - id: azureopenai:chat:gpt4
    config:
      apiHost: testinstance.openai.azure.com
      temperature: 0
      max_tokens: 128
prompts:
  - prompt.txt
tests:
  - description: 'poem'
    vars:
      message: Write a poem about the sea.
    assert:
      - type: contains
        value: sea

```

![image](https://github.com/promptfoo/promptfoo/assets/6631983/9a9d0ea1-f928-464e-8adb-8b54e752a97f)


Hi @FJLopezGarcia,

You can add the `apiKey` property under each provider config, like so:

```yaml
providers:
  - id: openai:chat:gpt-3.5-turbo
    config:
      apiKey: abc123
      temperature: 0
      max_tokens: 128
  - id: azureopenai:chat:gpt4
    config:
      apiKey: abc123
      apiHost: testinstance.openai.azure.com
      temperature: 0
      max_tokens: 128
```

It looks like `env` config is not being passed when used in CLI.  e8170a7f0e9d4033ef219169115f6474d978f1a7 fixes this in the next release.  Thanks for opening this issue!

`env` is supported properly in 0.33.1.  Let me know if you encounter any issues
Repository: promptfoo/promptfoo
Issue: Does the provider support loading from files?
Root cause: Documentation Deficiency

## Does the provider support loading from files?

my `promptfooconfig.yaml` file
```yaml
prompts: [prompt.txt]
providers:
  - id: openai:chat:gpt-3.5-turbo-0613
    config:
      functions:
        [
          {
            'name': 'get_current_weather',
            'description': 'Get the current weather in a given location',
            'parameters':
              {
                'type': 'object',
                'properties':
                  {
                    'location':
                      {
                        'type': 'string',
                        'description': 'The city and state, e.g. San Francisco, CA',
                      },
                    'unit': { 'type': 'string', 'enum': ['celsius', 'fahrenheit'] },
                  },
                'required': ['location'],
              },
          },
        ]
```

I want to have this grammar

**promptfooconfig.yaml**

```yaml
prompts: [prompt.txt]
providers:
  - id: openai:chat:gpt-3.5-turbo-0613
    config:
      functions:
         - file://./get_current_weather.json
```

**get_current_weather.json**

```json 
{
    'name': 'get_current_weather',
    'description': 'Get the current weather in a given location',
    'parameters':
      {
        'type': 'object',
        'properties':
          {
            'location':
              {
                'type': 'string',
                'description': 'The city and state, e.g. San Francisco, CA',
              },
            'unit': { 'type': 'string', 'enum': ['celsius', 'fahrenheit'] },
          },
        'required': ['location'],
 }
```



You can load providers from file, which is pretty similar:

```yaml
providers:
  - file://./provider.json
```

provider.json
```json
{
  "id": "openai:chat:gpt-3.5-turbo-0613",
  "config": {
    "functions": [
      {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            },
            "unit": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"],
              "description": "The unit in which to return the temperature"
            }
          },
          "required": ["location"]
        }
      }
    ]
  }
}
```

I can see why it might be useful to load individual config keys from file...
Repository: promptfoo/promptfoo
Issue: How to increase the number of thread when redteaming
Root cause: Documentation Deficiency

Hi,

I am just wondering how to increase the number of the thread when running a redteam test. 
Didnt find any instruction in the docs. 

<img width="506" alt="Image" src="https://github.com/user-attachments/assets/50bd690b-8fff-41ba-9bb6-54baf8075743" />

Thank you so much.

Use the `--max-concurrency` arg :)

```
promptfoo eval -c new_test.yaml --max-concurrency 20
```

The shorthand for this arg is `-j`

Thank you!
Repository: promptfoo/promptfoo
Issue: Promptfoo shall support mono-file sandboxed tests natively
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently, building sandbox tests requires to use a custom Python checker, in which one has to call another seemingly not actively maintained lib (epicbox), and define the right Docker, link it back to the test config, etc.
It is overall a fairly complex process.

**Describe the solution you'd like**
We would like promptfoo configs to support sandboxed tests natively, typically mimicking llm-rubric asserts. The assert config could look like this

```yaml
type: sandbox
canonical_solution: |
    def the_solution():
         print('this is a great solution!')
unit_test_type: std_out_equality
sandbox: "python:latest-alpine"
```
- type: defines the assert type: here a sandboxed test
- canonical_solution: what we consider to be the truth for the coding task required
- unit_test_type: what kind of unit test to perform between the canonical solution and the llm answer within the sandbox. Types could be: output_equality (the 2 solutions result in the same outputs), pass (the llm solution simply runs through), raise_same (the 2 solutions raise the same exception), std_out_equality (the 2 solutions output the same messages), runtime_threshold (the LLM solution runtime shall not be above n*runtime of canonical solution), etc.
- sandbox: name of the local Docker image to use for the test

Promptfoo shall be prescriptive of what Docker images to use, in certain well-known situations, if possible. Having at least some clear examples in the doc would help a lot.

Promptfoo shall let the user define whether image is defined in a defaultTest (same image for all tests) or overridden by another image at assert level. 

**Additional context**
Ideally promptfoo shall have 
- a caching mechanism to avoid tearing down completely images between tests so that the same image can be reused multiple times during the entire run, thus improving runtime



If you do not want to use epicbox, you could replace it with a bespoke Docker setup in the custom provider here: https://www.promptfoo.dev/docs/guides/sandboxed-code-evals/#set-up-the-python-assertion-script

Sandboxing is somewhat out of the scope of Promptfoo, but we would absolutely update the tutorial if there's a better approach that you can get working!
Repository: promptfoo/promptfoo
Issue: Clearly document that the tool sends telemetry, possibly with explicit opt in

First up, thanks for working on this - it's an interesting project.

It's usually a surprise to find that an app of this sort is collecting telemetry, particularly when it's not clearly documented as doing so.

At a minimum I'd suggest that a clear message is output whenever telemetry will be collected with instructions to disable telemetry, or disable the clear message (an explicit opt in).

Will follow up on this as I'm traveling today, however I've added some details here: https://promptfoo.dev/docs/configuration/telemetry

https://github.com/typpo/promptfoo-docs/commit/6ff0b0c6a3151ce00b8624485d058e93a1ac03d6

A notice on the first run will be out in the next release: #39   And as noted earlier, there is also a docs page on telemetry :)  Thanks!
Repository: promptfoo/promptfoo
Issue: Error saving result: TypeError: Converting circular structure to JSON
Root cause: Resource Mishandling

**Describe the bug**
Using the promptfoo Node package (v0.107.4), I am seeing this issue when running my evals.

Seeing this error when saving the eval results:
```
Error saving result: TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'TLSSocket'
    |     property 'parser' -> object with constructor 'HTTPParser'
    --- property 'socket' closes the circle {"provider":{"id":"google:gemini-2.0-flash","label":"gemini 2.0 flash","config":{"maxO}
```

I've been seeing this issue with multiple providers since upgrading to 0.107.4 from 0.93.3. The exact same evals now show this error.

**To Reproduce**
Here's an example configuration. 
```
const results = await promptfoo.evaluate(
  {
    description: "Workflow",
    prompts: ["Some prompt"],
    providers: [
      {
        id: "google:gemini-2.0-flash",
        label: "gemini 2.0 flash",
        config: {
          maxOutputTokens: 5000,
          temperature: 0,
          generationConfig: {
            responseMimeType: "application/json",
            responseSchema: {
              name: "example_workflow",
              description: "Handles the workflow",
              input_schema: {
                type: "object",
                properties: {
                  response: {
                    type: "string",
                    description: "Your accurate and helpful response.",
                  },
                  reasoning: {
                    type: "string",
                    description:
                      "Your step by step thought process and reasoning for the workflow.",
                  },
                  required: ["response", "reasoning"],
                },
              },
            },
          },
        },
      },
    ],
    defaultTest: {
      options: {
        provider: {
          id: "bedrock:anthropic.claude-3-5-haiku-20241022-v1:0",
          label: "haiku 3.5",
          config: { region: "us-west-2" },
        },
      },
      assert: [
        // assertions
      ],
    },
    tests: [
      // Test cases
    ],
    writeLatestResults: true,
  },
  { maxConcurrency: 5, showProgressBar: true }
);
```

**Expected behavior**
No errors when running stable evals.

**Screenshots**
n/a

**System information:**

```
{
  "version": "0.107.4",
  "platform": {
    "os": "darwin",
    "release": "22.6.0",
    "arch": "arm64",
    "nodeVersion": "v18.18.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```

- Promptfoo version: 0.107.4

**Additional context**
Add any other context about the problem here.


Thanks for flagging - #3511 should fix in the next release :)
Repository: promptfoo/promptfoo
Issue: Add provider for Together AI
Root cause: Documentation Deficiency

Together AI is a popular cloud provider to run LLMs: https://www.together.ai/

Hm, it looks like we can use the OpenAI's environment variable name with the Together AI API token: https://docs.together.ai/docs

Based on their documentation, something like this should work although I haven't tried it myself:

```yaml
providers:
  - id: openai:chat:mistralai/Mixtral-8x7B-Instruct-v0.1
    config:
      apiBaseUrl: https://api.together.xyz
      apiKey: xxx    # can also set OPENAI_API_KEY envar
```
Let me know how that works for you and I can add it to the documentation.
Repository: promptfoo/promptfoo
Issue: Python assert do not work
Root cause: Validation Gap

After upgrading to version 0.33.1, no Python asserts work anymore. I get the following error for all python asserts, single and multiline ones:

```
Python code execution failed: Python assertion must return a boolean, number, or {pass, score, reason} object
```

Some python asserts that I have in the config file:

```yaml
          # Starts with capital letter
          - type: python
            value: output.strip()[0].isupper()

          - type: python
            value: |
              paragraph = context["vars"]["paragraph"]
              paragraph_words = len(paragraph.strip().split())
              output_words = len(output.strip().split())
              response = (output_words > 0.5 * paragraph_words) and (output_words < 1.6 * paragraph_words)
              print(response)
```

Using version 0.32 works as expected.

Hi @miltondp,

Thanks for the report.  #353 should fix this and will be published shortly.

Please let me know if you still have trouble using 0.33.2

It works, thank you, @typpo!

However, `promptfoo view` does not work with the latest version. It works with v0.32.0

Hmm, looks like I published a bad build.  The web view should be good in 0.33.3.
Repository: promptfoo/promptfoo
Issue: Share result isnt working
Root cause: External Dependency Breakage

Hi to all,

I am using share function to make my scan result publicly. It worked yesterday (27th Mar). However, when I use this function today, it shows the internal server error(Figure below). 

![Image](https://github.com/user-attachments/assets/811de272-ca27-4f6e-be6c-e2f1b87b1ee2)

Is this my problem? or is there any solution for that. 

Really thanks for any help.

@mrpitaaaaa We're in the process of deprecating that platform. Our new platform is available at https://www.promptfoo.app/login . You can signup for a free account that will let you share eval results securely with your team.