zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Export Tokens/sec to .json
Root cause: Unimplemented Feature Gap

When exporting to `.json` format, information about multiple metrics such as 1) token usage 2) latency 3) total passed asserts is stored. For example the total number of passed asserts for a provider are stored under the json path `["results"]["table"]["head"]["prompts"][index]["metrics"]["assertPassCount"]`.

I'd collect these results to assemble a report. For this purpose it would be convenient to have tokens per second also available from a `.json` results file. It appears that currently the tokens/sec information is only available via the promptfoo web view.

To calculate tok/sec, I recommend just doing `metrics.tokenUsage.total / metrics.totalLatencyMs * 1000`

Thanks! I had assumed latency was related to the time it takes for the server to respond with a connection, not till completion was finished. Given the `a.xyz ms` format and switching between "," and "." as decimal vs thousand separators had me confused.

No worries :)
Repository: promptfoo/promptfoo
Issue: why openai is requested here and how to change it to deepseek
Root cause: Documentation Deficiency


 why openai is requested here and how to change it to deepseek

![Image](https://github.com/user-attachments/assets/0b0053c9-a30f-4055-a54b-bd5ca6fede0d)

You can override the redteam provider in your config with 

```yaml
redteam: 
  plugins: ...
  provider: `openouter:deepseek-r1`
```
 for example. Who is hosting deepseek for you?

The redteam part has been modified to the following but still reports an error. Could you please tell me what the problem is?
```
redteam:
  purpose: 'The objective of the application is: Electric Power Knowledge Assistant'
  plugins:
    - id: ascii-smuggling  # Tests vulnerability to Unicode tag-based instruction smuggling attacks
  strategies:
    - id: jailbreak  # Single-shot optimization of safety bypass techniques
    - id: jailbreak:composite  # Combines multiple jailbreak techniques for enhanced effectiveness
  providers:
  - id: deepseek:deepseek-chat
    config:
      temperature: 0.7
      max_tokens: 4000
      apiKey: sk-xxx
```

> You can override the redteam provider in your config with
> 
> redteam: 
>   plugins: ...
>   provider: `openouter:deepseek-r1`
> for example. Who is hosting deepseek for you?

base_url:https://api.siliconflow.cn

Hi @blu3sh0rk could you run your evaluation with the -v command `promptfoo redteam run -v --force --no-progress-bar` and paste the output so its easier to see where the error might be occurring.



@blu3sh0rk - this question was answered above. You can override the redteam provider in your config with the `provider` key under `redteam`.

For DeepSeek via SiliconFlow:

```yaml
redteam:
  plugins: [...]
  provider:
    id: openai:deepseek-chat
    config:
      apiBaseUrl: https://api.siliconflow.cn/v1
      apiKey: your-key
```

Closing as answered. Feel free to open a new issue if you're still having trouble with the configuration.
Repository: promptfoo/promptfoo
Issue: custom scripts for embedding 
Root cause: Documentation Deficiency

Hi,
    Is there any sample to do custom script for embedding in **similar** assertion? Because all LLMs and Embedding models are deployed and served internally. It's easy to write script to call them and llm custom script is working. Just have no idea how to do it with embedding models

Happy to provide an example.  Are you writing the script as a Javascript `ApiProvider` class, or are you using `exec`?

> Happy to provide an example. Are you writing the script as a Javascript `ApiProvider` class, or are you using `exec`?

Hi typpo, thanks for reply. I managed to do it with ApiProvider :) like:

```
prompts: [prompts.txt]
providers:
  - ./customLLMProvider.js:
      id: custom-llm-provider
      config:
        url: https://xxxxx
tests:
  - description: Another basic substring check
    assert:
      - type: icontains
        value: "u"
      - type: similar
        value: "Bonjour qui êtes-vous ?"
        threshold: 0.1
    options:
      provider:
        embedding:
          id: ./customEmbeddingProvider.js
          config:
            url: https://xxxxx
```
Repository: promptfoo/promptfoo
Issue: "----------" in prompt texts cause multiple outputs during eval
Root cause: Algorithmic Error

**Describe the bug**
If the character sequence "-----------------------------------" is part of the prompt text, multiple outputs get generated during eval and the values are not correctly assigned to the variables.

e.g. below prompt:

```
Do xyz and take into account below data.

data:
{{dataA}}
-----------------------------------
{{dataB}}
-----------------------------------
{{dataC}}
-----------------------------------
{{dataD}}
```
will be actually sent to the LLM 4 times, producing 4 outputs. 

![Image](https://github.com/user-attachments/assets/e694be3a-c5df-4149-96df-53b3b98880d7)

- The first generation contains the whole prompt + filled variable data for `{{dataA}}`. But it's missing `{{dataB}}`, `{{dataC}}`, `{{dataD}}`.
- The other 3 generations are not expected and only contain
```
--
{{dataB}}
```
```
--
{{dataC}}
```
```
--
{{dataD}}
```
without any other text and are sent to the LLM.


**To Reproduce**

1. Run an eval as per the above example prompt, including the "-----------------------------------" char sequence in the prompt text ("separator line") and the defined vars/placeholders.

**Expected behavior**
One prompt should be sent to the LLM, with all 4 data fields replaced into the defined vars.

**Screenshots**
see above

**System information:**
UNIX

Promptfoo version: 0.112.7

**Additional context**
n/a
Repository: promptfoo/promptfoo
Issue: {{env.variable}} does not work in prompts (question / answer)
Root cause: Configuration Error

I want to use env variables in my prompt. Currently  in the testrun it works like an escaped variable. When I run `promptfoo view` it seems converted to the variable value, but not in all cases

**To Reproduce**

This is my yml file for the test.

```
tests:
  - vars:
      question: 'How is the wheather at location {{env.LOCATION}}?'
    assert:
      - type: llm-rubric
        value: 'The wheather is {{env.WHEATHER}}'
```

env.LOCATION=Karlsruhe
env.WHEATHER=sunny

**Expected behavior**

It should resolve the environment variables and use the values of it - in the question and also in the answer.

**Screenshots**

In `promptfoo view` it shows, that it cannot resolve the variable in the prompt - but it seems in the failure description.
![Image](https://github.com/user-attachments/assets/8afb3875-cf75-4121-a0a9-a832a3c50b33)

When I look into the details, it also looks weird:

![Image](https://github.com/user-attachments/assets/830f4b78-a90b-443f-be09-f480e1993dbf)

**System information:**

```
  "version": "0.113.2",
  "platform": {
    "os": "win32",
    "release": "10.0.19045",
    "arch": "x64",
    "nodeVersion": "v22.14.0"
  },
```

This remains an issue in promptfoo configs in v.0.114.2. Not only does the env var not render in `promptfoo view`, but they're also not interpolated for the configuration. 

![Image](https://github.com/user-attachments/assets/9cb0cb19-354d-41bf-aae4-7dcb7638fa33)

Hi @kylestratis! It's in main but not released yet. We will cut a release later today to resolve it! 

Ah, great to hear, thanks @mldangelo!

has this been fixed ? i'm trying to pass env vars by command but the interpolation is not working

Hi @kylestratis and @danielsotopino! Sorry for the delay.  This was fixed. Pease pull the latest version and give it a try. Feel free to comment or re-open if you encounter any other issues. 

@mldangelo It looks like interpolation almost works in that I have a new error, but it seems on the critical path to using string interpolation at least for the Azure provider config:

```
TypeError: this.apiHost?.replace is not a function

TypeError: this.apiHost?.replace is not a function
    at AzureChatCompletionProvider.getApiBaseUrl (/Users/kyle/.local/share/mise/installs/npm-promptfoo/0.114.7/lib/node_modules/promptfoo/dist/src/providers/azure/generic.js:150:36)
```

This is happening when I'm trying to set `apiHost: {{ env.AZURE_OPENAI_ENDPOINT }}`. Removing this from the config and setting the env var `AZURE_API_HOST` works, and I am now using it as a workaround, but we'd like to be able to use our own env var naming scheme and inject them into the promptfoo config via the templating system.

it doesn't interpolate at defaultTests, it just works at the prompt level

Just as feedback; my original use case is now working. Thanks for the fix.

@kylestratis Maybe open a seperate issue for the new problem.
Repository: promptfoo/promptfoo
Issue: Cannot set `response_format` to external file with OpenAiResponsesProvider
Root cause: Configuration Error

**Describe the bug**
The `openai-structured-output` [example](https://github.com/promptfoo/promptfoo/blob/main/examples/openai-structured-output/promptfooconfig.responses.yaml) seems to suggest that passing an external json schema file as the `response_format` is supported for the Responses API. However the logs in openai seems to suggest that  `response_format` is always `text`. Is this supported yet, or am I using this incorrectly?

I am able to specify the schema properly when using the inline schema format.

Thanks! 

**To Reproduce**
External file as response_format
```
providers:
  - id: openai:responses:o3
    config:
      response_format: file://response_format.json
```

Inline response_format
```
providers:
  - id: openai:responses:o3
    config:
      response_format:
        type: json_schema
        json_schema:
          name: example
          strict: true
          schema:
            type: object
            properties:
              rows:
                type: array
                items:
                  type: array
                  items:
                    type: string
            additionalProperties: false
            required:
              - rows
```

- Promptfoo version: 0.113.2

Hi @jennkao! This was fixed ~2 weeks ago. Sorry for not closing the issue earlier. Please give it a try and let me know how it goes. Feel free to re-open the issue or reach out (michael@promptfoo.dev) if you'd like to chat about other features or like some promptfoo swag!

Hi @mldangelo - running into same problem for AzureOpenAI responses. Promptfoo version: 0.117.5
Repository: promptfoo/promptfoo
Issue: langfuse traces eval
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
No

**Describe the solution you'd like**
promptfoo already provides langfuse integration, but this integration is retrieving prompts directly.
user wants to use promptfoo to evaluate traces or sessions as well.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.


This is a great suggestion! We are using langfuse too as the traces registry. This will be very helpful as we can leverage promptfoo as an declarative evaluation engine with langfuse source
Repository: promptfoo/promptfoo
Issue: Question:  Can I assert and test a value?
Root cause: Documentation Deficiency

For example if my returned text from the LLM always contained text like "Score: 90%"  or "Score:10%" and I wanted to make pass based on score>50% or something.

Possible?

Yep, you can do this in one of two ways:

1. With the `javascript` or`python` assertion type.  Write some code that parses the number out and assert that it's greater than the threshold score.

2. If you are doing this for many assertions, the [`postprocess` property](https://promptfoo.dev/docs/configuration/guide/#postprocessing) may come in handy.  

  You'll still have to write a little Javascript to parse the number from `output`.  Here's a one liner that I use:

  ```yaml
  postprocess: 'parseFloat(/\d+(?:\.\d+)?/.exec(output)?.[0])'
  ```
Repository: promptfoo/promptfoo
Issue: Add link to docs on www.promptfoo.dev

Not a bug, but it took me a little longer to find the docs than I'd like to admit. I would expect a link at the top right corner of the page, next to the github repo link.

Thanks for the awesome project

Thanks for the suggestion, this is done in https://github.com/promptfoo/promptfoo/commit/925976062f2e6fddb45748b519dd522322fafdf1
Repository: promptfoo/promptfoo
Issue: How to do red teaming evaluation with different language?
Root cause: Documentation Deficiency

I walked through the doc and change add a line to the YAML file: 

```
redteam:
     language: 'Chinese'
```

However the prompts generated are still in English and feedback results are in English too. Do you have some idea how to solve this problem so that I want most prompts and response in the report are in Chinese, thank you in advance!

- Promptfoo version: 0.112.4


Try this 

```
  strategies:
    - jailbreak
    - prompt-injection
    - id: multilingual
      config:
        languages: ['en', 'es', 'fr','cn']  # Testing multilingual capabilities
```

> Try this
> 
> ```
>   strategies:
>     - jailbreak
>     - prompt-injection
>     - id: multilingual
>       config:
>         languages: ['en', 'es', 'fr','cn']  # Testing multilingual capabilities
> ```

It works, thank you very much:)
Repository: promptfoo/promptfoo
Issue: Improve docs for OpenAI messages protocol

Hi, I *think* I have this figured out now, but I wanted to document a point of confusion for me when first reading the docs.

The [OpenAI integration docs](https://www.promptfoo.dev/docs/providers/openai/#formatting-chat-messages) imply that input for chat models should be in the `{ "role" : "system", "content": "foo"}` JSON format. Yet the [Getting Started](https://www.promptfoo.dev/docs/getting-started) guide gives prompts as plain text. I get the impression both options are available and plain text messages are automatically wrapped in the equivalent JSON.

Is this correct?

Yes, you are correct :)

On Sun, Jan 28, 2024, 5:23 AM Natan Yellin ***@***.***> wrote:

> Hi, I *think* I have this figured out now, but I wanted to document a
> point of confusion for me when first reading the docs.
>
> The OpenAI integration docs
> <https://www.promptfoo.dev/docs/providers/openai/#formatting-chat-messages>
> imply that input for chat models should be in the { "role" : "system",
> "content": "foo"} JSON format. Yet the Getting Started
> <https://www.promptfoo.dev/docs/getting-started> guide gives prompts as
> plain text. I get the impression both options are available and plain text
> messages are automatically wrapped in the equivalent JSON.
>
> Is this correct?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/421>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJRNEMVURQCWQHEY57LYQZGMFAVCNFSM6AAAAABCOFWT46VHI2DSMVQWIX3LMV43ASLTON2WKOZSGEYDIMJWGE4DAOA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Thanks, feel free to close.

38bfdc4 improves the documentation on this point.  I appreciate the docs feedback, keep it coming!

Will do, thanks.

On Sun, Jan 28, 2024, 18:15 Ian Webster ***@***.***> wrote:

> Closed #421 <https://github.com/promptfoo/promptfoo/issues/421> as
> completed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/421#event-11620360321>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADYUBYEORXBFL6DYJFA2WDYQZ2TRAVCNFSM6AAAAABCOFWT46VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJRGYZDAMZWGAZTEMI>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: how to import OpenAiChatCompletionProvider
Root cause: Unimplemented Feature Gap

`import { AzureOpenAiChatCompletionProvider } from "promptfoo/dist/src/providers/azureopenai";`

```
Error: Missing "./dist/src/providers/azureopenai" specifier in "promptfoo" package
 ❯ e node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:21914:25
 ❯ n node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:21914:627
 ❯ o node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:21914:1297
 ❯ resolveExportsOrImports node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:29466:20
 ❯ resolveDeepImport node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:29485:31
 ❯ tryNodeResolve node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:29210:20
 ❯ Context.resolveId node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:28978:28
 ❯ Object.resolveId node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:63987:64
 ❯ TransformContext.resolve node_modules/vite/dist/node/chunks/dep-9A4-l-43.js:63692:23

```

This is an internal implementation that isn't exposed by the package right now.  Out of curiosity, what are you trying to use it for?
Repository: promptfoo/promptfoo
Issue: Deepseek-R1-0528 is here
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
https://api-docs.deepseek.com/news/news250528

**Describe the solution you'd like**
Please add this model to propmtfoo. 

**Describe alternatives you've considered**
Not having the latest Deepseek in promptfoo -- unacceptable :-) 



Hi @mpasternak -  DeepSeek-R1-0528 is already fully supported in promptfoo through the deepseek: provider. DeepSeek uses stable API aliases, so `deepseek-reasoner` automatically points to the latest DeepSeek-R1-0528 model behind the scenes.

```yaml
providers:
  - id: deepseek:deepseek-reasoner
    config:
      max_tokens: 8000
      showThinking: true  # Shows the reasoning process
```

I'm going to update the docs to clarify! Please let me know if you'd like to see it in any other inference providers too.
Repository: promptfoo/promptfoo
Issue: Allow configuration of delimiter
Root cause: Documentation Deficiency

I was using `---` in some existing prompts, and was surprised by promptfoo's behaviour until I found https://www.promptfoo.dev/docs/configuration/parameters/#multiple-prompts-in-a-single-file

Thanks for the suggestion, this definitely has tripped other people before too.  I hope to completely remove this functionality one day, but for now 0722c70 adds a `PROMPTFOO_PROMPT_SEPARATOR` environment variable which you can use to alter this behavior.

Wonderful! Thank you.
Repository: promptfoo/promptfoo
Issue: Provider-level xfail setting
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I've taken over an eval suite that is partly failing and I want to get it running in CI. Failure causes are varied, but some appear to come from changes to the grader and promptfoo. I don't know exactly what settings were used originally. To move forward, it'd be useful to mark certain assertions as expected to fail for specific providers, and then fix things incrementally.

Also, I think that this sort of setting would be useful in general as a way to document provider choice: If evals are failing for provider X and passing for provider Y, it'd be great to assert that this remains the case when the eval suite is rerun.

**Describe the solution you'd like**
As I want the full suite to be green in CI, it'd be great to be able to mark assertions as expected to fail for specific providers, in the style of [pytest's xfail](https://docs.pytest.org/en/stable/how-to/skipping.html) or the [equivalent jest feature](https://github.com/jestjs/jest/issues/4627).

**Describe alternatives you've considered**
The main alternative I can think of is to parse/query the outputs and implement my own solution.

However, I'm new to promptfoo, so I'm missing something obvious I'd love to hear about it.
Repository: promptfoo/promptfoo
Issue: Feature Request: Ability to render markdown in results
Root cause: Documentation Deficiency

Just a nicety :)

Some API would also work. (E.g. specify JS that is used to render each cell or specify a JS transformer that can return HTML)

This is actually available as of 0.37.0 under `Table Settings`!

![image](https://github.com/promptfoo/promptfoo/assets/310310/7487631c-dc30-4ec7-94ec-3e2fb5960291)

Note that you'll have to increase the max text length for the full rendering, otherwise some markdown may get cut off.




Thanks, I had searched for "markdown" in the docs before opening the issue. Happy to contribute to the docs if you can share instructions for how to build and test them locally.
Repository: promptfoo/promptfoo
Issue: Allow configuration options for generate_tests
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Most other Python and Javascript callbacks can be configured in Python code, but this one cannot.

**Describe the solution you'd like**
It should be possible to pass an object to `tests:` and this object should take config options.

**Describe alternatives you've considered**
I considered using a 'test provider' but I think that providers can only manage a single test at a time.
Repository: promptfoo/promptfoo
Issue: Feature Request: Better workflow for testing prompt stability
Root cause: Unimplemented Feature Gap

Promptfoo is very good at comparing different prompts across a set of test cases. However, when I test prompts I usually want to know not just "is this prompt better than this prompt when run once", but also "will I consistently get good results from this prompt  vs that prompt".

Today I set evaluateOptions.repeat=5 (or some other low number where I can manually review results) and compare the responses. So for each testcase X prompt I review 5 rows and compare them manually.

Ideally, the repeated results would appear in a single tablerow and give me some indication if I got identical responses. (E.g. allow navigating between different responses with left/right arrows on each cell. If a response was received multiple times, show it only once and list how many times each response was received.)

It would also be nice to have some form of automated eval re. response similarity and/or an AI explanation of the differences between them.

I'm dreaming a little on this, but I wanted to share how I'm using Promptfoo today and how it could be more useful for me. If there are other workflows/solutions for this sort of thing, would love to hear.

One more comment on this: the "Show different only" view is very nice, but it's for comparing prompts not repeat runs on the same prompt.

I have the same idea, i'm willing to submit a PR, but i'm not familiar with javascript.

I'm closing this, as its a bit too "general". I'll open issues with more specific ideas for improvement, starting with this.

https://github.com/promptfoo/promptfoo/issues/992
Repository: promptfoo/promptfoo
Issue: Feature Request: Ability to give specific runs a name

I run Promptfoo a *lot* as I experiment. I'd like to refer back to specific runs, without going through the whole list. I'd love the ability to give specific runs custom names and to favorite them.

Makes a lot of sense.  Right now you can add a `description` at the top of your config file:

```yaml
description: 'Simple test'
```

This will display in the dropdown, which makes it a little easier to search and find.
![image](https://github.com/promptfoo/promptfoo/assets/310310/01086af0-bffb-478a-89a8-b6135177da1d)


Thanks, will be using that as a workaround.