zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: [FAIL]: Thread run failed: requires_action. OpenAI Assistant function calling
Root cause: Validation Gap

When i tried to test function calling for openAi assistants api, when I got this error - `[FAIL]: Thread run failed: requires_action`.  I think the issue is code checks for run is completed before checking for the steps and the tool_calls in those steps.  Error occurs at this block mostly - https://github.com/promptfoo/promptfoo/blob/b352cae2e4f8080839cc41f599c18c9a6fc4fc69/src/providers/openai.ts#L454

<img width="1487" alt="Screenshot 2023-12-05 at 8 18 50 PM" src="https://github.com/promptfoo/promptfoo/assets/82715038/78d1bc65-176f-4515-bb52-4c7835ecff48">




Hi @luchenkob, 

Thanks for catching this.  I've made a change to not error on `requires_action`, but this does open the question of what should happen.  e.g. should the run stop there, should we auto-confirm the action, etc.  What are your thoughts?

Hi @typpo,

Require action is same as the tools_calls in the steps.

> if (toolCall.type === 'function') {
>             outputBlocks.push(
>               '[Call function ${toolCall.function.name} with arguments ${toolCall.function.arguments}]',
>             );
>             outputBlocks.push('[Function output: ${toolCall.function.output}]');
> }
> 

So we can send the output back same way as we do in steps and end the run.

But to have complete test suite, it will be necessary to be able to send back a response when a function is called. As I would like to check how the assistant will respond to different types of submit-output for same  function.

Maybe for this user can create a prompt that says, `This is output for xyz function- {{output}}`, and hopefully it will be close enough.

This PR https://github.com/promptfoo/promptfoo/pull/473 adds support and will be in the next published release.  Thanks @aloisklink!
Repository: promptfoo/promptfoo
Issue: Broken links in plugins readme md page

**Describe the bug**
All of the links on the site/docs/red-team/plugins/index.md are broken and give a 404 page.

**To Reproduce**
1. Go to the path as above
2. Click on any link for example [MITRE ATLAS](https://github.com/promptfoo/promptfoo/blob/main/docs/red-team/configuration/#mitre-atlas)

- Promptfoo version: NA


Hi @justary27, the docs are designed to be rendered on the live website: https://www.promptfoo.dev/docs/red-team/plugins/.  Looks like it's working ok!

Oh okay thank you!
Repository: promptfoo/promptfoo
Issue: Overriding providerId in custom python provider
Root cause: Documentation Deficiency

Hello,

Is it possible to override providerId inside custom python provider script as we did in javascript provider as below? I need this because I am generating a unique id inside provider python script and than I am using this id inside custom assertion by reading from context[provider][providerId] and also output json file.

```

class CustomApiProvider {
  constructor(options) {
    // Provider ID can be overridden by the config file (e.g. when using multiple of the same provider)
    this.providerId = options.id || 'custom provider';

    // options.config contains any custom options passed to the provider
    this.config = options.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt, context) {
   ....
    ....
  }
}

module.exports = CustomApiProvider;
```


Hi @merveozbay! We are actively working on this. Going to re-open for you. 


Yes, Here's a way to override the providerId in your Python provider:

```python
class CustomPythonProvider:
    def __init__(self, options):
        # Provider ID can be overridden just like in JavaScript
        self.provider_id = options.get('id', 'custom provider')  # Same as options.id || 'custom provider'
        self.config = options.get('config', {})  # Same as options.config
    
    def id(self):
        return self.provider_id
    
    def call_api(self, prompt, options, context):
        return {
            "output": f"Response from provider '{self.provider_id}': {prompt}",
            "metadata": {
                "providerId": self.provider_id,  # This will be in the output JSON
                "config": self.config
            }
        }

def call_api(prompt, options, context):
    provider = CustomPythonProvider(options)
    return provider.call_api(prompt, options, context)
```
Repository: promptfoo/promptfoo
Issue: Loading variables from a file with scenarios does not work
Root cause: Algorithmic Error

According to the [documentation](https://promptfoo.dev/docs/configuration/guide#avoiding-repetition), it is possible to load the content of a variable (under `vars`) from a file using the `file://` prefix. However, this does not seem to work when the variables are under `scenarios` / `config`. For example:

```yaml
prompts:
  [file://../../prompts/baseline.txt, file://../../prompts/candidate.txt]
providers:
  - ollama:mistral:7b-instruct-fp16
scenarios:
  - config:
      - vars:
          test_description: spelling/grammar errors
          title: Projecting genetic associations through gene expression patterns highlights disease etiology and drug mechanisms
          keywords: 'genetic studies, functional genomics, gene co-expression, therapeutic targets, drug repurposing, clustering of complex traits'
          paragraph: file://./spelling_errors.md
    tests:
      - description: Introduction
        assert:
          - type: python
            # Contains the header
            value: |
              print("## Introduction\n" in output)
```

does not load the file `spelling_errors.md` on the variable `paragraph`, but instead the value `file://./spelling_errors.md`. The file `spelling_errors.md` has Markdown content like this:

```md
## Introductin

Genes work together in context-specific networks to carry out different functions [@pmid:19104045; @doi:10.1038/ng.3259].
Variations in these genes can change their functional role and, at a higher level, affect disease-relevant biological processes [@doi:10.1038/s41467-018-06022-6].
```

Maybe this only works when the variables are defined under `tests`? It would be nice to load the content of variables regardless of the test structure (standard tests vs scenarios).

Hi @miltondp,

Thanks for the report.  You're right, this does seem like an issue with how tests are constructed from scenarios.  Working on this fix in #331 

Awesome, thank you so much, @typpo, and thanks for this great project!

Hi @miltondp, thank you for reporting this issue and for your positive feedback on promptfoo! We forgot to close this issue. This should be supported now. Please update to the latest version and give it a try. If you have any more questions or run into further issues, feel free to reach out. Thanks again!
Repository: promptfoo/promptfoo
Issue: 如何输出deepseek的think过程
Root cause: Documentation Deficiency

我想要在输出的时候显示LLM返回的think+output，而不是只有最终的output，如何实现？


谢谢你的反馈！能否提供一下你的 promptfooconfig 配置文件，以便我们定位问题？另外，你当前使用的是哪个托管服务提供商呢？

可以参考一下 DeepSeek 推理模型的 API 文档：
https://api-docs.deepseek.com/guides/reasoning_model

欢迎随时回复提供更多信息，我们会协助你进一步调试！

十分感谢您的答复，我是一个代码初学者，下面是我的config，我不知道是不是需要设置输出的参数


![Image](https://github.com/user-attachments/assets/af7e105d-ab8f-4a56-9fb5-82dbef9bfe39)

我知道需要使用参数来输出think，但是不知道如何在promptfoo中输出。

![Image](https://github.com/user-attachments/assets/7f998fcf-aa49-44ff-8738-c0bf062ac567)

Hi @FaderYin! We shipped support for this (a while ago). Please pull the latest version of promptfoo and give it a try. Let us know if anything isn’t working as expected, and feel free to comment or reopen the issue. Thanks again for the bug report! 🙌
Repository: promptfoo/promptfoo
Issue: evaluateOptions are not honored if config file is in another directory
Root cause: Configuration Error

**Describe the bug**
When running `promptfoo eval -c somefolder/promptfooconfig.yaml` from a directory other than the config file’s own directory, the evaluateOptions defined in the config file (e.g. maxConcurrency: 3) are not applied during evaluation.

**To Reproduce**
1. Define a `promptfooconfig.yaml`
```yaml:promptfooconfig.yaml
providers:
  - id: openai:gpt-4o-mini

prompts:
  - "translate {{ input }} into language {{ language }}"

tests:
  - vars:
      language: French
      input: Hello world
  - vars:
      language: Spanish
      input: Where is the library?

evaluateOptions:
  maxConcurrency: 3
```
2. Run it from the same folder, eg `promptfoo eval -c promptfooconfig.yaml`
maxConcurrency is set to 3

3. Run it from some other folder, eg `promptfoo eval -c some-folder/promptfooconfig.yaml`
maxConcurrency defaults to 4

**Expected behavior**
Should be no difference where the config file resides.

**System information:**

- Promptfoo version:
0.106.0


Hey @hwaxxer, thank you so much for reporting this issue and providing clear reproduction steps. We just released a fix in [0.106.1](https://github.com/promptfoo/promptfoo/releases/tag/0.106.1).

Please let me know if this fully resolves the issue or if anything else pops up. Closing the issue for now but please feel free to comment / re-open.
Repository: promptfoo/promptfoo
Issue: Question: Can we have two promptfooconfig.yaml files for two separate cases?
Root cause: Documentation Deficiency

Currently, I'm trying to bring an automation pipeline to our existing code and I wanted to run test cases on each of our prompts. In this case, I have x number of prompts and each prompt has their own set of variables that I need to test them on. (No variable is common between any of the prompts) 

I'm not sure if promptfoo has this but I would like to run those test cases on each prompt without having to configure the `promptfooconfig.yaml` file again and again. Is it possible to have multiple yaml files and then run any one file that I like for eval? Is there some existing configuration that already does this?

Right now, correct me if I'm wrong, the `promptfooconfig.yaml` file can only allow me to use a prompt.txt file with a set of prompts that all use the same variables. However, in my case, each prompt uses their own set of variables. 

Hi @anoura12,

If I'm understanding correctly, sounds like you can do this by specifying the config file you want to run:

```
promptfoo eval -c myconfig1.yaml
```

If desired, you can combine multiple configs:

```
promptfoo eval -c myconfig1.yaml myconfig2.yaml
```

Yes, that's what I was looking for. 

Additionally, is it possible to define prompts, providers and tests through the command line?

Yep, the command lines args for [`promptfoo eval`](https://promptfoo.dev/docs/usage/command-line#promptfoo-eval) override the config settings.  So you should be able to set prompts, providers, and tests for each individual eval run.

Let me know if you have any more questions :)
Repository: promptfoo/promptfoo
Issue: Moderation using Azure Content Safety API
Root cause: Unimplemented Feature Gap

It would be really useful to configure promptfoo for moderation using Azure Content Safety API. 

https://learn.microsoft.com/en-us/rest/api/contentsafety/text-operations/analyze-text?view=rest-contentsafety-2024-09-01&tabs=HTTP

I'm not sure if this issue is possible to build or if it is in the roadmap already, anyway I'm opening this issue to hear from you.

Our privacy and company policy does not allow us to use any external API/provider besides Azure services and we can not move forward with promptfoo unless the moderation is done using Azure Content Safety API.



Hey @alex-pollan, you can use this assertion for Azure guardrails(if you are using azure deployment and have guardrail/content filter configured)  https://www.promptfoo.dev/docs/configuration/expected-outputs/guardrails/#basic-usage

Or do you want the moderation to use Azure for you app deployments elsewhere?

Feel free to grab time with me here https://cal.com/guangshuo-zang-mjy19w if you feel like it will help unblock you better.


Hey @MrFlounder, I want to use the Azure Content Safety API in red teaming test cases that use plugins that require moderation (i.e.: plugin `harmful`). 
Currently only OpenAI and Llama via Replicate are supported for moderation (as per: https://www.promptfoo.dev/docs/configuration/expected-outputs/moderation/).
I would like to leverage those plugins (specially test cases generation and strategies!) instead of having to write custom cases myself with the guardrail assertion. Makes sense?


@alex-pollan Thanks! We added Azure API Moderation `https://www.promptfoo.dev/docs/configuration/expected-outputs/moderation/`. You can use it the same way as current OpenAI or LlamaGuard just by setting up environment variables. Release: (https://github.com/promptfoo/promptfoo/releases/tag/0.106.1)

That's amazing news, thanks!
We'll give it a try ASAP!
Repository: promptfoo/promptfoo
Issue: Question: Output of tests and timeouts
Root cause: Documentation Deficiency

This application is awesome. We have been using it for a few hours and it is a great productivity tool for what we do.

We did find 2 things that you may already have solutions for, but as newbes we could not find how to solve them in the documentation.

1 - with the web interface, it would be nice to be able to get the test results stored to a file in a computer readable format (maybe json). This would allow us to perform a higher level of analysis on them or to aggregate results from multiple sets of tests.

2 - for the ollama provider, it would be nice to have an optional setting that lets you set a maximum amount of time a particular prompt is allowed to run. Sometimes we know what the maximum amount of time we want to spend on a prompt. So once that time has elapsed, the test would be considered a failure.

Again thanks for creating a great tool.

Hi Jim,

Thanks for the kind words and I'm glad you're finding the tool useful!

1. By default, eval results are written to the `~/.promptfoo/output` directory in JSON format.  You may prefer to read these programmatically rather than using the webview (adding a download button to the webview is a good idea, though).

2. promptfoo supports a `REQUEST_TIMEOUT_MS` environment variable that can be overridden.  e.g. to set a 20-second limit
    ```
    REQUEST_TIMEOUT_MS=20000 promptfoo eval
    ```

Closing this one out.  Let me know if you have any more questions :)
Repository: promptfoo/promptfoo
Issue: Azure App Services do not support mapping /home to custom mounted storage
Root cause: Documentation Deficiency

After upgrading to v0.106 our cloud promptfoo instance (App Service with Linux container) stopped loading the evals.

After analysis we realized that the previous versions look for the SQLite db in the mounted path `/root`, but now it does look for the db in `/home`.

The problem is that App Services do not support a mounted path for `/home`. See: https://learn.microsoft.com/en-us/azure/app-service/configure-connect-to-azure-storage?tabs=basic%2Cportal&pivots=container-linux

_"Mapping / or /home to custom-mounted storage isn't supported."_

**Describe the solution you'd like**
This problem could be sorted out by using an environment variable to override the path to the promptfoo configuration and DB.

**Describe alternatives you've considered**
Enabling the App Service internal storage would persist the DB but it won't let us to manage the persisted DB appropriately (backups, etc).

Thanks!

Hey @alex-pollan, thanks for flagging this! You can control the configuration and SQLite DB location using the `PROMPTFOO_CONFIG_DIR` environment variable (see details [here](https://github.com/promptfoo/promptfoo/blob/0.106.1/src/util/config/manage.ts#L10-L18)). Let me know if this works or if you have any other questions - happy to help further.

Perfecto!, it works as expected, thanks!
Repository: promptfoo/promptfoo
Issue: Env variables not supported anymore in header fields
Root cause: Configuration Error

I have custom providers which uses env variables in the id, the body, the header.
Till v0.104.4 everything was working. With v0.105.x the env variables in the **header** section are empty / ignored.

**To Reproduce**

Use a provider like that:

```
id: 'https://{{env.TENANT}}-{{env.APPLICATION}}.{{env.BASE_URL}}/api/{{env.ASSISTANT_API}}'
config:
  method: 'POST'
  headers:
    'Content-Type': 'application/json'
    'Cookie': 'REGION-SESSION={{ env.REGION_SESSION }}; XSRF-TOKEN={{ env.XSRF_TOKEN }}; SESSION={{ env.SESSION }};'
    'X-XSRF-TOKEN': '{{env.XSRF_TOKEN}}'
  body:
    messages: '[{"role":"user","content":"{{question}}"}]'
  responseParser: 'json'
```

All these parameters must defined in the env file. But when running a test using this provider the variables below headers are ignored and empty.

With 0.104.4 it is working
With > 0.105.x it is not working anymore.

Workaround: use 0.104.4 till it is fixed.

Thanks @mldangelo works fine again with 0.107.1
Repository: promptfoo/promptfoo
Issue: Expose all assertions for jest integration
Root cause: Unimplemented Feature Gap

Thanks for the project. I've been working on integrating it with jest for testing my project. According to the [docs for jest integration](https://www.promptfoo.dev/docs/integrations/jest)

> For more information on supported checks, see [Expected Outputs documentation](https://www.promptfoo.dev/docs/configuration/expected-outputs/).

and these are the supported assertions: https://www.promptfoo.dev/docs/configuration/expected-outputs/#assertion-types

But when using with jest, i cannot find how to call some of them, for example the `rouge-n` assertion. Based on the code although there is `handleRougeScore` defined in `assertions.ts`, it is not exported. Only these are exposed

https://github.com/promptfoo/promptfoo/blob/91f43cc8c6a7f562dab5f8c48503e76efb687123/src/assertions.ts#L1045-L1056

Is it possible to use other assertions with jest integration? Or any concerns to export them?

Thanks

Hey @joshuali925, I agree it makes sense to export the other assertions. The rationale was more or less that other assertions are pretty straightforward to reproduce (e.g. `contains` or `equals`), but the model-based ones aren't.  You're right that maybe some useful ones slipped through and are not exported.

That makes sense, most of them are trivial to implement. For my case I'm actually planning to call python code to run `rouge` and some other metrics together, so it's ok if they are not supported. You can close this issue or leave it open to see if others have similar needs, thanks.
Repository: promptfoo/promptfoo
Issue: When the evaluation of "npx promptfoo@latest" fails, the process exit code is still 0
Root cause: Validation Gap

```
$ npx promptfoo@latest eval
...
Successes: 0
Failures: 2
Token usage: Total 4297, Prompt 0, Completion 0, Cached 4297
Done.
$ echo $?
0
```

Thanks for catching this.  Fixed in [65d79fb](https://github.com/promptfoo/promptfoo/commit/65d79fb428f311c7fc495f96385addd5a2990e3d)
Repository: promptfoo/promptfoo
Issue: Add Bedrock deepseek support (serverless)
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**

Lack of deepseek model support.

**Describe the solution you'd like**

Add support for deepseek models.

**Describe alternatives you've considered**

N/A

**Additional context**

https://aws.amazon.com/blogs/aws/deepseek-r1-now-available-as-a-fully-managed-serverless-model-in-amazon-bedrock/


Hi @moltar this is supported in the latest version of promptfoo. Please see the [bedrock](https://github.com/promptfoo/promptfoo/tree/main/examples/amazon-bedrock) example. Please let us know into any other issues!
Repository: promptfoo/promptfoo
Issue: [Bedrock] AI21 TypeError: Cannot read properties of undefined (reading '0')
Root cause: Validation Gap

Configuring like this:


```ts
    {
      'bedrock:ai21.jamba-1-5-large-v1:0': {
        id: 'AI21 Jamba 1.5 Large',
        config: {
          region: 'us-east-1',
          temperature: TEMPERATURE,
        },
      },
    },
```

Fails with:

> TypeError: Cannot read properties of undefined (reading '0')

Happens here:

https://github.com/promptfoo/promptfoo/blob/70c3c403b3426785616e99072c2f6c82e275b157/src/providers/bedrock.ts#L430-L434

I'm not able to repro with the following yaml:

```yaml
description: 'Bedrock A21 Eval'

prompts:
  - 'Write a tweet about {{topic}}'

providers:
  - id: bedrock:ai21.jamba-1-5-large-v1:0
    config:
      region: us-east-1

tests:
  - vars:
      topic: Our eco-friendly packaging
```

Can you please provide the full debug output with `--verbose`?  It would help to know what the API is actually returning.

Opened a PR that will gracefully handle the case you've identified as an undefined output.
Repository: promptfoo/promptfoo
Issue: if using prompt function enable skipping random nunjacks rendering
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
i am populating variables myself in my prompt function so i have no need nunjucks to get involved


currently it throws an error and adds no value as far i am concerned in terms of features used

```
//nodejs/22.14.0/lib/node_modules/promptfoo/node_modules/nunjucks/src/lib.js:32
    err = new Error(old.message);
          ^

Template render error: (unknown path) [Line 14, Column 50]
  expected name as lookup value, got *
    at Object._prettifyError (//nodejs/22.14.0/lib/node_modules/promptfoo/node_modules/nunjucks/src/lib.js:32:11)
    at Template.render (//nodejs/22.14.0/lib/node_modules/promptfoo/node_modules/nunjucks/src/environment.js:442:21)
    at Environment.renderString (//nodejs/22.14.0/lib/node_modules/promptfoo/node_modules/nunjucks/src/environment.js:313:17)
    at renderPrompt (//nodejs/22.14.0/lib/node_modules/promptfoo/dist/src/evaluatorHelpers.js:252:56)
    at async runEval (//nodejs/22.14.0/lib/node_modules/promptfoo/dist/src/evaluator.js:143:28)
    at async processEvalStep (//nodejs/22.14.0/lib/node_modules/promptfoo/dist/src/evaluator.js:648:26)
    at async //nodejs/22.14.0/lib/node_modules/promptfoo/dist/src/evaluator.js:859:11
```

this is the error thrown, i solved it by putting  an early return in the code

<img width="843" alt="Image" src="https://github.com/user-attachments/assets/109af59b-dbbf-454d-95b6-cd7941dfd485" />


https://github.com/promptfoo/promptfoo/blob/main/src/evaluatorHelpers.ts#L177

happy to make a pr with the correct guidance as how you would do it? i feel like i might be missing some complexity as well given that a few lines lower i see a function called resolveVars, but i already used them in the function so 🤷🏿 


https://gist.github.com/billybonks/4c35b734c7bf4b7b41402694faf67138

this feature already exists
Repository: promptfoo/promptfoo
Issue: Using external python assertion on Mac faced  Bad escaped character when vars with double quote
Root cause: Validation Gap

Having the Python code execution failed: Bad escaped character issue when trying:
https://www.promptfoo.dev/docs/configuration/expected-outputs/python/#external-py
and
https://www.promptfoo.dev/docs/providers/custom-script

**Files used:**

promptfooconfig.yaml
`prompts: 
  prompt_gpt4_parameters.txt: gpt4_parameters
providers:
  - openai:gpt-4-1106-preview:
      id: gpt4
      prompts: 
       - gpt4_parameters
tests: 
  - vars:
      input: |-
        You said "Hello World"
    assert:
      - type: python
        value: file://assert.py
`

assert.py

`import sys

print (sys.argv[2])`

Error:
Python code execution failed: Bad escaped character in JSON at position 698 (line 1 column 699)

**System Info**

Mac using chip M2 and OS is Senoma 14.2

**Command**
npx promptfoo@latest eval







Hey @kylekhchen, thanks for the report.  Fix underway here: https://github.com/promptfoo/promptfoo/pull/338

Resolved in version 0.33.0