zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Transform context for model based evaluation from custom provider
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**

We want to use promptfoo to evaluate the performance of our RAG system.

```
import promptfoo from 'promptfoo';

export default {
    prompts: ['{{input}}'],
    providers: [{
        id: API_URL,
        label: ENV,
        config: {
            headers: {
                "Content-Type": "application/json",
            },
            body: {
                prompt: "{{input}}",
                type: "text",
            },
            evaluateOptions: {
                maxConcurrency: 1,
                delay: 1000
            }
        },
    }],
    defaultTest: {
        options: {
            transform: "output.content", // extract the "content" field from the response
        }
    },
    "tests": get_test_set(),
};
```

The page about the context-recall (https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/context-recall/) shows that the context needs to be provided within the testcase:
```
tests:
  - vars:
      query: 'What is our maternity leave policy?'
      context: file://docs/policies/maternity.md
    assert:
      - type: context-recall
        threshold: 0.9
        value: |
          Employees get 4 months paid maternity leave.
          Leave can be taken before or after birth.
          Additional unpaid leave is available upon request.
```
In our case the custom provider also returns the documents in the "output.source_documents" field. 

**Describe the solution you'd like**

A way to add a transformer of some kind to take the custom provider response and get the context from it.

**Describe alternatives you've considered**


**Additional context**
Add any other context or screenshots about the feature request here.


Hey @blafab-hg, this is now supported via the `contextTransform` property on assertions.

For your RAG system where the context is in `output.source_documents`, you can do:

```yaml
tests:
  - vars:
      query: 'What is our maternity leave policy?'
    assert:
      - type: context-recall
        threshold: 0.9
        contextTransform: 'output.source_documents.map(d => d.content).join("\n")'
        value: |
          Employees get 4 months paid maternity leave.
          Leave can be taken before or after birth.
```

The `contextTransform` accepts a JavaScript expression that receives the full provider response as `output`. You can use it to extract and format the context however you need.

Docs: https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/#dynamically-via-context-transform
Repository: promptfoo/promptfoo
Issue: After upgrading to latest version 0.103.17 from 0.93.3, I get provider not found issue
Root cause: External Dependency Breakage

**Describe the bug**
I have a docker image that runs promptfoo version 0.93.3. I am using an Azure OpenAI model.
I created a new version for this image, with the latest version - 0.103.17 and when I try to run promptfoo tests, I get the error: 
Could not identify provider: .

This is my test.yaml confguration for the provider:
```
defaultTest:
  options:
    provider:
      id: azureopenai:chat:gpt-4o
      label: gpt-4o-2024-05-13
```

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. add the provider configuration to the tests file:
```
defaultTest:
  options:
    provider:
      id: azureopenai:chat:gpt-4o
      label: gpt-4o-2024-05-13
```
2. Make sure you have promptfoo version 0.93.3.
3. Set environment variables:
- AZURE_OPENAI_API_KEY
- AZURE_OPENAI_API_HOST
- SINGULARITY_API_URI

4. Run 'promptfoo eval -c tests/default.yaml --verbose --output test_results.json'
5. See that it is working
6. Update to latest promptfoo version 0.103.17
7. See the error: Could not identify provider: .

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

{
  "version": "0.103.17",
  "platform": {
    "os": "linux",
    "release": "6.10.14-linuxkit",
    "arch": "x64",
    "nodeVersion": "v20.18.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}

- Promptfoo version: 0.103.17

**Additional context**
Add any other context about the problem here.


Can you try using `azure:` instead of `azureopenai:`?

I tried it just now and it has the same behaviour.

Following the [documentation](https://www.promptfoo.dev/docs/providers/azure/#configuration), I have tried the following, but still it is not working:
```
providers:
  - id: azure:chat:deploymentNameHere
    config:
      apiHost: 'xxxxxxxx.openai.azure.com'
      apiKey: 'your-api-key'
```

Hey @alinv0, throw some time on my calendar and we can look at this together: https://cal.com/michael-dangelo/15min
Repository: promptfoo/promptfoo
Issue: Automatic tests does not work with Azure
Root cause: Documentation Deficiency

I have set the AZURE_OPENAI_API_KEY and : 

tests are failing like this :
![Capture d’écran 2023-11-24 à 17 14 24](https://github.com/promptfoo/promptfoo/assets/17323800/b043f96c-f41e-4bbf-aff4-48c4ff58d8d3)

with`promptfooconfig.yaml` set like this : 
```
prompts: file://prompts.txt
providers:
  - id: azureopenai:chat:xxxxx
    config:
      apiHost: 'xxxxxxx.openai.azure.com'
      temperature: 0.5
      max_tokens: 1024
defaultTest:
  assert:
    - type: llm-rubric
      value: Do not mention that you are an AI or chat assistant
tests:
  - vars:
      name: Bob
      question: Can you help me find a specific product on your website?
```
      
 Tests are Ok like this :
![Capture d’écran 2023-11-24 à 17 17 15](https://github.com/promptfoo/promptfoo/assets/17323800/28b189fd-7111-45f3-9c06-7a574169f0db)

when `promptfooconfig.yaml` is set like this : 
```
prompts: file://prompts.txt
providers:
  - id: azureopenai:chat:xxxxx
    config:
      apiHost: 'xxxxxxx.openai.azure.com'
      temperature: 0.5
      max_tokens: 1024
tests:
  - vars:
      name: Bob
      question: Can you help me find a specific product on your website?
```
      


Hi @pierrelandry,

The `llm-rubric` assertion uses OpenAI by default.  You'll have to override that provider.  Try updating `defaultTest` to something like this:

```yaml
defaultTest:
  options:
    provider:
      id: azureopenai:chat:xxxx
      config:
        apiHost: 'xxxxxxx.openai.azure.com'
        temperature: 0.5
        max_tokens: 1024
  assert:
    - type: llm-rubric
      value: Do not mention that you are an AI or chat assistant
```
Repository: promptfoo/promptfoo
Issue: After upgrading to latest version 0.103.17 from 0.103.17, Unable to config defined in typescript
Root cause: External Dependency Breakage

**Describe the bug**
Unable to run Promptfoo configs written in Typescript since update to version 0.103.17

**To Reproduce**

1. define config in config.ts file:

```typescript
import type { UnifiedConfig } from 'promptfoo';

const config: UnifiedConfig = {
  description: 'Simple Test',
  prompts: ['Say hello to {{person}}'],
  providers: [
    {
      id: 'openai:gpt-4o-mini',
    },
  ],
  tests: [
    {
      vars: {
        person: 'carl',
      },
      assert: [],
    },
  ],
};

export default config;

```

2 run `npx promptfoo@latest eval -c ./config.ts`

**System information:**

```
Attempting to import module: {"resolvedPath":"/home/carl09/dev/testing/src/test/promptfoo-config.ts","moduleId":"test/promptfoo-config.ts"}
TypeScript/ESM module detected, importing tsx/cjs
ESM import failed: Error: Cannot find module 'tsx/cjs'
Require stack:
- /home/carl09/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/esm.js
- /home/carl09/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/util/index.js
- /home/carl09/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/commands/cache.js
- /home/carl09/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/main.js
Attempting CommonJS require fallback...
CommonJS require also failed: SyntaxError: Unexpected token '{'
file:///home/carl09/dev/testing/src/test/promptfoo-config.ts:1
import type { UnifiedConfig } from 'promptfoo';
            ^

SyntaxError: Unexpected token '{'
```

- Promptfoo version: 0.103.17

**Additional context**

I think this happened because the package `tsx` was moved to a devDependancy in this PR #2884 


Thanks for the fix!

The fix has been released in [0.103.18](https://github.com/promptfoo/promptfoo/releases/tag/0.103.18) Please pull the latest version and let us know if it resolves your issue.
Repository: promptfoo/promptfoo
Issue: [BUG] `rubricPrompt` breaks when using `.json` file
Root cause: Validation Gap

**Describe the bug**

The `rubricPrompt` throws an error when used with json file

**To Reproduce**
I am using custom LLM rubric messages stored in a `.json` file:

```json
[
  {
    "role": "system",
    "content": "Evaluate the summary for factual correctness"
  },
  {
    "role": "user",
    "content": "Here is the form in YAML format {{ form }}.\n\nThis is the summary generated by the system: {{ output }}.\n\nYour Task: {{ rubric }}.\n\nReply in JSON format as follows: {\"reason\": \"<Analysis of the rubric and the output>\", \"score\": <0.0-1.0>, \"pass\": <true or false>}"
  }
]
```

And use the following `promptfooconfig.yaml`:
```yaml
description: "Bug demo"

prompts:
  - 'summarize this form value in 10 words: {{ form }}'

providers:
  - bedrock:amazon.nova-micro-v1:0

tests:
  - vars:
      form: |
        title: "bug report"
        description: "I found a bug: the system crashed."
    assert:
      - type: llm-rubric
        rubricPrompt: file://rubric.json
        value: Evaluate summary correctness and clarity.
        provider: bedrock:amazon.nova-micro-v1:0
```

As a result, I get an error:
```
Error: Invariant failed: rubricPrompt must be a string
```


Meanwhile, if I just switch the file type of `rubric.json` to `.txt` it works just fine:

<img width="1154" alt="Image" src="https://github.com/user-attachments/assets/dbffc196-85ed-49c9-b439-b92ecb400bef" />


**Expected behavior**

Support `.json` files for `rubricPrompt`.

**System information:**

- Promptfoo version: 0.103.17



Thanks for the report.  This should be fixed in #2931
Repository: promptfoo/promptfoo
Issue: Error invoke o3-mini
Root cause: Interface Contract Mismatch

**Describe the bug**
When invoke latest OpenAI o3-mini, I got the following error:
```
API error: 400 Bad Request
{"error":{"message":"Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.","type":"invalid_request_error","param":"max_tokens","code":"unsupported_parameter"}}
```

**To Reproduce**
I'm using the following config
```yaml
providers: 
### OpenAI
  - openai:o3-mini:
      id: openai-o3-mini
      config:
        apiKey: xxxx
```


**System information:**

MacOS

- Promptfoo version:
0.103.18



It's fixed in the next release! I'll cut one for you! 

@mldangelo thx for the update. When is it coming out, and any way to work around it until then?

As a workaround, you can go into node_modules/promptfoo/dist/src/providers/openai.js

Line 372: 

`const isO1Model = this.modelName.startsWith('o1');`

Can be replaced with:

`const isOModel = this.modelName.startsWith('o');`

(make sure to change all of the references for the variable from the isO1Model to isOModel)

This is released in 0.103.19
Repository: promptfoo/promptfoo
Issue: Promptfoo splits big prompts
Root cause: Documentation Deficiency

**Describe the bug**
When using a big prompt, I noticed that it gets split in multiple parts. So, the tests are all invalidated because performed with partial prompts.

**To Reproduce**
Simple. Use > 4k prompt with a model that supports it. You'll see that the prompt is split in multiple parts and the provider called multiple times. In the promptfoo ui that's evident because you'll see multiple prompt columns.

I used this config:

```
description: "My eval"

prompts:
  - file://big-prompt.txt

providers:
  - "watsonx:meta-llama/llama-3-2-11b-vision-instruct"

tests:
  - vars:
      inquiry: "I have a problem with my order"
```

with the attached prompt:

[big-prompt.txt](https://github.com/user-attachments/files/18628727/big-prompt.txt)

**Expected behavior**
Send the entire configured prompt to the provider.

**Screenshots**
Multiple output columns for the split prompt.

![Image](https://github.com/user-attachments/assets/bdc5ac11-125d-4b41-b4ed-3907bdd81906)

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version:

**Additional context**
Add any other context about the problem here.


Hi @mkrudele, the behavior you're seeing is due to promptfoo’s mechanism for splitting prompts by a delimiter. By default, our configuration uses the string `---`, which means that if your prompt text inadvertently includes that string, it will be split into multiple parts.

Here are a few workarounds you can try:

1. **Change the Delimiter:** You can override the default delimiter by setting the environment variable `PROMPTFOO_PROMPT_SEPARATOR` to a high-entropy string (such as a UUID) or any value that does not appear in your prompt. For example:
   ```bash
   export PROMPTFOO_PROMPT_SEPARATOR="unique-delimiter-1234"
   ```
   This approach ensures that your long prompt remains intact without being split unexpectedly.

2. **Use Markdown Files:** If you rename your prompt file from `.txt` to `.md`, it will be processed by our Markdown processor, which does not perform any splitting on the prompt. This works well when you want to ensure that the entire content is sent as a single prompt.

For more details, please refer to our documentation on [Multiple prompts in a single text file](https://www.promptfoo.dev/docs/configuration/parameters/#multiple-prompts-in-a-single-text-file).

I hope this helps! Please let me know if you have any further questions or concerns.

@mldangelo thanks very much for the quick reply. Feel free to close the issue.
Repository: promptfoo/promptfoo
Issue: [BUG] `rubricPrompt` does not render '.json' templates
Root cause: Configuration Error

**Describe the bug**

The `rubricPrompt` throws an error when used with `.json` nunjucks template

**To Reproduce**
I am using custom LLM rubric messages stored in a `.json` file:

```json
[
  {%- set system_prompt -%}
    Evaluate the summary for factual correctness
  {%- endset -%}
  {
    "role": "system",
    "content": {{ system_prompt | dump }}
  },
  {
    "role": "user",
    "content": "Here is the form in YAML format {{ form }}.\n\nThis is the summary generated by the system: {{ output }}.\n\nYour Task: {{ rubric }}.\n\nReply in JSON format as follows: {\"reason\": \"<Analysis of the rubric and the output>\", \"score\": <0.0-1.0>, \"pass\": <true or false>}"
  }
]
```

And use the following `promptfooconfig.yaml`:
```yaml
description: "Bug demo"

prompts:
  - 'summarize this form value in 10 words: {{ form }}'

providers:
  - bedrock:amazon.nova-micro-v1:0

tests:
  - vars:
      form: |
        title: "bug report"
        description: "I found a bug: the system crashed."
    assert:
      - type: llm-rubric
        rubricPrompt: file://rubric.json
        value: Evaluate summary correctness and clarity.
        provider: bedrock:amazon.nova-micro-v1:0
```

As a result, I get an error:
```
SyntaxError: Expected property name or '}' in JSON at position 5
```


Meanwhile, if I just switch the file type of `rubric.json` to `.txt` it works just fine, renders proper JSON string and sends the following to the provider API:

```json
[ {
  "role" : "system",
  "content" : "\n    Evaluate the summary for factual correctness"
}, {
  "role" : "user",
  "content" : "Here is the form in YAML format title: \"bug report\"\ndescription: \"I found a bug: the system crashed.\".\n\nThis is the summary generated by the system: Bug report found system crash..\n\nYour Task: Evaluate summary correctness and factual accuracy..\n\nReply in JSON format as follows: {\"reason\": \"<Analysis of the rubric and the output>\", \"score\": <0.0-1.0>, \"pass\": <true or false>}"
} ]
```


**Expected behavior**

Template rendering works equivalently for `.json` and `.txt` files


**System information:**

- Promptfoo version: 0.103.19
Repository: promptfoo/promptfoo
Issue: feature: Amazon Bedrock
Root cause: Unimplemented Feature Gap

Add Amazon Bedrock as provider for Claude models

Hey,

I have a pull request for Amazon Bedrock. It could not debug and the test the cache(). So, it is not included in this PR.
This PR only supports Claude as right now. I plan working on other models in the future. Any help with the cache() would be appreciated, thanks!

Thanks!  Just landed your PR and will include it in the next release soon.

Released in 0.29.0
Repository: promptfoo/promptfoo
Issue: add `.js` file support to `rubricPrompt`
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently there is limited support for `file://something.js` is `rubricPrompt` parameter: while the `js` file is red and the `module.exports = () => {}` actually gets executed (but only if assertion `value` is an empty string 🤷), the actual value sent to the provider (as verified by adding a breakpoint on [this line](https://github.com/promptfoo/promptfoo/blob/e329701cfeeaf8b1486432f97c69063497d8edb6/src/matchers.ts#L420)) is the raw content of the original js file.

**Describe the solution you'd like**

I believe, adding the support for `js` files would require similar changes as were done in https://github.com/promptfoo/promptfoo/pull/2890, that is adding `valueFromScript` parameter to `handleLlmRubric` function.

Done in #2972
Repository: promptfoo/promptfoo
Issue: Allow `threshold` for `llm-rubric`
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

It is frustrating and quite unreliable (at least in my experience) to ask the grading LLM to decide the correct `pass` value for the given output.

**Context**
To have a greater control over the grading process, I use `rubricPrompt` which concludes with something line the following:

```
##FORMAT##
Respond strictly with a JSON object in this format:
{
  "reason": string, // Detailed analysis of the ##SUMMARY## in one plain text paragraph
  "score": number, // Value between 0.0 and 1.0 representing the accuracy of the ##SUMMARY##
  "pass": boolean  // false if score >= 0.5, true if score < 0.5 
}
```

Nonetheless, I often observe combinations of `{ "score": 0.75, "pass": false }` if the `"reason"` is overly focused on criticising the llm output. 

**Describe the solution you'd like**
Rather than providing if/else logic into rubric instructions and making the LLM to decide whether to output `true` or `false` token (that could be  almost equally probable) its much more convenient to programmatically define a `threshold` value and compare it with the final `score`.

Solved by #2999
Repository: promptfoo/promptfoo
Issue: Local server fails after importing from JSON
Root cause: Validation Gap

**Describe the bug**
Imported json file fails with the following error:
```
/Users/comandeer/.nvm/versions/node/v20.18.0/lib/node_modules/promptfoo/dist/src/util/invariant.js:24
    throw new Error(value);
          ^

Error: Invariant failed: Prompt not found
    at invariant (/Users/comandeer/.nvm/versions/node/v20.18.0/lib/node_modules/promptfoo/dist/src/util/invariant.js:24:11)
    at /Users/comandeer/.nvm/versions/node/v20.18.0/lib/node_modules/promptfoo/dist/src/server/routes/eval.js:170:29
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
```
Also, I see that the paths to prompts in the json are absolute (check the screenshot)


**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Export json file with `promptfoo export`
2. Import json file with `promptfoo import` on another machine
3. Run `promptfoo view`
4. Give thumbs up/down

**Expected behavior**
It's possible to evaluate a dataset after importing a json file

**Screenshots**

<img width="263" alt="Image" src="https://github.com/user-attachments/assets/81be1f9e-ed94-4d62-a4c1-0cc91f7732ba" />


Hey @leoni-q - feel free to email the the json file directly so we can repro on our side
Repository: promptfoo/promptfoo
Issue: Question - Webviewer URL in EvaluateSummaryV3?
Root cause: Documentation Deficiency

Hi team,

Is it currently possibly to receive a web viewer URL when running evaluations via the Node package? I'm using the Node package version `^0.100.0`.

I'm currently doing something like:

```
  const testSuite = {
    prompts: [prompt],
    description,
    providers: [providerFunction],
    tests: tests as TestCase[],
    writeLatestResults: true,
    sharing: true,
  };

  // run evaluation
  const evalData = await promptfoo.evaluate(testSuite, { cache: false });
```

However `evalData` does not contain a web viewer URL like you would get if you used equivalent flags with the promptfoo CLI. Any information you can provide on this functionality and the Node package, as well as the web viewer in general (I know there's a 2 week TTL) would be appreciated thanks!
Repository: promptfoo/promptfoo
Issue: error: promptfoo eval --config folder/promptfooconfig.yaml
Root cause: Validation Gap

Hey,

I am getting the following error when I run `eval` using --config with a directory.

Example:
npx promptfoo@latest eval --config examples/amazon-bedrock/promptfooconfig.yaml

```
promptfoo/dist/src/prompts.js:111
            if (pathOrGlob.startsWith('file://')) {
                           ^

TypeError: Cannot read properties of undefined (reading 'startsWith')
    at promptfoo/dist/src/prompts.js:111:28
    at Array.flatMap (<anonymous>)
    at readPrompts (promptfoo/dist/src/prompts.js:110:45)
    at Command.<anonymous> promptfoo/dist/src/main.js:260:57)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
```

It runs fine when the `promptfooconfig.yaml` is on the root directory, e.g. `npx promptfoo@latest eval`

Thanks for catching this. Fix in #304
Repository: promptfoo/promptfoo
Issue: question: rag evaluation with amazon bedrock
Root cause: Documentation Deficiency

I would like to evaluate rag systems with models from Amazon Bedrock. Got an error when running `npx promptfoo@latest eval` with `providers: [bedrock:anthropic.claude-v2]` using the examples in example/rag-eval. It says, `Error: OpenAI API key is not set. Set the OPENAI_API_KEY environment variable or add apiKey to the provider config.`

1) Which functions need to be implemented to support RAG evaluation? 
2) I found some providers have an ApiEmbeddingProvider. How is the embedding used in this project? AWS supports multiple embedding models via Bedrock. I'd like to understand this more and see if we can add them in the near feature.

[Model-graded assertions](https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/) such as `factuality` or `context-recall`, etc, use OpenAI by default. If you are using Bedrock/Anthropic, you must override the grader to point to Bedrock:

The easiest way to do this for all your test cases is to add the [defaultTest](https://promptfoo.dev/docs/configuration/guide/#default-test-cases) property to your config:

```yaml
defaultTest:
  options:
    provider: bedrock:anthropic.claude-v2
```

You can also do this for individual assertions:

```yaml
  # ...
  assert:
    - type: llm-rubric
      value: ...
      provider: bedrock:anthropic.claude-v2
```

For embeddings (used for e.g. similarity assertions), you can override the provider in a similar way, but we'd have to implement `callEmbeddingApi` in the Bedrock provider.