zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Issue in promptfoo UI dataset page.
Root cause: Algorithmic Error

**Describe the bug**
I am facing issue related to mapping of evals, testcase and prompts with specific datasets in dataset table generated in the dataset page of Promptfoo UI.

By default the table in UI shows 10 rows of datasets every mappings seems fine in first page i.e. first 10 rows, after 10th row a new page is formed for any new entry and in that new page whenever I click on specific datasets to see the evals and test case related to that dataset it shows wrong mapping for every value in the pop-up.

I am using `ghcr.io/promptfoo/promptfoo:main ` this image and running a simple docker container and mounted volume for persisting the evals and everything.

For 11th row when I open any dataset in the pop-up it shows the data related to 1st row for 12th row it shows data related to second row and this continues.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

Try to have more than 10 different datasets and when you click on the 11th dataset to see its related test cases and evals you will encounter the errori in the pop up.


**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version:
ghcr.io/promptfoo/promptfoo:main
Repository: promptfoo/promptfoo
Issue: Support using provider credentials in JS options instead of environment
Root cause: Unimplemented Feature Gap

**Describe the bug**
I am trying to run an evaluation using model-assisted assertions, and I want to specify the grader via the config I pass into `promptfoo.evaluate()`.  I pass in my provider id, along with a config in the form of `{apiHost: xxxx, apiKey: xxxx}`.  When I run the evaluation it fails with a message telling me to set env vars for the host and key.  Since I'm using the npm package, I would like to be able to specify these credentials in javascript, and not manipulate my environment.

**To Reproduce** 
```javascript
promptfoo.evaluate({
providers: <my providers>
prompts: <my prompts>
tests: <my tests>
defaultTest: {
            options: {
              provider: {
                id: <my azure provider id>,
                config: {
                  apiHost: <my provider host>,
                  apiKey: <my provider key>,
                }
              }
            }
})
```

- Promptfoo version: 0.97.0



Hey @bennibbelink, this should be resolved by https://github.com/promptfoo/promptfoo/pull/2664 

Hey @bennibbelink,

This was fixed back in January via #2664. You can now pass provider credentials directly in your JS config without needing environment variables:

```javascript
promptfoo.evaluate({
  providers: [...],
  prompts: [...],
  tests: [...],
  defaultTest: {
    options: {
      provider: {
        id: 'azure:chat:your-deployment',
        config: {
          apiHost: 'your-host.openai.azure.com',
          apiKey: 'your-api-key'
        }
      }
    }
  }
});
```

The fix has been available since v0.97.x. If you're still running into issues on a recent version, feel free to reopen and we can dig in further.

Thanks for reporting this!
Repository: promptfoo/promptfoo
Issue: Unable to utilize Azure OpenAI provider for similarity assert
Root cause: Documentation Deficiency

I have configured Azure OpenAI Provider with AZURE_OPENAI_API_KEY, yet receiving 'OPENAI_API_KEY Must Be Set for similarity comparison' error when attempting similar in assert

I am using the Azure OpenAI provider, so why am I encountering an error to set the OPENAI_API_KEY?

Perhaps you need to set the provider on the test or assertion object:

```yaml
assert:
  - type: similar
    value: 'The expected output'
    provider: azureopenai:embedding:text-embedding-ada-002
```

It worked! Thank you @typpo.
Repository: promptfoo/promptfoo
Issue: Unable to utilize Azure OpenAI completion model
Root cause: Documentation Deficiency

I'm attempting to utilize the GPT-3.5-turbo API from Azure OpenAI, but I'm encountering the following error:

[FAIL(0.49)] API response error: TypeError: Cannot read properties of undefined (reading '0'): {"error":{"code":"BadRequest","message":"logprobs, best_of and echo parameters are not available on gpt-35-turbo model. Please remove the parameter and try again. For more details, see https://go.microsoft.com/fwlink/?linkid=2227346."}}

Is there a way I can handle this?

Hi @ishikagoel5628, would you be able to provide your config?  That will help me troubleshoot.  At a minimum, would need to see the `providers` block.

prompts: [prompts.txt]
providers:
  - id: azureopenai:completion:gpt-35-turbo
    config:
      temperature: 0.0

@typpo I am using completion API from Azure OpenAI

Ah, I see.  GPT 3.5 Turbo is a chat completion model.  You should specify `azureopenai:chat:gpt-35-turbo`
Repository: promptfoo/promptfoo
Issue: Support for gpt-4o-audio-preview
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
There is no way to upload audio on the audio-preview models using promptfoo

**Describe the solution you'd like**
Provide a way to upload an audio file to gpt-4o-audio-preview

**Describe alternatives you've considered**
Tried using HTTP provider and building a custom provider. It's hacky and doesn't quite work with caching.

**Additional context**
GPT-4o-audio can be called by base64 encoding the audio file:

```bash
curl "https://api.openai.com/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -d '{
      "model": "gpt-4o-audio-preview",
      "modalities": ["text", "audio"],
      "audio": { "voice": "alloy", "format": "wav" },
      "messages": [
        {
          "role": "user",
          "content": [
            { "type": "text", "text": "What is in this recording?" },
            { 
              "type": "input_audio", 
              "input_audio": { 
                "data": "<base64 bytes here>", 
                "format": "wav" 
              }
            }
          ]
        }
      ]
    }'
```

More here - https://platform.openai.com/docs/guides/audio

It's in process in this branch! https://github.com/promptfoo/promptfoo/compare/main...feat/openai-audio Please check back in ~ 1 week (unless you'd like to take it over). 

Awesome @mldangelo!

Just found that this also works:

prompt.json
```yaml
[
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "{{question}}"
        },
        {
          "type": "input_audio",
          "input_audio": {
            "data": "{{audio_in_base64}}",
            "format": "wav"
          }
        }
      ]
    }
  ]
```

test.yaml
```yaml
tests:
  - vars:
      question: 'What do you hear? Transcribe exactly.'
      audio: file://./preamble.b64
```

However, it's a pain to convert each audio file into base64 manually. Your branch looks nice with all the audio related HTML features as well!

Looking forward to it!

Hey @mldangelo want any assistance or help on the branch?

This is live now https://github.com/promptfoo/promptfoo/tree/fa2ea2714bb9040e8fae3204419496c42f347764/examples/openai-audio
Repository: promptfoo/promptfoo
Issue: Unit tests are not structured to work when invoked via `jest --randomize`
GitHub labels: Open Source

**Describe the bug**
Upon running `jest --randomize` - which is the recommended way to test the application in the [contributing](https://www.promptfoo.dev/docs/contributing/#writing-tests) doc - many tests fail.

**To Reproduce**
1. Make sure you are on latest `main` with dependencies installed 
2. Invoke `jest --randomize`
3. Observe that not all of the tests pass
![image](https://github.com/user-attachments/assets/ef129a20-52ec-47c8-bd12-a5aa34d5c0f6)

**Expected behavior**
Tests should be structured in a way that works with randomize. Alternatively, if this is no longer the desired path, the contributing doc should be updated to reflect this.

**Screenshots**
Test run with --randomize:
![image](https://github.com/user-attachments/assets/23e474fa-f092-4eb2-b9b0-9d3f4736a4db)

Test run without --randomize:
![image](https://github.com/user-attachments/assets/67a4b78c-e4ff-4f58-aee8-8fa7367dbc75)


**System information:**

- Promptfoo version: Running commit `3b02a9d13c92c7a1db095269a544d8cc3c7f9391`

**Additional context**
Current contributing doc [reference ](https://www.promptfoo.dev/docs/contributing/#writing-tests) - 
![image](https://github.com/user-attachments/assets/955fecba-8a9c-4f77-9323-c3e956130f5a)



Hi @eharris128, you’re absolutely right that we could do a better job cleaning up the tests to ensure they work consistently with jest --randomize. It's on my backlog! After we'll enable jest --randomize in the CI. 

where shall i get started?

Hi @aUsABuisnessman, thanks for offering to help out! The best way to start is by running our tests locally with:

```bash
npx jest --randomize
```

When you find a failing test, you can isolate it using the --seed parameter. For instance, I noticed that:

```bash
npx jest --randomize test/providers/shared.test.ts --seed 1321157748
```

consistently fails.

Feel free to tackle that one or another failing test of your choice. Please feel free to ask if you have any questions.
Repository: promptfoo/promptfoo
Issue: Passing Function Call Schemas as Variables in OpenAI Function Configurations
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**  
Currently, function call schemas cannot be passed dynamically as variables when configuring OpenAI function calls. This limitation makes it difficult to reuse or dynamically generate schemas in a clean and modular manner. For example, attempting to define a schema in a `vars` block and reference it in a function configuration results in an output of `"parameters":{"[object Object]":null}` instead of the expected schema.

This behavior is frustrating as it reduces flexibility and increases redundancy in defining schemas, particularly for use cases where schemas might need to be generated or reused across multiple tools or scenarios.

---

**Describe the solution you'd like**  
I would like the ability to pass schemas as variables in function call configurations. For instance, the following configuration should work as expected:

```yaml
providers:
  - id: openai:chat:gpt-4o-audio-preview
    config:
      tools: [
        {
          "type": "function",
          "function": {
            "name": "transcribe_audio",
            "description": "Transcribe the audio",
            "parameters": {{ schema | dump | safe }}
          }
        }
      ]
      tool_choice: 'auto'

tests:
  - vars:
      schema: {
        "type": "object",
        "properties": {
          "transcription": {
            "type": "string",
            "description": "The transcription of the audio"
          }
        },
        "required": ["transcription"]
      }
```

The schema variable should correctly resolve and be passed into the `parameters` field without breaking.

---

**Describe alternatives you’ve considered**  
1. **Hardcoding schemas directly into function configurations**: While this works, it results in code duplication and makes schema management cumbersome, especially for large projects with multiple tools or similar schema structures.  
2. **Preprocessing the configuration**: Writing custom scripts to inject schemas before runtime, but this adds unnecessary complexity and reduces the maintainability of configurations.  

Neither of these alternatives is as clean or efficient as supporting schema variables natively.

---

**Additional context**  
Here’s an example of the error observed when attempting to pass a schema variable:  

```yaml
"parameters": {
  "[object Object]": null
}
```

Allowing schemas to be passed as variables would enable dynamic and modular schema configurations, improving development workflows and reducing redundancy. This feature would be particularly useful for scenarios requiring reusable or dynamically generated function call schemas.  

Thank you for considering this feature! Let me know if further clarification or examples are needed.
Repository: promptfoo/promptfoo
Issue: Allow multiple output formats simultaneously with `outputPath`.
Root cause: Unimplemented Feature Gap

Currently, `outputPath` in `promptfooconfig.yaml` supports only one argument of type string.
In addition, `outputPath` should support type `List[str]` to allow outputting multiple formats from a single run simultaneously.
Currently, supported extensions are `csv, txt, json, yaml, yml, html`.
An additional extension should be `viewer`.
Where `viewer` writes to the web or console viewer.
Meanwhile, `html` should produce an HTML file at the provided path. 

Thanks for the PR!  Closing this out as #229 is in the latest release.
Repository: promptfoo/promptfoo
Issue: promptfoo view not able to connect to server via browser on 0.25.0
Root cause: Validation Gap

Hello,

I am currently using VSCODE dev container to install promptfoo. When I do a promptfoo view on 0.25.0, I run into the following on the browser:

Application error: a client-side exception has occurred (see the browser console for more information).

This is what I see on the browser console:

596-297f7ff4a0436e87.js:1 TypeError: Cannot read properties of null (reading 'toFixed')
    at eo (891-0286b3010dc452ac.js:1:7166)
    at ed (891-0286b3010dc452ac.js:1:8541)
    at ri (fd9d1056-fba4b53a2f01213b.js:9:38931)
    at im (fd9d1056-fba4b53a2f01213b.js:9:111754)
    at oM (fd9d1056-fba4b53a2f01213b.js:9:90098)
    at fd9d1056-fba4b53a2f01213b.js:9:89920
    at oT (fd9d1056-fba4b53a2f01213b.js:9:89927)
    at ov (fd9d1056-fba4b53a2f01213b.js:9:87565)
    at oy (fd9d1056-fba4b53a2f01213b.js:9:86427)
    at E (596-297f7ff4a0436e87.js:25:1389)

Have you run any evals, and are you able to share the config of that eval and/or the contents of `~/.promptfoo/output/latest.json`?

Release 0.25.1 contains a possible fix.
Repository: promptfoo/promptfoo
Issue: Nested Nunjucks variables aren't rendered when config variable points to a file
Root cause: Unimplemented Feature Gap

**Describe the bug**
Nested Nunjucks variables, such as `{{ foo.bar }}` aren't rendered when rendering the content of a variable in prompt that points to a file.

**To Reproduce**
See the shared promptfoo: https://app.promptfoo.dev/eval/f:701acdb5-d5f6-48ec-bad8-647780a25d79/

Config: 
```yaml
prompts: Output the following "{{ prompttest }}"
providers:
  - openai:gpt-4o-mini
tests:
  - description: 'Test translation to French'
    vars:
      test1: test1-content
      test2: 
        sub: hello-content
      prompttest: file://test.txt
    assert:
      - type: contains
        value: hello-content
```

Test.txt contains the following line:

```
"Hey, this is {{ test1 }}, this is {{ test2 }} and this is {{ test2.sub }}"
```

**Expected behavior**
`{{ test2.sub }}` is rendered as `hello-content`

**Actual behaviour**

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

```json
{
  "version": "0.99.1",
  "platform": {
    "os": "darwin",
    "release": "23.3.0",
    "arch": "arm64",
    "nodeVersion": "v20.17.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```

Currently, nested objects in vars are not officially supported, which is why you may see a schema warning in your editor when defining test2 (if your editor is configured with a schema validator).

<img width="679" alt="Screenshot 2024-12-03 at 11 40 32 AM" src="https://github.com/user-attachments/assets/9bdf891e-d4dc-4dbb-8a17-075792c40e70">


To work around this, I recommend flattening your vars structure. For example, `{ key1: { key2: ... } }` -> `{ 'key1.key2': ... }`.

If you’re interested in adding support for nested vars, we’d love your help! The main piece of code to explore is [here](https://github.com/promptfoo/promptfoo/blob/main/src/evaluatorHelpers.ts#L78).

Let us know if you have any further questions or need guidance!

Thanks @mldangelo! The use case I'm trying to solve is: the AI agent I'm working with receives a `{{ context }}` Nunjucks variable with all the required data as a nested object. I can't control the AI agent's behaviour wrt rendering, and I want to use promptfoo for testing the prompt. Is there a workaround I'm not seeing? I tried using transformVars, but it doesn't support objects as well, it seems.

Ah! Makes sense. In that case I recommend using a prompt function https://www.promptfoo.dev/docs/configuration/parameters/#prompt-functions in python or javascript. You can do the vars substitution right in the function without nunjucks.
Repository: promptfoo/promptfoo
Issue: issue: yaml config with npx promptfoo eval
Root cause: Environment Incompatibility

Trying to run examples in the lib that use yaml config file on 0.8.3 using

`OPENAI_API_KEY=[secret] npx promptfoo eval`
or
`OPENAI_API_KEY=[secret] npx promptfoo eval -c promptfooconfig.yaml`

results in
```bash
error: required option '-p, --prompts <paths...>' not specified
```

Only the `js-config` example, which uses a .js config, still works.
```bash
Loaded default config from promptfooconfig.js
```

It seems to be only an issue with the codebase though, not when using it as a package on another project, so maybe I'm just doing something wrong here.

I can't repro on a fresh repository.  What version are you running?  `npx promptfoo --version`

Alternatively, ensure that you are running the latest with `npx promptfoo@latest eval`

If that doesn't work, it would be good to see the contents of promptfooconfig.yaml.

Let me clarify: The intention if using `npx promptfoo` was to run the `bin`: `promptfoo` from the `package.json` for development, not the latest version from the npm registry.

Beforehand I used `pnpm prepare` and `pnpm build:watch`, but to no avail to get it to run.

I eventually used `pnpm local` (aka `ts-node --esm src/main.ts`), but I had to change all the explicit imports from `.js`, ( i.e. `import { evaluate } from './evaluator.js';` to `import { evaluate } from './evaluator';` ) since they are ts file in their native state.

At least this finally worked :).

I now run `ts-node --esm  ../../src/main.ts` in order to run from the dev version from the /examples directory (and the new example I work on).

Regarding native ts-node comp, I pushed a PR:
https://github.com/typpo/promptfoo/pull/25/
Repository: promptfoo/promptfoo
Issue: web view version not getting updated a lot of times
Root cause: Resource Mishandling

web view version not getting updated a lot of times and in order to that I have to again and again first run it in cli and then in web to see the updated version

Does refreshing the webpage work?  Instead of re-running `promptfoo view`.  Have you left the previous `promptfoo view` command running?

Hey @typpo yeah I tried that too!,
will try to create a video  of it and share later on!

@KlausMikhaelson lmk if you have a repro for this!
Repository: promptfoo/promptfoo
Issue: Getting apiKey not present error in test cases when trying to use model-graded-closedqa or llm-rubric
Root cause: Documentation Deficiency

below is just a sample snippet of the test that I am trying to run, I do have the apiKey in my config under  providers
like 
```
providers:
- openai:gpt-3.5-turbo-0613:
    config:
      apiKey: HAVE_ADDED_KEY_HERE
      max_new_tokens: 200
      temperature: 0
    prompts: openai_prompt
- replicate:meta/llama-2-7b-chat:8e6975e5ed6174911a6ff3d60540dfd4844201974602551e10e9e87ab143d81e:
    config:
      apiKey: HAVE_ADDED_KEY_HERE
      max_new_tokens: 200
      temperature: 0
    prompts: firework_prompt
```
 but still getting the below error

![image](https://github.com/promptfoo/promptfoo/assets/100528412/d0a4330f-d757-4e61-bfd4-1921aec717cf)


 - vars:
      message: What's the weather in New York right now?
    assert:
      - type: model-graded-closedqa
        value: does not claim to know the current weather in New York

Hi @KlausMikhaelson, this is happening because the assertions use a different provider than the ones you have configured up top.

Try explicitly setting the provider in the assertion:

```yaml
    assert:
      - type: llm-rubric
        value: does not claim to know the current weather in new y
ork
        provider:
          id: openai:gpt-4
          config:
            apiKey: abc123
```

You may also configure this under `defaultTest.options` so you don't have to override the default grader for every assertion:

```yaml
defaultTest:
  options:
    provider:
      id: openai:gpt-4
      config:
        apiKey: abc123
```

that worked thank you! @typpo
Repository: promptfoo/promptfoo
Issue: AWS Bedrock anthropic.claude-3-5-sonnet-20241022-v2:0
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
using the provider Bedrock im unable to use the latest Sonnet version 'bedrock:anthropic.claude-3-5-sonnet-20241022-v2:0'  the older version 'bedrock:anthropic.claude-3-5-sonnet-20240620-v1:0' works.  I have made sure to request access to the new version, which is approved on the account. 

There are two configurations i have tried, the diff is the id value for each:

**1).**
``` yaml
- id: "bedrock:anthropic.claude-3-5-sonnet-20241022-v2:0"
    config:
      region: 'us-east-1'
      temperature: 0.7
      max_tokens: 256
```

error: 
```
[FAIL] API call error: ValidationException: Invocation of model ID anthropic.claude-3-5-sonnet-20241022-v2:0 with on-demand throughput isn’t supported. Retry your request with the ID or ARN of an inference profile that contains   │
│ this model.
```


**2).**
``` yaml
- id: "bedrock:us.anthropic.claude-3-5-sonnet-20241022-v2:0"
    config:
      region: 'us-east-1'
      temperature: 0.7
      max_tokens: 256
```

error:
```
[FAIL] Error: Unknown Amazon Bedrock model: us.anthropic.claude-3-5-sonnet-20241022-v2:0                                                                                                                                              │
│ Error: Unknown Amazon Bedrock model: us.anthropic.claude-3-5-sonnet-20241022-v2:0                                                                                                                                                     │
│     at getHandlerForModel (/opt/homebrew/lib/node_modules/promptfoo/dist/src/provi..
```


**Describe the solution you'd like**
I'd like to be able to use promptfoo for bedrock:anthropic.claude-3-5-sonnet-20241022-v2:0.  If anyone else has been successful, i'd appreciate some guidance.

**Describe alternatives you've considered**
No alternatives i can think of, i want to use the latest Sonnet to see if there's improvement with output

**Additional context**
I've also tested in a python app using langchain_aws with the model_id of 'us.anthropic.claude-3-5-sonnet-20241022-v2:0' which works.


@sam86ni Thanks for reporting this issue! This should now be fixed in version [0.100.6](https://github.com/promptfoo/promptfoo/releases/tag/0.100.6). 

The latest release also adds proper support for regional model identifiers in Bedrock. I've also updated our [AWS Bedrock documentation](https://www.promptfoo.dev/docs/providers/aws-bedrock/) to include how to properly format regional model identifiers and troubleshooting steps if a model isn't enabled in your region.

Please upgrade to v0.100.6 and let us know if you run into any other issues. Thanks for helping us improve promptfoo! 🙌