zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Hyperlinks for full URLs in metadata drop down

**Is your feature request related to a problem? Please describe.**
I have several hyperlinks as metadata fields in my TestCases. I'd like them to be clickable

Recently got easy copy+paste support which is great: https://github.com/promptfoo/promptfoo/issues/3376

**Describe the solution you'd like**
The ability to click hyperlinks in these metadata fields which then opens in a new tab (think jumping to a historical in-app conversation that was created by my private provider API call).

**Describe alternatives you've considered**
Currently copying + pasting in a new tab URL which is enabled by the above feature, but this would be easier.

I understand this may be counter to providing raw metadata fields, as you're adding some additional formatting. But does improve end-user experience, hopping from promptfoo to an application.

This is done in https://github.com/promptfoo/promptfoo/pull/3569 !
Repository: promptfoo/promptfoo
Issue: Column selections not saving in Web View
Root cause: Unimplemented Feature Gap

# Issue
When rerunning evals in promptfoo, the webview automatically updates but hidden column settings are not retained between runs. So when using a large number of variables, I have to re-hide columns that I don't want to display every time I run a new eval. This requires a lot of clicking each time. This was mentioned in below thread, but can't find any other related threads of this issue being reported. 

# Related Background

              @DarioBernardo unfortunately this PR was closed without merging it. Hiding columns and remembering state is on our backlog. We will get to it soon.

_Originally posted by @mldangelo in https://github.com/promptfoo/promptfoo/issues/1069#issuecomment-2383552426_
            

Yea, this is super needed :)

It seems that the column visibility is being persisted, but it's tied to an evalId.

https://github.com/promptfoo/promptfoo/blob/0534451f99e9c324ff3105ee9bd62e6e0c931e19/src/app/src/pages/eval/components/store.ts#L99

Hello, just encountered this myself. Awesome tool, but this is super annoying :) any ETA on the update?

What do you think is the best behavior here?  Global setting to show/hide columns with the same name between different evals?
Repository: promptfoo/promptfoo
Issue: Loading multi prompt with config set in command line doesn't work
Root cause: Validation Gap

In [document](https://promptfoo.dev/docs/guides/gemini-vs-gpt#step-2-set-up-the-prompts), prompt can be set as map like below.

```yaml
prompts:
  prompts/gpt_prompt.json: gpt_prompt
  prompts/gemini_prompt.json: gemini_prompt
```

However, this doesn not work when specify at command line like below.

```console
promptfoo eval -c xxx.yaml
```

And error log is below
```console
/home/runner/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/util.js:153
            prompts.push(...config.prompts);
                                   ^

TypeError: Found non-callable @@iterator
    at /home/runner/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/util.js:153:21
    at Array.forEach (<anonymous>)
    at readConfigs (/home/runner/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/util.js:148:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Command.<anonymous> (/home/runner/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/main.js:237:26)
```

Maybe it is allowed only Array or string type, not Map type [here](https://github.com/promptfoo/promptfoo/blob/main/src/util.ts#L141-L147)

Thanks for flagging this, you're right the problem is that the code doesn't handle map type.

Fix is in #375

Fix published in 0.35.0
Repository: promptfoo/promptfoo
Issue: Import multiple providers from yaml file
Root cause: Validation Gap

**Describe the bug**
Referencing a yaml file with a list of providers throws an error: `Multiple providers found in ./../../providers/providers.yaml. Use loadApiProviders instead of loadApiProvider.`

The yaml config executed with eval: 
```
providers:
  - file://./../../providers/providers.yaml
```
providers.yaml
```
- id: bedrock:us....
  config:
    region: 'us-west-2'
- id: bedrock:us...
  config:
    region: 'us-west-2'
```

**To Reproduce**
1. Create a yaml file with a list of providers.
2. Create a yaml file that refers to the providers 
```
providers:
  - file://./../../providers/providers.yaml
```
3. Run eval, and the error will be thrown


**Expected behavior**
The providers from the file are used normally without throwing an error

**System information:**
Windows, Javascript, Node.js v20.10.0
- Promptfoo version: 0.108.0

**Additional context**
I looked at the code, the added tests from the [pull request](https://github.com/promptfoo/promptfoo/issues/3153) that introduced this feature clearly call the right function 'loadApiProviders', but it seems like eval does not use that function.. instead it calls 'loadApiProvider' even when there is an array, and then throws the error

```
$ promptfoo eval -c promptfoo/test.yaml 
redacted\node_modules\promptfoo\dist\src\providers\index.js:53
            throw new Error(`Multiple providers found in ${filePath}. Use loadApiProviders instead of loadApiProvider.`);
                  ^

Error: Multiple providers found in ./../providers/providers.yaml. Use loadApiProviders instead of loadApiProvider.
    at loadApiProvider (redacted\node_modules\promptfoo\dist\src\providers\index.js:53:19)
    at redacted\node_modules\promptfoo\dist\src\providers\index.js:116:24
    at Array.map (<anonymous>)
    at loadApiProviders (redacted\node_modules\promptfoo\dist\src\providers\index.js:114:42)
    at resolveConfigs (redacted\node_modules\promptfoo\dist\src\util\config\load.js:512:68)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async runEvaluation (redacted\node_modules\promptfoo\dist\src\commands\eval.js:138:55)
    at async doEval (redacted\node_modules\promptfoo\dist\src\commands\eval.js:408:12)
```


Thanks for filing. This issue was fixed in PR #3617 (merged April 8, 2025). The `loadApiProviders` function now properly handles file references containing multiple providers by using the new `loadProvidersFromFile()` helper function.  Verified working with comprehensive test coverage in `test/providers.test.ts` and manual testing. Closing as resolved.
Repository: promptfoo/promptfoo
Issue: factuality and llm-rubric with Anthropic model causing bug when used at same time
Root cause: Interface Contract Mismatch

**Describe the bug**
When `llm-rubric` and `factuality` provider are both on with an ANTHROPIC_API_KEY, the `resp.output` returns an `Object` instead of a `string`, causing eval to fail.

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

eval this config:

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
description: "Testing Claude Sonnet 3.7 with factuality and llm-rubric"

providers:
  - id: anthropic:messages:claude-3-7-sonnet-20250219
    config:
      max_tokens: 1000

prompts:
  - "{{input}}"

tests:
  - vars:
      input: "What is the capital of California?"
    assert:
      - type: factuality
        value: "The capital of California is Sacramento."
      - type: llm-rubric
        value: "Based on your internal knowledge, is the answer right?"

```

If you remove `llm-rubric` the `factuality` providers operates as expected.

**Expected behavior**
Both tests should be run

**Screenshots**

<img width="1281" alt="Image" src="https://github.com/user-attachments/assets/4aca225f-f73c-4f88-b540-c96a45c1f3cd" />

`resp.output` is of type `Object` instead of `string` here: https://github.com/promptfoo/promptfoo/blob/3e6b2eeb3c14c6f825dd2f8546b84b93a511e898/src/matchers.ts#L448

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: 0.109.0

Hello, The issue has been resolved
Repository: promptfoo/promptfoo
Issue: Import from JSON does not work correctly
Root cause: Validation Gap

**Describe the bug**
[The documentation states](https://www.promptfoo.dev/docs/configuration/parameters/#import-from-jsonjsonl) that tests can be defined as json in a format like:

```
[
  { "vars": { "var1": "value1" }, "assert": [], "description": "Test #1" },
  { "vars": { "var1": "value2" }, "assert": [], "description": "Test #2" }
]
```

But the code seems to parse those contents as vars directly. This means any assertions defined in "assert" do not run, resulting in what look like passed tests!

**To Reproduce**

promptfooconfig.yaml
```
prompts:
  - 'Translate the following text to {{target_language}}: {{text}}'

providers:
  # Just repeats the markdown back to us
  - echo

tests: file://cases.json
```

cases.json
```
[
  { "vars": { "text": "Hello", "target_language": "Spanish" }, "assert": [{ "type": "equals", "value": "Hola" }], "description": "Test #1" }
]
```

Running promptfoo eval with the above configuration gives:

```
┌────────────┬────────────┬────────────┬────────────┐
│ assert     │ descripti… │ vars       │ [echo]     │
│            │            │            │ Translate  │
│            │            │            │ the        │
│            │            │            │ following  │
│            │            │            │ text to    │
│            │            │            │ {{target_… │
│            │            │            │ {{text}}   │
├────────────┼────────────┼────────────┼────────────┤
│            │ Test #1    │            │ [PASS]     │
│            │            │            │ Translate  │
│            │            │            │ the        │
│            │            │            │ following  │
│            │            │            │ text to :  │
└────────────┴────────────┴────────────┴────────────┘
==================================================

```

You can see that the vars were not parsed, the asserts did not come through and the prompt variables were not substituted. The test passes!

**Expected behavior**
This test should fail. The assert and vars should be properly parsed.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version:

**Additional context**
Add any other context about the problem here.
Repository: promptfoo/promptfoo
Issue: Specifying multiple __expected column would test only one of them
Root cause: Documentation Deficiency

For example, If I want to test using both icontain & grade, it uses only one of the assertions.

It's also not clear how to specify multiple matches for "icontains-all" using csv

Thanks!

Hi @naoro,

The way to do this would be to use `__expected1`, `__expected2`, etc. as column names, with one assertion per column.  Let me know how that works for you.

Thanks @typpo , it works!
Repository: promptfoo/promptfoo
Issue: promptfoo share unknown option '-y'
Root cause: Validation Gap

**Describe the bug**
Starting v0.108.0, running the command _npx promptfoo share -y_ returns **error: unknown option '-y'**
As per the [documentation](https://www.promptfoo.dev/docs/usage/command-line/#promptfoo-share), this should still be supported

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run 'promptfoo eval -c promptfooconfig-default.yaml --no-cache'
2. Run 'npx promptfoo share -y', which returns **error: unknown option '-y'**

**Expected behavior**
Running 'npx promptfoo share -y' should skip confirmation and creating shareable URL

**Screenshots**
![Image](https://github.com/user-attachments/assets/e58516ba-ef1f-4709-8acd-a70e3af5858c)

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

- Promptfoo version: 0.108.0 onwards

**Additional context**
Add any other context about the problem here.


Hi @marvin-tan, thanks for catching this regression!

This was fixed in v0.109.1 shortly after you reported it. The `-y` flag now works again (though it doesn't actually do anything since shares are private by default now - we kept it for backwards compatibility).

Thanks for the detailed bug report with screenshots - it really helped us fix this quickly! 🙏 If you would like some promptfoo swag or if you have any other thoughts / comments /etc, please feel free to send me an email!
Repository: promptfoo/promptfoo
Issue: Stop sequence validation error for `gpt-4-vision-preview`
Root cause: Validation Gap

Calls to `openai:chat:gpt-4-vision-preview` fail due to a validation error with the `stop` sequence parameter.

Example config:
```
prompts:
  - "Write a tweet about {{topic}}"
  - "Write a very concise, funny tweet about {{topic}}"

providers: [openai:chat:gpt-4-vision-preview]

tests:
  - vars:
      topic: bananas
```

Result:
<img width="1157" alt="image" src="https://github.com/promptfoo/promptfoo/assets/424892/45d2826e-bcb5-4916-94b8-6bafe84f8cc3">

I fixed this in my config by just setting the stop sequence value like so:
```
providers: 
  - id: openai:chat:gpt-4-vision-preview
    config:
      stop: 
        - 

```

Thanks @cameronk.  Just pushed a fix that will be included in the next release.  It looks like `gpt-4-vision-preview` handles undefined `stop` differently from other chat models.

Fix is in 0.34.0
Repository: promptfoo/promptfoo
Issue: `promptfoo share` ignores `sharing.apiBaseUrl` and `sharing.appBaseUrl` in `promptfooconfig.yaml`
Root cause: Configuration Error

Given the following in `promptfooconfig.yaml` (as described in [the docs](https://www.promptfoo.dev/docs/usage/sharing/#advanced-self-hosted-sharing)):

```yaml
sharing:
  apiBaseUrl: https://doesnotexist.google.com
  appBaseUrl: https://doesnotexisteither.google.com
```

and running

```bash
promptfoo share
```

I get

```
» You need to have a cloud account to securely share your results.

1. Please go to https://promptfoo.app to sign up or log in.
2. Follow the instructions at https://promptfoo.app/welcome to login to the command line.
3. Run promptfoo share
```

So it looks like promptfoo is ignoring `sharing.apiBaseUrl` and `sharing.appBaseUrl` from `promptfooconfig.yaml`.

I would expect to get the same error as when doing this:

```bash
PROMPTFOO_REMOTE_API_BASE_URL=https://doesnotexist.google.com PROMPTFOO_REMOTE_APP_BASE_URL=https://doesnotexisteither.google.com 
```

which is:

```
[...]
    errno: -3008,
    code: 'ENOTFOUND',
    syscall: 'getaddrinfo',
    hostname: 'doesnotexist.google.com'
[...]
```

---

I've confirmed that my `promptfooconfig.yaml` is really parsed by adding some nonsense to it:

```yaml
sharing:
  apiBaseUrl: https://doesnotexist.google.com
  appBaseUrl: https://doesnotexisteither.google.com
asdasd
```

and running

```bash
promptfoo share
```

again:

```
YAMLException: can not read a block mapping entry; a multiline key may not be an implicit key (5:1)
```

---

Version used:

```bash
promptfoo --version
```

```
0.109.1
```
Repository: promptfoo/promptfoo
Issue: Unable to pass images to Llama 3.2 on Bedrock
Root cause: Interface Contract Mismatch

**Describe the bug**
When I  implement the Nova template (https://github.com/promptfoo/promptfoo/blob/main/examples/amazon-bedrock/nova_multimodal_prompt.json) for Llama 3.2 on bedrock, I encounter the error: 

`[ERROR] TypeError: message.content.trim is not a function`

**To Reproduce**
Use https://github.com/promptfoo/promptfoo/blob/main/examples/amazon-bedrock/nova_multimodal_prompt.json, but configure the provider to point to a `bedrock:<LLAMA ARN>`.  Pass and reference an `{{image}}` var. 

**Expected behavior**
Base64 image is passed to Llama on bedrock for evaluation

**Screenshots**

**System information:**

```
====================================================================================================
Promptfoo Debug Information
====================================================================================================
{
  "version": "0.109.1",
  "platform": {
    "os": "linux",
    "release": "6.11.0-21-generic",
    "arch": "x64",
    "nodeVersion": "v22.14.0"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
====================================================================================================

- Promptfoo version: 0.109.1

**Additional context**


@decaffeinatedio have you tried running the example located in `examples/amazon-bedrock/promptfooconfig.nova.multimodal.yaml`? 

When I run that example which uses the prompt you reference, I get 
<img width="683" alt="Image" src="https://github.com/user-attachments/assets/19e49070-03c1-40f6-93c7-8c97604722c6" />

@mldangelo Yep! I have no problems running the example against Nova. But when I use the bedrock Llama provider:

```
  - id: bedrock:arn:aws:bedrock:us-east-1:<<REDACTED>>:inference-profile/us.meta.llama3-2-11b-instruct-v1:0
    config: 
      max_gen_len: 256
      temperature: 0.7
      top_p: 0.9
```

I'm encountering that error. When I simplify my prompt to text-only, I get responses back from Llama, so I know it isn't a broader connection/permissions issue. 

@mldangelo took a quick look at source and this doesn't seem straightforward to address. Assume it won't be addressed in the short term?
Repository: promptfoo/promptfoo
Issue: huggingface dataset wrong test case count
Root cause: Algorithmic Error

**Describe the bug**

run the huggingface dataset example. I got 4 test cases with one row dataset. each test case has one choice. 
expected 1 test case with choices column has 4 options.

```yaml
description: 'DeepSeek-R1 vs o1 comparison on MMLU reasoning tasks'

prompts:
  - prompts.py:prompt

providers:
  - bedrock:amazon.nova-lite-v1:0
  # - bedrock:amazon.nova-pro-v1:0
  # - bedrock:us.anthropic.claude-3-5-sonnet-20241022-v2:0
  # - deepseek:deepseek-reasoner

defaultTest:
  assert:
    # Inference should complete within 60 seconds
    - type: latency
      threshold: 60000
    # Check for step-by-step reasoning
    # - type: llm-rubric
      # value: Response must include clear step-by-step reasoning
    # Check that it ends with a clear answer choice
    - type: regex
      value: "Therefore, the answer is {{answer}}."

tests:
  # Load MMLU test sets for reasoning-heavy subjects
  # - huggingface://datasets/cais/mmlu?split=test&subset=abstract_algebra&config=abstract_algebra&limit=10
  # Optionally load other subjects
  # - huggingface://datasets/cais/mmlu?split=test&subset=formal_logic&config=formal_logic&limit=10
  # - huggingface://datasets/cais/mmlu?split=test&subset=high_school_mathematics&config=high_school_mathematics&limit=10
  # - huggingface://datasets/cais/mmlu?split=test&subset=college_mathematics&config=college_mathematics&limit=10
  # - huggingface://datasets/cais/mmlu?split=test&subset=logical_fallacies&config=logical_fallacies&limit=10
  - huggingface://datasets/cais/mmlu?split=test&subset=machine_learning&config=machine_learning&limit=1

evaluateOptions.maxConcurrency: 1
```

**Expected behavior**
1 test case

**Screenshots**

![Image](https://github.com/user-attachments/assets/b4af0c14-b646-436f-80be-165529cb543b)

**System information:**
node 22

- Promptfoo version: prompt 0.109
Repository: promptfoo/promptfoo
Issue: Guardrails server

Hey!

Looking at this file:
https://github.com/promptfoo/promptfoo/blob/main/src/guardrails.ts

I noticed the guardrails service is sending requests to a hosted API. 

Could you clarify:

- Where is the actual guardrails server hosted?

- Is the underlying model or service publicly available?

- Is there an open-source implementation for running this locally?

thanks!

@typpo
Repository: promptfoo/promptfoo
Issue: Reading prompts, providers and assertions from environment variables
Root cause: Unimplemented Feature Gap

Hello,

Is it possible to read paths from an environment variable in the promptfooconfig.yaml?
As you can see in the following config.yaml, I read prompt, provider, assertions from path "../../test". I want to read this path from an environment variable and then concatenate with the file name like '{{env.TEST_PATH}}'+'\prompt.txt'. Or I'm also open to any other suggestions. 


```yaml
description: 'test'

prompts: 
  - ../../test/prompt.txt

providers:
  - ../../test/custom_provider.js

defaultTest:
  options:
    provider: file://../../../test/model_provider.yaml
tests:

  - vars:
      question: What is the capital of France?
    assert:
      - type: python
        value: file://../../test/custom_assert.py

```

Thank you
Repository: promptfoo/promptfoo
Issue: Question: What models actually support the perplexity metric?
Root cause: Documentation Deficiency

Recently discovered that the perplexity metric has been added into the test assertions which is great and I wanted to try it out however, when I run the perplexity metric on the azure gpt 4 model, I get an error stating that the provider doesn't support logProbs. I believe gpt-4 should return logProbs - gpt-35-turbo doesn't which expectedly triggered an error. I was just wondering if you know what models work with this metric. 

I've tested this with the 1106 models via OpenAI as it is a recent addition to their API.  I'm not sure Azure supports logprobs - maybe with their 1106 model versions?  However, promptfoo's Azure provider does not support this yet.

#376 adds support for logprobs/perplexity for Azure OpenAI chat completions.   This doesn't guarantee it'll work for all Azure models, but it means it will work when logprobs are available (probably for the 1106+ version of GPTs)

A change published in 0.35.0 adds `perplexity` support for Azure OpenAI models, if the models themselves support `logprobs`
Repository: promptfoo/promptfoo
Issue: Is there a way to assign a prompt file to a provider
Root cause: Documentation Deficiency

I have 2 prompts of different structures and 2 custom providers to handle the different structures. I need to compare the results of the 2 custom providers, so if I were to add both custom prompts in the following format, I would end with a 2x2 matrix of results.

prompts: [prompt1.json, prompt2.json]
providers: [customProvider1.js, customProvider2.js]
tests: vars.csv

Instead, would it be possible to assign the prompt via the providers? E.g.:

providers:
    - id : customProvider1.js
            config:
                   prompt: prompt1.json
  - id : customProvider2.js
            config:
                   prompt: prompt2.json


Yes, this is possible.  I've updated the documentation to describe this here: https://promptfoo.dev/docs/configuration/parameters#different-prompts-per-model
Repository: promptfoo/promptfoo
Issue: Prompt and variable files are not always relative to configuration directory
Root cause: Configuration Error

When a prompt is declared like:

```
prompts:
  file://prompts/java.json
```
It's relative to the configuration directory, however when following syntax is used:
```
prompts:
  prompts/java.json: java
```
The path is not treated as relative to configuration path.


On the other hand when variable is loaded from file using `file://` syntax:
```
tests:
  - description: Cached Property
    vars:
      prompt: file://prompts/1_cachedproperty.txt
```
that file path is not treated as relative to configuration file.

This inconsistent behavior makes it difficult to tun promptfoo with multiple configuration file.

Thanks for the report!  Fix is in #375, which should consistently resolve paths relative to the configuration path.

Fix published in 0.35.0