zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: not working in windows 10
Root cause: Environment Incompatibility

Edición	Windows 10 Pro
Versión	22H2
Instalado el	‎4/‎08/‎2023
Compilación del sistema operativo	19045.5011
Experiencia	Windows Feature Experience Pack 1000.19060.1000.0

![SNAG-0194](https://github.com/user-attachments/assets/28dbebea-8b35-4e8d-bb49-cbbea5f93eeb)

promptfoo-win-x64

how can solve this?

Hi @johnfelipe, Unfortunately, our binaries aren’t fully supported due to some build process bugs. I’d recommend installing via npm. You can find installation instructions in our docs: https://www.promptfoo.dev/docs/installation/. Please let us know if you run into any more issues.

i close this, because i install in ubuntu, but pls solve that build process bugs, asap, for windows users in community
Repository: promptfoo/promptfoo
Issue: persistent evaluation ids on export + import
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I've been using a database to store promptfoo evals - love the JSON output for `promptfoo eval` with -o (and also `promptfoo export`)
I've run into an issue where the evaluation id changes on `promptfoo import`. (confirmed by the output of promptfoo import - `Eval with ID {} has been successfully imported`). Since I'm storing and using these in a database, I need the ids to stay the same even through imports and exports.

**Describe the solution you'd like**
I've done some digging and see that v2 of the import function had persistent evaluation ids - a flag (`promptfoo import {} --persistent`) or an environment variable to keep the ids persistent would be great.


**Describe alternatives you've considered**
for now, I'm changing the version of the json to "v2" instead of "v3" since that will keep the ids persistent.

Thanks!

Thanks for flagging!  #2112 should fix.

Released in 0.96.0!
Repository: promptfoo/promptfoo
Issue: Prompt template variable substitution issue
Root cause: Documentation Deficiency

Summary
Prompt template variable substitution incorrectly inserts the entire context dictionary instead of the variable value.

## Current Behavior
The system inserts the entire context dictionary:
# {'vars': {'raw_text': 'Extract relevant quotes from the document and put it within citation tags.', '': ''}, 'provider': {'id': 'google:gemini-1.5-flash-002', 'label': ''}, 'config': {}}
## Expected Behavior
The system should insert only the raw text value:
# Extract relevant quotes from the document and put it within citation tags.
## Workaround
Using the full dictionary path in the template works as expected:
# {raw_text["vars"]["raw_text"]}

promptfoo version: 0.95.0


Can you provide a full working example of a config that doesn't work correctly? 

[Text-Improver.zip](https://github.com/user-attachments/files/17686163/Text-Improver.zip)

The configuration is included in the zip file. Only the personal information in the prompt has been removed, and everything else remains unchanged.

Promptfoo is an exceptional tool for prompt engineering. Thank you for your outstanding work.


This looks like expected behavior - the python prompt function takes a `context` dict that includes `vars`.  So I'd rename your argument to `context` and reference it like `context['vars']['raw_text']`.

https://www.promptfoo.dev/docs/configuration/parameters/#examples

Is there anyplace where you saw an example to the contrary?  We can update examples/docs accordingly.

Thank you for your patience in answering my questions. Let's use the new method. I recall that in the previous version, test case variables were directly passed. On October 19th, when using promptfoo for prompt evaluation, it was like that. In November, I noticed it had changed to passing the context dictionary, possibly due to a version update.

Ah that's probably it. We made the change in order to make it consistent
with other functions. Sorry for the trouble!

On Sat, Nov 9, 2024, 6:00 PM Stanley18520 ***@***.***> wrote:

> Thank you for your patience in answering my questions. Let's use the new
> method. I recall that in the previous version, test case variables were
> directly passed. On October 19th, when using promptfoo for prompt
> evaluation, it was like that. In November, I noticed it had changed to
> passing the context dictionary, possibly due to a version update.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/2110#issuecomment-2466545531>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJVYPXYW7SWI7KEZ4ADZ7245RAVCNFSM6AAAAABROSVZU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINRWGU2DKNJTGE>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
Repository: promptfoo/promptfoo
Issue: assertion support .cjs file
Root cause: Unimplemented Feature Gap

I have an environment that can't execute .js file but .cjs file. Is it reasonable to support .cjs or .mjs files?

Hi @zhlmmc, just to clarify, are you referring to `javascript`-type assertions that begin with `file://`?

> Hi @zhlmmc, just to clarify, are you referring to `javascript`-type assertions that begin with `file://`?
Yes,
![image](https://github.com/promptfoo/promptfoo/assets/1726527/4a5203e9-0535-4372-a298-41c70fb4dd56)


That should be easy to add, at least for the cjs case.  Opened PR #214 which will be in the next release
Repository: promptfoo/promptfoo
Issue: Broken Anthropic System Message
Root cause: Algorithmic Error

**Describe the bug**

I use JS functions to return prompts. Here's how a test file looks:

```
description: "Audit"

providers:
  - id: "anthropic:messages:claude-3-5-haiku-20241022"
    config:
      temperature: 0
      max_tokens: 4096

prompts:
  - ../build/prompts/index.js:audit_criteria

tests: GOOGLE_SHEETS_URL
```

In the latest version (v0.96.0) I get this error on prompts that use the system message:
```
API call error: messages: Unexpected role "system". The Messages API accepts a top-level `system` parameter, not "system" as an input message role., status 400, type invalid_request_error
```

After reviewing some code, I think this change in `src/providers/anthropic.ts` broke it:
https://github.com/promptfoo/promptfoo/commit/3c29b39ff26c6a9e77c1a0da344960a81fc8b59c#diff-afce651d866a1cd4afb1936e9e431f2aab127727a60530b1774dac7348961c65

~~It also seems as though anthropic's prefill messages aren't working anymore. I could not find any related code to such.~~ This could just be a model issue. Prefill works with other models. Can ignore.

+1

Hi @kaulsh and @aandis, sorry for breaking this. We just issued a release in version 0.96.1 that includes a fix for handling system messages in parseMessages. Please try updating to the latest version and let me know if you encounter any other issues! 

@mldangelo No problem and thanks for the quick patch. Everything works as expected now :+1:
Repository: promptfoo/promptfoo
Issue: Continued persistent evalID issues
Root cause: Algorithmic Error

              Thanks for flagging!  #2112 should fix.

_Originally posted by @typpo in https://github.com/promptfoo/promptfoo/issues/2108#issuecomment-2466287630_

Hey there! thanks for the support. Unfortunately, I don't think the changed code fixes the persistent evalId issue- 

Here is some code that shows the evalId changes on import and export - 

```bash

#!/bin/bash

# Define paths
CONFIG_PATH=".../promptfooconfig.yaml"
JSON_PATH=".../output.json"
DB_PATH=".../promptfoo.db"

# Print promptfoo version
promptfoo --version

# Clear any previously created promptfoo data
rm -f "$DB_PATH"

# Do eval and get eval id
promptfoo eval -c "$CONFIG_PATH" -o "$JSON_PATH" --no-table --no-progress-bar

# Extract eval ID using jq
EVAL_ID=$(jq -r '.evalId' "$JSON_PATH")
echo "created evalId: $EVAL_ID"

# Test import duplicate eval (maybe warning or error raised?)
promptfoo import "$JSON_PATH"

# Export eval (check eval id)
promptfoo export "$EVAL_ID" -o "$JSON_PATH"

# Clear local db
rm -f "$DB_PATH"

# Import eval (should be same as above, no duplicate)
promptfoo import "$JSON_PATH"
```

output:

```
0.96.0
created evalId: eval-0fe-2024-11-11T17:41:14
Eval with ID eval-VfC-2024-11-11T17:41:16 has been successfully imported.
Eval with ID eval-0fe-2024-11-11T17:41:14 has been successfully exported to ...
Eval with ID eval-ohZ-2024-11-11T17:41:17 has been successfully imported.
```
Repository: promptfoo/promptfoo
Issue: Erroneous warning that `$ref` is invalid in the providers section of promptfooconfig
Root cause: Validation Gap

**Describe the bug**
In 0.95 and earlier, `$ref` worked great to retrieve the model(s) you wanted to use, e.g.:

```yaml
providers:
  - anthropic:messages:claude-3-5-sonnet-20240620
  - $ref: promptfoo/defaultParams.yaml#/model
  - openai:gpt-4o-mini
```

Now in 0.96, I get a yellow warning when I run an eval:

```
Invalid configuration file promptfoo/magic_copy_paste_text_to_cards_advanced/all_card_types/promptfooconfig.yaml:
Validation error: Unrecognized key(s) in object: '$ref' at "providers[1]"
```

Despite this “validation error,” the eval still runs as expected, with the correct value retrieved from the ref.

**To Reproduce**

Create a promptfooconfig.yaml, e.g.:

```yaml
description: concept_panel_define
prompts:
  - test {{text}}
providers:
  - $ref: promptfoo/defaultParams.yaml#/model
temperature: 1

tests:
  - vars:
      text: bla bla bla
```

Create the `defaultParams.yaml` file referenced, with some value for `model` that's a valid model, e.g.:

```yaml
model: openai:gpt-4o-mini
```

Run `promptfoo eval` on the config.

**Expected behavior**
No warning appears, since the syntax is valid and the eval runs normally.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information (please complete the following information):**

- Promptfoo version: 0.96


Thanks for flagging.  Will get a fix in.  This shouldn't affect the actual run of the eval 👍 

Hi @sobjornstad, thanks for filing this issue! We addressed this in the latest release.  Please update to 0.96.1 and let us know if you encounter any further issues
Repository: promptfoo/promptfoo
Issue: Pass assert result down to next assert
Root cause: Unimplemented Feature Gap

First of all I just want to say, that I have used promptfoo for two projects so far and really enjoy using it, especially all the possibilities for customization.


I was just wondering if it is possible to pass down the result of a previous assert or have some sort of conditional asserts, as I could not find anything regarding this in the documentation or existing issues.

My use case is that i have asserts defined like this:

```yaml
assert:
      - type: is-sql
        value: ...
      - type: python
        value: file://metrics/execution_accuracy.py
      - type: python
        value: file://metrics/...
 ```
 
Where it might not make sense to execute my custom python metric if the output isn't SQL in the first place and passing down the result of the `is-sql` assert saves me the time to check for sql validity before computing my custom metrics and just return a failure immediately.
 
Maybe this is intended and having asserts be independent of each other is actually desirable. If that is the case feel free to close this issue and I will just double-check sql validity for each assert.
Repository: promptfoo/promptfoo
Issue: API errors cause CLI progress bars to redraw
Root cause: Architectural Constraint

**Describe the bug**

Basically if there is an API error (in the provided case, a gateway error), the error is printed and the progress bars are redrawn below it, with the progress bars above it freezing.

<img width="755" alt="image" src="https://github.com/user-attachments/assets/af51a4d6-1407-4527-b353-19d48ee334f0">

**To Reproduce**

Not sure of the best approach to reproduce since it's intermittent. Perhaps removing network connectivity and trying to run an eval?

**Expected behavior**

Ideally, the error would be displayed somewhere but the progress bars would either clear or redraw.

**Screenshots**

See above

**System information (please complete the following information):**

- Promptfoo version: I am using v0.96.1, although I think I saw this after https://github.com/promptfoo/promptfoo/issues/1970 was merged and before our organization OpenAI rate limit was raised. So I think it has been around for a little bit at least.

**Additional context**

This is not the biggest deal, but wanted to file (feel free to close if not significant enough.)

Thanks for filing this @panozzaj! This bothers me too and is on my backlog. When I have more time, I’d like to address this by redoing the progress bars with a more powerful library like [Ink](https://github.com/vadimdemedes/ink). In the meantime, you could try piping stderr to a different output than stdout as a workaround.  If you have the time, I’d love any help on this (or any higher value features you think promptfoo is missing). 

Makes sense. Thanks for the pointer on Ink, that seems useful!

FYI one thing that might be handy for my case around progress bars would be seeing the successes/failures in real-time. Then if I see a lot of failures I can break execution early and investigate what is going on.

if still relevant, I could try taking a stab at this @mldangelo !
Repository: promptfoo/promptfoo
Issue: Broken Anthropic System Message for Amazon Bedrock
Root cause: Interface Contract Mismatch

**Describe the bug**

Similarly to #2123, the current message parsing logic does not extract the system message from the message list to be used as a separate parameter.

To reproduce, use the existing example from `examples/claude-vs-gpt-image/promptfooconfig.yaml` with the `bedrock:anthropic.claude-3-sonnet-20240229-v1:0` provider.

The associated API call returns the following error:

```
API call error: ValidationException: Malformed input request: #: subject must not be valid against schema {"required":["messages"]}#/messages/0/role: system is not a  valid enum value, please reformat your input and try again.
```

**System information (please complete the following information):**

- Promptfoo version: 0.96.1


Hi @maximcherny, thanks for catching this, and apologies for the disruption! I just issued a fix in release 0.96.2. Could you give it a try and let me know if it resolves the issue on your end? Feel free to reach out if you notice anything else or need further support.

Thanks so much @mldangelo and apologies for the delay. 

If I test with the latest HEAD at `05f324d6deb69581f93bab59e3a27231532f788e` and only this provider `bedrock:anthropic.claude-3-sonnet-20240229-v1:0` I am now getting the following error:

```
[FAIL] Error: OpenAI API key is not set. Set the OPENAI_API_KEY environment variable or add `apiKey` to the provider config.
```

```
npm run local -- --version
> promptfoo@0.97.0 local
> ts-node --cwdMode --transpileOnly src/main.ts --version

0.97.0
```

I am not sure why this config setting would be required for the provider I am using?

Correction - if I change the `defaultTest` provider for the `llm-rubric` assertion everything works as expected, as per the troubleshooting docs, thanks again for fixing this.
Repository: promptfoo/promptfoo
Issue: Token usage no longer displayed in web UI
Root cause: Algorithmic Error

Token usage information is not displaying in the current web ui.

Possibly due to eval model refactor?

<img width="1082" alt="image" src="https://github.com/user-attachments/assets/0c23af3f-f03e-4d7c-b3f9-6b3856d976ea">
Repository: promptfoo/promptfoo
Issue: Why is docker version lagging behind git?
Root cause: Configuration Error

**Describe the bug**
I pulled docker file from main and it is 0.93.3 instead of latest - 0.96.2.

**To Reproduce**
Follow the documentation to pull docker image from the main branch.
https://www.promptfoo.dev/docs/usage/self-hosting/

**Expected behavior**
Container has the latest version of promptfoo running.

**System information (please complete the following information):**

- Promptfoo version: 0.93.3 & 0.96.2


Hi @yelonek, thanks for reaching out! Can you please try pulling the latest image again with the following commands?

```sh
docker pull ghcr.io/promptfoo/promptfoo:latest
docker run -it ghcr.io/promptfoo/promptfoo:latest promptfoo --version
```
When I run these, I get version 0.96.2. I'll update the docs to reflect latest over main. Please feel free to reach out with other comments / concerns / questions. 

Hi @mldangelo!
Thanks for such a quick reponse. Couple of seconds ago I deleted the old image and downloaded a fresh one. The response is `0.96.1`.

Which is still confusing, but good enough for me now. Thanks!

EDIT: What is even more confusing is that when I go to the website it says it's 0.96.2!

![image](https://github.com/user-attachments/assets/52ab1e4e-d1af-4ce2-b87d-c789cc20bb09)

![image](https://github.com/user-attachments/assets/8b3380a0-fe57-4f51-90b7-7ed650bdc8f3)


@yelonek A new Docker release should be available in about 20 minutes. You can follow the progress here: https://github.com/promptfoo/promptfoo/actions/runs/11975522898. I’m going to close this issue, but feel free to reopen it if you encounter any other issues or have suggestions for Docker features, improvements, tags, registries, etc.
Repository: promptfoo/promptfoo
Issue: Webhook provider support "body" message
Root cause: Documentation Deficiency

Currently, webhook provider only transfer one parameter to backend:
```json
{
  "prompt": "..."
}
```
which is fine but it would be better to support transfer arbitrary data to backend to support more scenarios. It could be implemented like:
```yaml
providers: 
  - webhook:https://fabb16d9-c83b-4128-a498-d0df44e65faf--main.babel.run/minimax:
      id: minimax
      memo: "arbitrary string"
```
The POST message may look like:
```json
{
  "prompt": "...",
  "memo": "arbitrary string"
}
```

This was added in 0.25.0.  Details here: https://promptfoo.dev/docs/providers/webhook/#passing-custom-properties
Repository: promptfoo/promptfoo
Issue: Question : How to evaluate a single value in a Json Output ?
Root cause: Documentation Deficiency

Hi !

It's been two weeks that I am using promptfoo for a project, trying to understand what's possible to do, and finally I'm stuck with this issue : I want an LLM to extract some information from a text and return them under a json format. I'm pretty happy with my prompt, but now I want to evaluate the result. So basically, I want to evaluate that some information are exactly the same and some are similar.

Example : we got this output from the LLM to evaluate:
{name : "Roger", age : "53", remark : "shy"}
what is expected:
{name : "Roger", age : "53", remark : "a bit shy"}

I want to evaluate that the values "name" and "age" match exactly what is expected, but I want to evaluate the "remark" similarly. The first one was obvious to me, I just compared it to a JSON schema with constants for name and age; but for remark, I don't want to enum all the possibilities, so how can I just test this value with the similarity?

Thank you for the answer ! 

Hi @BlahBlah314,

The way to do this is with a `postprocess` option: https://promptfoo.dev/docs/configuration/guide/#postprocessing.  

`postprocess` is a javascript snippet that acts on a test case.  For example:
```yaml
tests:
  - options:
      postprocess: JSON.parse(output).remark
    assert:
      - type: similarity
        value: shy
```

Let me know how this works for you!


> Hi !
> 
> It's been two weeks that I am using promptfoo for a project, trying to understand what's possible to do, and finally I'm stuck with this issue : I want an LLM to extract some information from a text and return them under a json format. I'm pretty happy with my prompt, but now I want to evaluate the result. So basically, I want to evaluate that some information are exactly the same and some are similar.
> 
> Example : we got this output from the LLM to evaluate: {name : "Roger", age : "53", remark : "shy"} what is expected: {name : "Roger", age : "53", remark : "a bit shy"}
> 
> I want to evaluate that the values "name" and "age" match exactly what is expected, but I want to evaluate the "remark" similarly. The first one was obvious to me, I just compared it to a JSON schema with constants for name and age; but for remark, I don't want to enum all the possibilities, so how can I just test this value with the similarity?
> 
> Thank you for the answer !

I believe you need a javascript assertion https://www.promptfoo.dev/docs/configuration/expected-outputs#javascript which should parse the output json and validate different fields accordingly. The built-in assertions can't do the combinations. 

Hi both of you !

Thank you for the quick answers! It worked very well with the postprocess parsing in my case. So again, thank you !

For further potential questions, I've joined the Discord community :)
Repository: promptfoo/promptfoo
Issue: apiVersion is interpreted as a timestamp
Root cause: Validation Gap

**Describe the bug**
In the config file I have:
```
  - id: azure:chat:gpt-4
    config:
      apiBaseUrl: "https://foo.bar"
      apiVersion: 2023-05-15
```
In `providers.ts` I added the line `console.log("config:", providerOptions);` and I get the following in the console:
```
config: {
  id: 'azure:chat:gpt-4',
  config: {
    apiBaseUrl: 'https://foo.bar',
    apiVersion: 2023-05-15T00:00:00.000Z,
    basePath: ''
  },
  env: undefined
} 
```
Notice the timestamp format of the `apiVersion` value.

When the REST endpoint gets called the URL is:
```
https://foo.bar/openai/deployments/gpt-4/chat/completions?api-version=Sun May 14 2023 20:00:00 GMT-0400 (Eastern Daylight Time)
```
Notice how the `apiVersion` is a human-friendly timestamp converted to the local time zone.

**To Reproduce**
See above.

**Expected behavior**
Produce a correct endpoint URL.

**Screenshots**
N/A

**System information (please complete the following information):**

- Promptfoo version: latest from [master](https://github.com/promptfoo/promptfoo/tree/17ad62ebe7f706bcf5dc2e78aa83db475f62a8f4)

**Additional context**
N/A

I wonder if this is some unintended magic from the yaml parser?

On Tue, Nov 19, 2024 at 4:34 PM rvernica ***@***.***> wrote:

> *Describe the bug*
> In the config file I have:
>
>   - id: azure:chat:gpt-4
>     config:
>       apiBaseUrl: "https://foo.bar"
>       apiVersion: "2023-05-15"
>
> In providers.ts I added the line console.log("config:", providerOptions);
> and I get the following in the console:
>
> config: {
>   id: 'azure:chat:gpt-4',
>   config: {
>     apiBaseUrl: 'https://foo.bar',
>     apiVersion: 2023-05-15T00:00:00.000Z,
>     basePath: ''
>   },
>   env: undefined
> }
>
> Notice the timestamp format of the apiVersion value.
>
> When the REST endpoint gets called the URL is:
>
> https://foo.bar/openai/deployments/gpt-4/chat/completions?api-version=Sun May 14 2023 20:00:00 GMT-0400 (Eastern Daylight Time)
>
> Notice how the apiVersion is a human-friendly timestamp converted to the
> local time zone.
>
> *To Reproduce*
> See above.
>
> *Expected behavior*
> Produce a correct endpoint URL.
>
> *Screenshots*
> N/A
>
> *System information (please complete the following information):*
>
>    - Promptfoo version: latest from master
>    <https://github.com/promptfoo/promptfoo/tree/17ad62ebe7f706bcf5dc2e78aa83db475f62a8f4>
>
> *Additional context*
> N/A
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/2174>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJTAGZTPN2NJARHDLG32BPKI7AVCNFSM6AAAAABSDLWEOWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGY3TGOJXHA4TCMQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


YAML treats the value as a time stamp. Adding quotes around the value fixes the issue:
```
     apiVersion: "2023-05-15"
```