zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Feature Request: Have optional vars in assertions with ability to define fallbacks
Root cause: Unimplemented Feature Gap

While writing custom assertions for test cases, there are several times we need an optional parameter to fine tune the assertions or provide an ability to switch custom logic on the fly (Like switching between different rubrics or based on the vars).

For eg. 

1. I want to make **threshold_percentage** in the following yaml snippet optional. I have a logic in custom python assertion script **(llm_similarity_rubrics_test.py)** to directly use 80% if no value is provided.

```
  - vars:
      question: Help me preparing a birthday cake!
      similarity_text: I'm sorry, but as a customer service AI assistant,
        I'm focused on helping you with your shopping experience.
        If you have any questions related to that, please feel free to ask!
      threshold_percentage: 70  #Removing this line from here should also work since I have fallbacks in custom python code
    assert:
      - type: python
        value: file://llm_similarity_rubrics_test.py
```


2. I want to have a separate custom python rubric that needs to have separate set of parameters to be passed in the same yaml file 

```
  - vars:
      question: Help me preparing a birthday cake!
      contains_text: I'm focused on helping you with your shopping experience. 
      #This rubrics just checks if a particular piece of text is in output or not. I should be able to switch the params for it on the fly.
    assert:
      - type: python
        value: file://llm_contains_rubrics_test.py
```


I'm also quite new to promptfoo so if there are other ways to achieve this, please educate me :)
Repository: promptfoo/promptfoo
Issue: Browser exception (Minified React error #31; visit) when clicking magnifying glass in output
Root cause: Validation Gap

Using promptfoo 0.57.1 on macOS 14.4. Chrome Version 124.0.6367.119 (Official Build) (arm64)

When viewing the Web UI, clicking on the magnifying glass (🔎) in the output works okay if the output dict is small.  But for some of our examples, we have a custom Python provider that returns a dictionary for `output` and some of the fields are very large. On those rows, clicking the magnifying glass results in this error in the browser console:

```
Error: Minified React error #31; visit https://reactjs.org/docs/error-decoder.html?invariant=31&args[]=object%20with%20keys%20%7Bstatement%2C%20reason%2C%20verdict%7D for the full message or use the non-minified dev environment for full errors and additional helpful warnings.
    at nz (fd9d1056-a95b789c94c05ff6.js:9:29694)
    at d (fd9d1056-a95b789c94c05ff6.js:9:31594)
    at fd9d1056-a95b789c94c05ff6.js:9:33994
    at s (fd9d1056-a95b789c94c05ff6.js:9:34285)
    at u (fd9d1056-a95b789c94c05ff6.js:9:35368)
    at lo (fd9d1056-a95b789c94c05ff6.js:9:52233)
    at im (fd9d1056-a95b789c94c05ff6.js:9:115510)
    at oM (fd9d1056-a95b789c94c05ff6.js:9:90098)
    at fd9d1056-a95b789c94c05ff6.js:9:89920
    at oT (fd9d1056-a95b789c94c05ff6.js:9:89927)
window.console.error @ 596-f3a131b7c336dd5d.js:1
r9 @ fd9d1056-a95b789c94c05ff6.js:9
n.callback @ fd9d1056-a95b789c94c05ff6.js:9
nf @ fd9d1056-a95b789c94c05ff6.js:9
nd @ fd9d1056-a95b789c94c05ff6.js:9
am @ fd9d1056-a95b789c94c05ff6.js:9
ay @ fd9d1056-a95b789c94c05ff6.js:9
aT @ fd9d1056-a95b789c94c05ff6.js:9
ay @ fd9d1056-a95b789c94c05ff6.js:9
aT @ fd9d1056-a95b789c94c05ff6.js:9
ay @ fd9d1056-a95b789c94c05ff6.js:9
aT @ fd9d1056-a95b789c94c05ff6.js:9
ay @ fd9d1056-a95b789c94c05ff6.js:9
aT @ fd9d1056-a95b789c94c05ff6.js:9
ay @ fd9d1056-a95b789c94c05ff6.js:9
aT @ fd9d1056-a95b789c94c05ff6.js:9
ay @ fd9d1056-a95b789c94c05ff6.js:9
aT @ fd9d1056-a95b789c94c05ff6.js:9
ay @ fd9d1056-a95b789c94c05ff6.js:9
aT @ fd9d1056-a95b789c94c05ff6.js:9
ay @ fd9d1056-a95b789c94c05ff6.js:9
(anonymous) @ fd9d1056-a95b789c94c05ff6.js:9
oO @ fd9d1056-a95b789c94c05ff6.js:9
nG @ fd9d1056-a95b789c94c05ff6.js:9
nJ @ fd9d1056-a95b789c94c05ff6.js:9
(anonymous) @ fd9d1056-a95b789c94c05ff6.js:9
```

I realize this minified stack trace isn't that helpful but happy to try and provide more details. It's 100% reproducible.

Any chance you'd be able to share an eval with me that repros?  Using `promptfoo export <evalId> -o <outputpath>.json`
Repository: promptfoo/promptfoo
Issue: Documentation for data structures GradingResult and ProviderResponse
Root cause: Documentation Deficiency

For the following structures, the documentation differs from what we've tested. Happy to make a PR to update the docs, if someone can confirm the following. I'm only using Python custom assertions and providers.

# GradingResult ([Typescript reference](https://www.promptfoo.dev/docs/configuration/reference/#gradingresult))
- `score` documented to be a `float` between 0 and 1, but we've been able to return a dictionary.
- `reason` is allowed to be `None` ( I guess `null` in Typescript)

# ProviderResponse ([Typescript reference](https://www.promptfoo.dev/docs/configuration/reference/#providerresponse))
- The [Python documentation](https://www.promptfoo.dev/docs/providers/python/#types) says that `output` must be `Optional[str]`.  However, the Typescript docs says it could also be a dictionary, and we confirmed that works in Python.

Thanks for catching these errors!

In the Typescript definition, [GradingResult](https://github.com/promptfoo/promptfoo/blob/main/src/types.ts#L340) expects a numeric `score` and a non-null `reason`.  We should treat that as the source of truth because the rest of the code depends on these types.  So I would consider this a validation issue in the Python assertion (maybe `javascript` too).

For [ProviderResponse](https://github.com/promptfoo/promptfoo/blob/main/src/types.ts#L153), it does look like `output` can be an object.  So the Python docs are wrong.
Repository: promptfoo/promptfoo
Issue: Fresh install: Cannot find module 'node:events'
Root cause: Environment Incompatibility

```
~/dev/example $ npx promptfoo init

Cannot find module 'node:events'
Require stack:
- /Users/blairanderson/.nodenv/versions/14.16.0/lib/node_modules/promptfoo/node_modules/minipass/dist/cjs/index.js
- /Users/blairanderson/.nodenv/versions/14.16.0/lib/node_modules/promptfoo/node_modules/path-scurry/dist/cjs/index.js
- /Users/blairanderson/.nodenv/versions/14.16.0/lib/node_modules/promptfoo/node_modules/glob/dist/cjs/src/glob.js
- /Users/blairanderson/.nodenv/versions/14.16.0/lib/node_modules/promptfoo/node_modules/glob/dist/cjs/src/index.js
- /Users/blairanderson/.nodenv/versions/14.16.0/lib/node_modules/promptfoo/dist/src/util.js
- /Users/blairanderson/.nodenv/versions/14.16.0/lib/node_modules/promptfoo/dist/src/telemetry.js
- /Users/blairanderson/.nodenv/versions/14.16.0/lib/node_modules/promptfoo/dist/src/main.js
```

Hi @blairanderson, this library requires Node 16 or above.  I'll add a note to the install guide.

Great thanks!

I saw this and assumed 12/14 would also work https://github.com/promptfoo/promptfoo/blob/main/package.json#L22

Fair assumption :sweat_smile:   I'll get that fixed.
Repository: promptfoo/promptfoo
Issue: How to get, Prompt, Output, and Assertion value into Python
Root cause: Documentation Deficiency

I am using a Python script as my prompt, and I was wondering how I could get the final LLM response from my provider and the assertion response.

I am using the `llm-rubric` assert type, and I will probably keep it that way. 

I've seen the "Provider" Python script, but I can't tell how to get the Assertion response back because I can't figure out how to get the test case's assertion value to compare the original outputs against.

Finding it a little hard to follow - can you give me an example of what you're trying to do?  

Hi @typpo, I got caught up with something. What I am asking for is a way to get all 4:
1) original prompt, 
2) the llm responses to that original prompt, 
3) the test case, 
4) and the llm-rubric assessment response comparing the test case to the original llm response

Having all 4 at once in Python would be cool, and I'm wondering if there is any way to achieve that, because I want to use all 4 into a different script.
Repository: promptfoo/promptfoo
Issue: Use test vars in function schema
Root cause: Unimplemented Feature Gap

I'm trying to evaluate a prompt that uses function calls. However, the function schema depends on one of the variables in the test output. It looks like function calls currently are part of the provider config and not of the prompt, so I don't see a way to inject variables into a function definition. Is there any other way to adjust the function schema per test case?

Thanks for the suggestion @tomquist!  #792 implements this and will be in the next release.

Released in 0.59.0!

@typpo Awesome! Thanks for the quick implementation. 🎉

I have one use-case which still wouldn’t be possible with this implementation:

I want to pass in a JSON Array as a var so I can use it in a `string` schema as the `enum` property. Would it be possible to support a function at any level of the template object? This waz you could do something like this:
```
tools: [
  {
    type: …, parameters: (context) => ({ enum: context.vars.options, …})
  }
]
```
_Sorry for the minimalistic example, I’m on mobile right now_

I think this could just be handled as another case in `renderVarsInObject()`.

Also I’m happy to open a new issue if you prefer.

@typpo I submitted a PR to implement this: https://github.com/promptfoo/promptfoo/pull/813
Repository: promptfoo/promptfoo
Issue: When writing JSON output of an eval, include the timestamp of the run
Root cause: Unimplemented Feature Gap

In the SQLite database of promptfoo, I can see that evals have a `createdat` timestamp, and this is visible in the Web UI as well.

However, when running via the CLI using something like `npx promptfoo eval --output eval_output.json`, the JSON file does not contain a timestamp anywhere. 

It would be nice if a timestamp could be included somewhere in the output dictionary.

Done in #785 !

Released in 0.59.0
Repository: promptfoo/promptfoo
Issue: Python scripts in prompts does not work with anthropic as provider
Root cause: Validation Gap

Stacktrace:
```
/usr/local/lib/node_modules/promptfoo/dist/src/prompts.js:225
                            args: [JSON.stringify(context)],
                                        ^

TypeError: Converting circular structure to JSON
    --> starting at object with constructor 'Anthropic'
    |     property 'completions' -> object with constructor 'Completions'
    --- property '_client' closes the circle
    at JSON.stringify (<anonymous>)
   ```
  
promptfooconfig.yaml
```
prompts:
  - file://prompt.py

providers:
  - openai:gpt-4o
  - anthropic:messages:claude-3-opus-20240229
  ``` 
   
Nodejs version - v20.12.2
Promptfoo version - 0.58.1
Please let me know if it is reproducible for you.


Thanks for flagging!  Fix in https://github.com/promptfoo/promptfoo/pull/784

As a temporary workaround, you can switch to using a specific function in the python file, e.g. `prompt.py:my_prompt`.  That should work alright in the current version.

Thanks for quick fix.

Released in 0.59.0!
Repository: promptfoo/promptfoo
Issue: Prettify messages json strings for multi-prompt chat conversations
Root cause: Unimplemented Feature Gap

If I use a multi-prompt chat per the [chat conversations](https://www.promptfoo.dev/docs/providers/openai#chat-conversations) docs, the view html output is rendered as JSON.

It would be nice if there was a way to render the role/content JSON in an easier to view form as a Table View Settings option as the raw JSON can be hard to read, especially long prompts or prompts that include code or other markup/markdown.

I believe this is addressed by the introduction of the `label` field on prompts. Closing it out, lmk if you have any issues! https://promptfoo.dev/docs/configuration/parameters#different-prompts-per-model

I'd like to add my 2 cents here. That's how JSON prompts are rendered in the UI. It's quite hard to read. 

In my case, prompts are generated by JS script, so it's impossible to read them easily in an other way. 

<img width="1219" alt="image" src="https://github.com/user-attachments/assets/d0a6f0db-b7f8-4bbb-9ff5-f8b181ff9040">
Repository: promptfoo/promptfoo
Issue: Standardize how LLM output is stored in a ProviderResponse output object
Root cause: Unimplemented Feature Gap

`ProviderResponse` has an `output` field that is [documented](https://www.promptfoo.dev/docs/configuration/reference/#providerresponse) to be either a string or an object.  

If it's a string, then the Web UI shows a nice diff between the LLM outputs when comparing two prompts or providers. 

However, if it's an object, the serialized object is shown and you lose the nice diff output.

Can a convention be established about what goes into the object, specifically if a field with a specific key exists, then that represents the LLM output? (And any other fields can be visible in the Web UI if you click on the magnifying glass).

This seems reasonable.  What do you think we should call the displayed field?  `content`, `display`, `text`?

> This seems reasonable. What do you think we should call the displayed field? `content`, `display`, `text`?

I prefer `text` or `content`.  

Added a frontend configuration for this in #893
Repository: promptfoo/promptfoo
Issue: Use Gcloud default authentication instead of an API key
Root cause: Validation Gap

Hi, thanks for the great package.  I am self hosting on Cloud Run and attempting to hook it up to the self-hosted Langfuse too.

I am using the new bucket mount feature to write the Prompt foo evaluations to disk, and calling the promptfoo eval within cloudbuild.yaml.

When running within GCP, there is an authentication token available in the metadata, so I don't need to use API keys etc.  It would be great if the vertex ai check could reuse this authentication, which is assigned at the service key level without needing the less secure JSON key.

e.g. currently:

```
Error: Google Vertex API key is not set. Set the VERTEX_API_KEY environment variable or add `apiKey` to the provider config. You can get an API token by running `gcloud auth print-access-token`

Error: Google Vertex API key is not set. Set the VERTEX_API_KEY environment variable or add `apiKey` to the provider config. You can get an API token by running `gcloud auth print-access-token`
    at VertexChatProvider.callApi (/usr/local/lib/node_modules/promptfoo/dist/src/providers/vertex.js:105:19)
    at Evaluator.runEval (/usr/local/lib/node_modules/promptfoo/dist/src/evaluator.js:277:62)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async processEvalStep (/usr/local/lib/node_modules/promptfoo/dist/src/evaluator.js:594:25)
 ```   
 
I don't know any node to offer a PR, but I'm sure there is a line that can be added that checks for default authentication details before looking for an API key.

Thanks for flagging this, Mark.  We definitely should get it working with gcloud's default application credentials.  It looks like the code already supports it, but there might be an extra apiKey check that we don't need.

When you're running locally, are you using default application credentials or manually providing the API key via `gcloud auth print-access-token`?

I’ve not actually run locally yet - only via cloud build. 

Got it.  Will have a look and include the fix in the next release, so that it picks up [default application credentials](https://cloud.google.com/docs/authentication/application-default-credentials).

Great thanks!

Fix landed in 0.59.0!  LMK how it works

Awesome, will try it out
Repository: promptfoo/promptfoo
Issue: Feature Request: Also Pass `logProbs` to Python Assertions
Root cause: Unimplemented Feature Gap

Currently, Python custom asserts only receive the output and the context from the provider. In order to provide full functionality, we also need to pass the `logprobs` object.

https://github.com/promptfoo/promptfoo/blob/7639fbe5bb40482be7faf4f9bbe1aed2a277d2fc/src/assertions.ts#L268

#790 adds `logProbs` to the `context` object passed to assertion functions :)

Released in 0.59.0!
Repository: promptfoo/promptfoo
Issue: How to test prompt with triple dashes inside without breaking into separate prompts?
Root cause: Configuration Error

I have a prompt that post-RAG has retrieved docs separated by a triple dash, `---`, i.e the below. Command I'm running is `npx promptfoo@latest eval  --tests tests.csv --providers openai:gpt-3.5-turbo --output output.json`, and the `promptfooconfig.yaml` is:

 ```
env: {}
description: ''
providers:
  - id: openai:gpt-3.5-turbo-0613
    config:
      temperature: 0.5
      max_tokens: 1024
      top_p: 1
      frequency_penalty: 0
      presence_penalty: 0
prompts:
  - "This is my system prompt...\nThe current date is Friday, 17 May 2024.n--- Snippet from blog (url_name = url) ---\n--- End snippet ---\n--- Snippet from blog (url_name = url) ---\n--- End snippet ---"
tests: 
defaultTest: {}
evaluateOptions: {}

```

```
This is my system prompt...
The current date is Friday, 17 May 2024.

--- Snippet from blog (url_name = url) ---
<text>
--- End snippet ---
--- Snippet from blog (url_name = url) ---
<text>
--- End snippet ---
```

Currently, these are truncating into separate prompts...

<img width="794" alt="image" src="https://github.com/promptfoo/promptfoo/assets/9962584/a845ad31-0b58-4a51-ad3b-12fbf3c828a3">

Is there a way around this?

A workaround is that you can set the `PROMPTFOO_PROMPT_SEPARATOR` environment variable to something else.  For example:

```
PROMPTFOO_PROMPT_SEPARATOR='xxxxxx' promptfoo eval
```
Repository: promptfoo/promptfoo
Issue: In the html report (view/shared), instruction tokens and completion tokens should be shown separately.
Root cause: Unimplemented Feature Gap

In the current implementation of the HTML report (view/shared), the instruction tokens and completion tokens are not differentiated. For better clarity and analysis, it is important to show instruction tokens and completion tokens separately in the report.



Shown in #794.

Amazing. Thanks for the prompt resolution. This tool is amazing.

Released in 0.59.0!
Repository: promptfoo/promptfoo
Issue: No Amazon Bedrock models can be used as embedding providers for similarity assertion
Root cause: Unimplemented Feature Gap

Hello,
No Amazon Bedrock models can be used as embedding providers for similarity assertion.  I have tried all of them with these results:

``` sh
Provider bedrock:cohere.embed-english-v3 is not a valid embedding provider for 'similarity check', falling back to default
```
``` sh
Provider bedrock:amazon.titan-embed-text-v2:0 is not a valid embedding provider for 'similarity check', falling back to default
```
``` sh
Provider bedrock:amazon.titan-embed-text-v1 is not a valid embedding provider for 'similarity check', falling back to default
```
If it is a new feature rather than my mistake and, you think its doable for a newbie I'm happy to try and create a PR for it.  It seems I would need to implement ```BedrockEmbeddingProvider``` class in [bedrock.ts](https://github.com/promptfoo/promptfoo/blob/bfaa9e9882fca26edaa4c9dd31de319b34e40855/src/providers/bedrock.ts).
Cheers,
Mark


#797 adds support for Bedrock embedding models!

Released in 0.59.0!
Repository: promptfoo/promptfoo
Issue: Browser view seems to fail with very large TestSuites 
Root cause: Resource Mishandling

It seems that when trying to render a large result set, 200+ tests with lots of data, the UI fails to display all of the information.

From first glance it looks like it might be possibly due to a limitation in the web socket delivery, but have not looked into it in detail.

How large is your `~/.promptfoo/output/latest.json` file and is there a version of the file you'd be able to share privately?  When you say "fails to display all of the information" can you elaborate a bit on what that failure looks like?

I've just tried it with a 470mb results file and it worked ok (albeit with a bit of a wait), so will likely need more debugging info to pinpoint an issue here.

Closing this!  Feel free to reopen with more repro details.
Repository: promptfoo/promptfoo
Issue: SQLITE_CONSTRAINT_NOT_NULL Error when running "promptfoo view"
Root cause: Validation Gap

"promptfoo eval" can be succesfully executed.
However, "promptfoo view" seems to have some operation with sqlite and is showing this error every time. Do you know how to fix it?
-----

🔁 Migrating 50 flat files to local database.
This is a one-time operation and may take a minute...
Backup of output directory created at /Users/tiffanychen/.promptfoo/output-backup-20240518
Moving files into database...

/usr/local/lib/node_modules/promptfoo/node_modules/drizzle-orm/better-sqlite3/session.cjs:79
    return this.stmt.run(...params);
                     ^
SqliteError: NOT NULL constraint failed: evals.created_at
    at PreparedQuery.run (/usr/local/lib/node_modules/promptfoo/node_modules/drizzle-orm/better-sqlite3/session.cjs:79:22)
    at QueryPromise.run (/usr/local/lib/node_modules/promptfoo/node_modules/drizzle-orm/sqlite-core/query-builders/insert.cjs:158:28)
    at writeResultsToDatabase (/usr/local/lib/node_modules/promptfoo/dist/src/util.js:424:10)
    at /usr/local/lib/node_modules/promptfoo/dist/src/util.js:583:19
    at Array.map (<anonymous>)
    at migrateResultsFromFileSystemToDatabase (/usr/local/lib/node_modules/promptfoo/dist/src/util.js:580:41)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async startServer (/usr/local/lib/node_modules/promptfoo/dist/src/web/server.js:42:5)
    at async Command.<anonymous> (/usr/local/lib/node_modules/promptfoo/dist/src/main.js:253:9) {
  code: 'SQLITE_CONSTRAINT_NOTNULL'
}

Node.js v18.15.0


Thanks for the report.  Just got a fix in and published as 0.59.1

You can upgrade to 0.59.1, or reverting to a previous version (0.58.1) should work too.
Repository: promptfoo/promptfoo
Issue: How can we secure self hosted promptfoo server?
Root cause: Unimplemented Feature Gap

Request for shareable URL is done without any authentication config. 

https://github.com/promptfoo/promptfoo/blob/6cf39faa4f2a22513a1c71017a07d6089ab88600/src/share.ts#L22



Alternatively, is there a way we can pass headers to the client used to talk to the server?

For example, if our self-hosted server is behind something that implements Basic Auth, we could configure the HTTP headers sent by the promptfoo client to pass the authentication secret.

Hello, like @efung mentioned, is there a possibility to define headers as part of the sharing object, and pass those headers through?

`export PROMPTFOO_REMOTE_API_BASE_URL=https://<user>:<pass>@<host.com>`

This works for basic auth.
Repository: promptfoo/promptfoo
Issue: Web ui doesn't support object prompts config
Root cause: Documentation Deficiency

I want to test different prompt for different model, and the `promptfooconfig.yaml` content like below ⬇️

```yaml
prompts:
  prompt-1.yaml: prompt-1
  prompt-2.yaml: prompt-2
providers:
  - id: openai:gpt-4-turbo-preview
    prompts:
      - prompt-1
  - id: openai:gpt-4o
    prompts:
      - prompt-2
```

when using promptfoo cli, it works as expected. but it's not working while directly pasting yaml to web ui, error shows below ⬇️

![image](https://github.com/promptfoo/promptfoo/assets/35442047/52a1f45e-02f5-4078-8e6f-f889b3932abc)


@kongmoumou We deprecated the object syntax in favor of array syntax. There's a good example of the syntax here: prompt-1 You should be able to run:

```yaml
prompts:
  - id: prompt-1.yaml
    label: prompt-1
  - id: prompt-2.yaml
    label: prompt-2
providers:
  - id: openai:gpt-4-turbo-preview
    prompts:
      - prompt-1
  - id: openai:gpt-4o
    prompts:
      - prompt-2
```

I hope this helps.