zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Do not display Basic Auth username and password after creating share URL
Root cause: Validation Gap

In the CLI, if I define my hosted promptfoo server using a Basic Auth username and password via environment variable, e.g.:

```
export PROMPTFOO_REMOTE_API_BASE_URL=https://my_user:supersecret@example.com
```

When I then run `npx promptfoo share -y`, the confirmation message afterward says:

```
View results: https://my_user:supersecret@example.com/eval/f:12345678-abcd-cafe-9999-123450abcdef
```

Could that URL be stripped of the username/password? (I think the Node `URL` class has an `origin` property that could be useful here.)

Although, I realize that this means that whoever is opening that URL needs to have some way to authenticate to the server, if viewing the shared eval requires authentication also.

(In our case, our nginx front end supports OAuth for human viewers. The Basic Auth is just for the promptfoo client when uploading via API.)

`promptfoo share` is useful to upload the eval to a shared server. We don't need the private URL functionality.
Repository: promptfoo/promptfoo
Issue: Gemini better handling of safety failures
Root cause: Validation Gap

With `promptfoo==0.69.2` and the provider `vertex:gemini-1.5-pro-001`, passing the prompt:

```none
Q: How is Bempegaldesleukin supposed to overcome Treg affinity for IL-2?

Options:
A) Insufficient information to answer this question
B) Preferentially binding to b2b3 complex over b1b2b3 complex
C) Overcome the Treg affinity for IL-2 by selectively depleting intratumoral Tregs
D) Tumor microenvironment targeting and checkpoint immune inhibition
E) Improve PK via PEGylation

Answer:
```

Leads to a `TypeError`:

```none
Gemini API response error: TypeError: Cannot read properties of undefined (reading 'parts'): [{"candidates":[{"content":{"role":"model","parts":[{"text":"The"}]}}]},{"candidates":[{"finishReason":"SAFETY","safetyRatings":[{"category":"HARM_CATEGORY_HATE_SPEECH","probability":"NEGLIGIBLE","probabilityScore":0.13892843,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.13695431},{"category":"HARM_CATEGORY_DANGEROUS_CONTENT","probability":"MEDIUM","blocked":true,"probabilityScore":0.64220816,"severity":"HARM_SEVERITY_MEDIUM","severityScore":0.46952027},{"category":"HARM_CATEGORY_HARASSMENT","probability":"NEGLIGIBLE","probabilityScore":0.13580412,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.15791446},{"category":"HARM_CATEGORY_SEXUALLY_EXPLICIT","probability":"NEGLIGIBLE","probabilityScore":0.30559373,"severity":"HARM_SEVERITY_NEGLIGIBLE","severityScore":0.056966383}]}],"usageMetadata":{"promptTokenCount":95,"candidatesTokenCount":1,"totalTokenCount":96}}]
```

I think `promptfoo` doesn't expect this shape of response from Gemini, because it says `TypeError`.

Can we make it such that `promptfoo` handles this without a `TypeError`?

For example, to fix this in Python with `langchain-google-vertexai==1.0.6` we had to do this:

```python
from langchain_google_vertexai import ChatVertexAI, HarmBlockThreshold, HarmCategory

client = ChatVertexAI(
    ...,
    safety_settings={
        HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_ONLY_HIGH,
        HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_ONLY_HIGH,
        HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_ONLY_HIGH,
        HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_ONLY_HIGH,
    },
)
```

Okay sorry, I realized this is already documented: https://www.promptfoo.dev/docs/providers/vertex/#safety-settings
Repository: promptfoo/promptfoo
Issue: Custom Nunjucks filter not found for eval executed outside config dir
Root cause: Configuration Error

The issue can be reproduced by attempting to run the `nunjucks-custom-filters` example from the root of the repository:

```
$ promptfoo eval -c examples/nunjucks-custom-filters/promptfooconfig.yaml

node_modules/nunjucks/src/lib.js:32
    err = new Error(old.message);
          ^
Template render error: (unknown path)
  Error: filter not found: allcaps
    at Object._prettifyError (node_modules/nunjucks/src/lib.js:32:11)
    at node_modules/nunjucks/src/environment.js:464:19
    at Template.root [as rootRenderFunc] (eval at _compile (node_modules/nunjucks/src/environment.js:527:18), <anonymous>:20:3)
    at Template.render (node_modules/nunjucks/src/environment.js:454:10)
    at Environment.renderString (node_modules/nunjucks/src/environment.js:313:17)
    at renderPrompt (src/evaluator.ts:259:21)
    at Evaluator.runEval (src/evaluator.ts:317:34)
    at processEvalStep (src/evaluator.ts:719:30)
    at node_modules/async/dist/async.js:151:38
    at replenish (node_modules/async/dist/async.js:448:21)
```

The issue does not occur if the eval is run from the same directory as the config file (`examples/nunjucts-custom-filters`).
Repository: promptfoo/promptfoo
Issue: Bug - Docker container healthcheck does not work
Root cause: Environment Incompatibility


Find below the logs for a container using a docker image built from this repository.
The error is pretty straightforward, curl is not installed. 

```json
{
"Health": {
			"Status": "starting",
			"FailingStreak": 2,
			"Log": [
				{
					"Start": "2024-07-19T08:16:12.438663134Z",
					"End": "2024-07-19T08:16:12.484082218Z",
					"ExitCode": 1,
					"Output": "/bin/sh: curl: not found\n"
				},
				{
					"Start": "2024-07-19T08:16:42.490393676Z",
					"End": "2024-07-19T08:16:42.530330759Z",
					"ExitCode": 1,
					"Output": "/bin/sh: curl: not found\n"
				}
			]
		}
}
```
Repository: promptfoo/promptfoo
Issue: Can't run promptfoo when Langgraph is getting system-prompts from config.yaml
Root cause: Configuration Error

I'm trying to evaluate my langgraph-model with promptfoo. However every time I'm trying to run the eval-command it tells me, that it can't find the config.yaml file where my system prompts are located.
I have written the different system prompts for my different nodes in this config.yaml and I'm importing them when needed.

@beschmitt I'm sorry you're running into an issue. Can you share more about your configuration including your promptfoo config and the directory structure surrounding it? Feel free to put time on my calendar if you would like to review this together: https://cal.com/michael-dangelo/30min

@mldangelo my directory structure looks something like this:
- root
    - promptfooconfig.yaml
    - project folder
        - subdirectory 1
            - subdirectory 2
                - config.yaml
                - subdirectory 3
                    - executable python file

The executable python is my chatbot that reads in the system prompts from my config.yaml
The executable python file has the following code:
```
import sys

from langgraph_constructor import app

question = sys.argv[1]

app.invoke({"question":question})
```

Where app is my compiled langgraph workflow.

The promptfooconfig.yaml has 
prompts: prompt.txt

providers:
    - "exec:python path/to/executable_python_file.py"

tests:
    - vars:
        question: Example question

My prompt.txt consists of {{question}} since all the system prompts are already in the graph.

# Edit:
Okay, the problem was with the way I wrote the location of the config.yaml in my code. It's working now
Repository: promptfoo/promptfoo
Issue: Docs bug: broken link on Gemini safety settings

Fyi https://www.promptfoo.dev/docs/providers/vertex/#safety-settings links https://ai.google.dev/api/rest/v1/SafetySetting, which is giving me a 404 error

Maybe there is a link checker for Docusaurus that can be added to CI or build process

Looks like this is the new link, which looks suspiciously un-futureproof as well: https://ai.google.dev/api/generate-content#safetysetting
Repository: promptfoo/promptfoo
Issue: Feature Request: Delay api calls (GPT-4 requirement)
Root cause: Documentation Deficiency

Promptfoo has been great so far. But we are running into problems with GPT-4.
Since it has lower rate & token limits, most api calls fail after around 20 calls.

Would be good to have an option to delay the calls, maybe after X number of calls or X number of tokens

I tried adding a delay in the __expected column of the csv but that didn't work either.
If you could add a guide on how to run the repo locally, I can also try adding it.

Thanks

Hi @sarons,

There are a few things you can do here:

1. **Reduce concurrency to 1** by setting `--max-concurrency 1` in the CLI, or by setting `evaluateOptions.maxConcurrency` in the config.
2. **Set a delay between requests** by setting `--delay 3000` in the CLI, or by setting `evaluateOptions.delay` in the config, or with the environment variable `PROMPTFOO_DELAY_MS` (all values are in milliseconds).
3. **Adjust the exponential backoff for failed requests** by setting the environment variable `PROMPTFOO_REQUEST_BACKOFF_MS`.  This defaults to 5000 milliseconds and retries exponential up to 4 times.  You can increase this value if requests are still failing, but note that this can significantly increase end-to-end test time.

Note that 2 and 3 were added in #122 (version 0.21.3), so you will have to update to get this functionality.

Hope this helps!

Oh awesome! Thanks @typpo, will give it a try.
I actually came up with another crazy solution in the meantime btw: I split my csv into mini csvs with 10 rows then ran promptfoo on each one to get mini output csv and then merged all the csvs into one csv [😅](https://emojipedia.org/grinning-face-with-sweat)

Now _that's_ a workaround... feel free to reopen this issue if the delay feature doesn't solve your problem!
Repository: promptfoo/promptfoo
Issue: How to set proper Azure config?
Root cause: Documentation Deficiency

```
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="",
    api_key="",
    api_version="",
    azure_deployment="",
)
response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[
    {"role": "user", "content": test}
  ]
)
```
 I couldn't set them up successfully due to the `azure_deployment` parameter. 
Any suggestions?

Hi @Stephenzza, have you taken a look at these docs? https://www.promptfoo.dev/docs/providers/azure/

If so, could you share the config you're using and what error you see?

Thank you for your response! I briefly checked the source code and noticed that the `model` parameter in the body is the same as the `deployment` parameter. However, in my use case, they are set separately. I tried configuring it this way, but it didn't work. Could you please advise me on how to set it up correctly? Thanks a lot!
```
providers:
  - id: azureopenai:chat:gpt-4o
    config:
      apiBaseUrl: "xxx.azure.com/openai/deployments/{my_deployment_name}"
      apiKey: "xxx"
      apiVersion: "xxx"
```

I'm showing my ignorance of Azure here, but I'm under the impression that in Azure OpenAI Studio, each deployment is associated with a single model.  With this setup, only the deployment name is necessary.  Is this no longer the case?

<img width="484" alt="image" src="https://github.com/user-attachments/assets/b4545a42-d023-4aef-8ab2-9c8f38d27790">


It is indeed a one-to-one correspondence. I mistakenly used gpt-4o, which caused the issue. Thank you for your patience and explanation!
Repository: promptfoo/promptfoo
Issue: YAML Validator Breaks Jinja-Templated YAML Prompt
Root cause: Validation Gap

There was an update validating the YAML format, and my previous test cases failed. 

error:
```
/opt/homebrew/lib/node_modules/promptfoo/node_modules/js-yaml/lib/loader.js:183
  return new YAMLException(message, mark);
         ^
YAMLException: end of the stream or a document separator is expected (7:1)

 4 |     Respond in {{ language }}.
 5 |     If the Human message is an e ...
 6 | 
 7 | {% for message in previous_messa ...
-----^
```


prompt:
```
- role: system
  content: Blah

{% for message in previous_messages %}
  {% for role, content in message %}
- role: "{{ role }}"
  content: {{ content | dump }}
  {% endfor %}
{% endfor %}

- role: user
  content: {{ user_input }}
```



By the way, I am not sure what happens to the JSON format? 
https://github.com/promptfoo/promptfoo/blob/main/examples/openai-chat-history/prompt.json


Thanks for flagging, should be fixed in #1226.  I'll wait for @mldangelo to have a look before merging.

@jay0129 we just deployed a fix for this in [0.73.1](https://github.com/promptfoo/promptfoo/releases/tag/0.73.1). Thank you for flagging it!
Repository: promptfoo/promptfoo
Issue: bug: vars in defaultTest do not appear in result chart columns
Root cause: Algorithmic Error

The variable is still taken into account in the tests because and i can see that it's executed two times (once for each value in "system_prompt").
However, the variable does not appear in an other column, thus making the readability quite complex without digging in the prompt itself.

> Maybe this is a voluntary behavior since the variables for `phi3_template_prompt` and `gemma_template_prompt` aren't in the columns as well. Maybe there is a difference to make between single "global vars" and the ones that imply the generation of multiple tests

I'm not sure from where it's coming from... My best bet is that it's something around here that messed up but it's quite a bold guess : https://github.com/promptfoo/promptfoo/blob/c50eadd8de250521dd4d17762c8878443821f40f/src/commands/eval.ts#L130.
From what I understand, it's `summary` that either not contain enough information, or the information is not used down the line.

Config used (with some "useless" stuff removed for clarity) : 
```yaml
prompts:
  - id: "{{phi3_template_prompt}}"
    label: phi3_template_prompt
  - id: "{{gemma_template_prompt}}"
    label: gemma_template_prompt

defaultTest:
  vars:
    phi3_template_prompt: file://./prompts/model-templates/phi3.txt
    gemma_template_prompt: file://./prompts/model-templates/gemma.txt
    system_prompt: [
      file://./prompts/agent-system-prompts/api-event-expert/phi3.v001.txt,
      file://./prompts/agent-system-prompts/api-event-expert/phi3.v999.txt
    ]

providers:
  - id: ollama:gemma:2b
    prompts:
      - gemma_template_prompt
  - id: ollama:chat:phi3
    prompts:
      - phi3_template_prompt

tests:
  - vars:
      user_prompt: "Is there a Youth Sports event happening on 25/12/2024?"
    assert:
      - type: equals
        value: { "category": "Sports", "start_datetime": "25/12/2024 00:00", "end_datetime": "25/12/2024 23:59" }
```

Output : 
<img width="938" alt="image" src="https://github.com/user-attachments/assets/8d9cec4e-aa26-426f-9b94-35288c4d982b">

Expected output:
<img width="945" alt="image" src="https://github.com/user-attachments/assets/c6041e25-937c-415d-bfeb-7983c0b4971e">

The expected output has been made with some trickery :
```yaml
# Step 1 :
# Rename defaultTest.vars.system_prompt ==> system_prompt_global

# Step 2 : 
# Add this as a var of the test
system_prompt: "{{system_prompt_global}}"
```

Thanks for flagging! #1306 will fix this all locations - cli, webui, etc.
Repository: promptfoo/promptfoo
Issue: When using cli, is it possible to specify OpenAI Key through config file instead of environment variables?
Root cause: Documentation Deficiency

When using cli, is it possible to specify OpenAI Key through config file instead of environment variables

Hi @zhlmmc,

Yes, it is possible like so (in the config file):

```yaml
providers:
  - id: openai:chat:gpt-3.5-turbo
    config:
      apiKey: abc123
```

great, thanks!
Repository: promptfoo/promptfoo
Issue: weighted assertions
Root cause: Documentation Deficiency

![image](https://github.com/promptfoo/promptfoo/assets/1726527/5d7e7692-07a6-4299-9711-a3272702a3c1)

As stated in above image, the doc is wrong?

Also, I want to know how do I get the score and specify above which score is considered as PASS?

Hi @zhlmmc,

Thanks for the catch, looks like score should be 0.33!  I'll fix the docs.

The combined weighted score is shown in the results view for each output when you run `promptfoo view`.  The aggregate score for each prompt is included at the top of each table column.

Right now, pass/fail is determined on a per-assert basis.  If a single assert fails, there is an overall failure.  For numeric responses, you can set a pass threshold using the `threshold` property on individual asserts.  Would it help to have a pass/fail threshold per test case?  That would be pretty easy to add.

0.21.4 adds an optional threshold property to the test case object. If set, the pass/fail status of a test case is determined by whether the combined weighted score of all assertions exceeds the threshold value.

For example:

```yaml
tests:
  threshold: 0.5
  assert:
    - type: equals
      value: "Hello world"
      weight: 2
    - type: contains
      value: "world"
      weight: 1
```

If the LLM outputs `Goodbye world`, the equals assertion fails but the contains assertion passes and the final score is 0.33. Because this is below the 0.5 threshold, the test case fails. If the threshold were lowered to 0.2, the test case would succeed.
Repository: promptfoo/promptfoo
Issue: Question: Is there a way for your prompt to contain a brackeded ( {json} ) json and the vars: to ignore that ? 
GitHub labels: question

I have an llm app I want to test without making heavy changes to the prompt

Given the following prompt example:

```
**JSON Response Structure:**
The top-level JSON structure (note this is an empty example) for ALL messages is:

```{"MyTitle":"","renderType":"","Myvalue":""}```

...
other parts of the prompt
...

The question is : {question}
```

is there a way to specify and assertion yaml config that would not take the first set of brackets, basically the json definition, as a vars ?

```
- description: "Simple SQL Contains Assertions"
  vars:
    question: Can you give me the first 10 renders? 
  assert:
    - type: icontains-all
      value: ["SELECT", "FROM"]
```


Great question @SysOverdrive! Templates in our system are rendered using Nunjucks. To include JSON or any code snippets without Nunjucks interpreting them as variables, you can use the `{% raw %}` and `{% endraw %}` tags to escape the curly braces. Here's how you can do it:

```yaml
{% raw %}
{
  "MyTitle": "",
  "renderType": "",
  "Myvalue": ""
}
{% endraw %}
```

Alternatively, for shorter code pieces, you can use double curly braces within quotes:

```yaml
{{ '{{ MyTitle }}' }}
```

This way, you can ensure that the JSON structure is not interpreted as variables in your template.

For more details, you can refer to the Nunjucks documentation on [escaping code](https://mozilla.github.io/nunjucks/templating.html#raw) or the related [GitHub issue](https://github.com/mozilla/nunjucks/issues/604). I'll make sure this is reflected in our docs. 

Hope this helps

Thanks. That is exactly what I have been looking for @mldangelo . Thanks for this complete and helpful answer !
Repository: promptfoo/promptfoo
Issue: sending non-string objects through the yaml file
Root cause: Documentation Deficiency

Hi, i would like to send in complex non-string payload in my post request. Specifically, I want to create a list of lists in my chat_history and send that through my post request. Here is what that looks like

```yaml
name: 'my Tests'
prompts: 
  - 'file://../model/prompts/*.txt'
providers:
  - id: 'http://localhost:5000/api/query_no_stream'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        question: "{{question}}"
        chat_history: 
        conversation_id: empty
        scenario: my_scenario
      responseParser: 'json.answer'
tests:
  - vars:
      question: What's the eather like?
      chat_history:
        - - 'hi'
          - 'hello'
        - - 'how are you'
          - 'I am good'
      reference: No ref
```


However, when i do this, the server receives the chat_history as `{'[object Object]': None}` instead of the nested lists. I know you use nunjucks as the templating engine. I tried using `{{chat_history | dump}}` but that tends to send the whole thing as a string instead of nested lists.

What's the easiest way to accomplish what I'm trying to achieve?

@rishimalhotra2 take a look at https://www.promptfoo.dev/docs/configuration/chat/ and let us know if it doesn't solve your issue. 

#1274 introduces a fix for this behavior, making it possible to nest objects in the request.

thanks for the speedy work. incase anyone reading this in the future finds it helpful, the below yaml file works for me. i'm closing this issue.


```yaml
name: 'Tests'
prompts: 
  - 'file://../model/prompts/*.txt'
providers:
  - id: 'http://localhost:5000/api/query_no_stream'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        question: "{{question}}"
        chat_history: "{{chat_history | dump}}"
        conversation_id: empty
        scenario: this_scenario
      responseParser: 'json.answer'
tests:
  - vars:
      question: | 
        What was the first question i asked you?
      reference: TODO
      chat_history:
        - - hey. what is 1+1?
          - i'm good
        - - what's the weather like?
          - i cannot answer that
```
Repository: promptfoo/promptfoo
Issue: Error while using uploaded config.yaml
Root cause: Resource Mishandling

Got the following error while running an eval 

```
Writing environment variables to .env file...
Loaded environment variables:
entrypoint.sh: line 3: can't create /app/src/web/nextui/.env: Permission denied
cat: can't open '/app/src/web/nextui/.env': No such file or directory
Starting server...
- ready started server on 0.0.0.0:3000, url: http://localhost:3000
Listening on port 3000 url: http://0.0.0.0:3000
[1539e61f-74bd-447e-8811-576303d7e300] 1/4
[1539e61f-74bd-447e-8811-576303d7e300] 2/4
[1539e61f-74bd-447e-8811-576303d7e300] 3/4
[1539e61f-74bd-447e-8811-576303d7e300] 4/4
[31mError running database migrations: EACCES: permission denied, mkdir '/.promptfoo'[39m
Error: EACCES: permission denied, mkdir '/.promptfoo'
[90m    at Object.mkdirSync (node:fs:1372:26)[39m
    at getConfigDirectoryPath [90m(/app/src/web/nextui/[39m.next/server/chunks/6423.js:1279:41[90m)[39m
    at getDbPath [90m(/app/src/web/nextui/[39m.next/server/chunks/6423.js:60:131[90m)[39m
    at getDb [90m(/app/src/web/nextui/[39m.next/server/chunks/6423.js:67:84[90m)[39m
    at writeResultsToDatabase [90m(/app/src/web/nextui/[39m.next/server/chunks/6423.js:1669:42[90m)[39m
    at Object.index_evaluate [as evaluate] [90m(/app/src/web/nextui/[39m.next/server/app/api/eval/job/route.js:9603:63[90m)[39m
[90m    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)[39m {
  errno: [33m-13[39m,
  code: [32m'EACCES'[39m,
  syscall: [32m'mkdir'[39m,
  path: [32m'/.promptfoo'[39m
}
```

cc @mldangelo @IanCal 

https://github.com/user-attachments/assets/710c8a3b-6c38-4743-91a9-9c1125a5cedb

The evaluation keeps hovering at 100 % on the UI

@IanCal @mldangelo Can you please share an example` config.yaml` that i could use to test with running using docker and kubernetes using upload button  ?

Hey @rrbanda, first off, I want to sincerely apologize for the delay in getting back to you. I really appreciate your involvement and patience. The issue you encountered should now be resolved. If you try again with the latest version, everything should work as expected. Please don't hesitate to reach out if you run into any more issues, and thank you again for helping us improve Promptfoo!



Hello @mldangelo ,

I am using promptfoo v-0.101.2 but still facing this issue on code deployment.

![image (2)](https://github.com/user-attachments/assets/f7eae15e-9c35-4b93-bacc-aac23637a8f3)

Thanks.