zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Javascript and python options missing Edit Test Cast on web interface
Root cause: Unimplemented Feature Gap

Really enjoying promptfoo, thank you! In the web interface, I noticed that when you hit:

Edit Eval > Edit Test Case > Add Assert 

the pulldown is missing the Type options for javascript or python. If a javascript assertion is present in the promptfooconfig.yaml, then that option is present in the list, but only for that test case.



Possible related issue in the Edit Test Case web interface— on more recent evals I've run, the Vars editable field is missing and under Vars it just says "Add variables to your prompt using the {{varname}} syntax." This occurs when variables do exist and function correctly when running evals. 

Thanks so much for reporting this issue @iradeyal2! This is supported now. Apologies for the delay in getting back to you. Please feel free to comment / re-open if you are still using promptfoo and have any other issues or feedback! Thanks!
Repository: promptfoo/promptfoo
Issue: promptfoo.evaluate() does not return config info
Root cause: Interface Contract Mismatch

I found a difference when call promptfoo using cli and using Node package.
The JSON returned in cli:
![image](https://github.com/promptfoo/promptfoo/assets/1726527/91d747ec-5965-4c5d-95d2-b3382b466349)

The JSON returned in Node package:
![image](https://github.com/promptfoo/promptfoo/assets/1726527/be0328ec-1769-45ed-9c12-e5f4540f18a3)


Curious what the intention is for this request.  If you're calling `promptfoo.evaluate`, don't you already have the full config?

> Curious what the intention is for this request. If you're calling `promptfoo.evaluate`, don't you already have the full config?

yes, you are right. I can get the info from the code.
Repository: promptfoo/promptfoo
Issue: Can we write our own custom plugins and strategies?
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
Although, the default plugins and strategies are great, we can include more (or sometimes custom use-cases may demand specific plugins/strategies).

**Describe the solution you'd like**
Promptfoo documentation to provide clear instructions on how to add custom/additional plugins and strategies.

**Describe alternatives you've considered**
Beginning to explore. Exploring what changes will be required if additional plugin type is created via a ts file. What other changes will be required to ensure the new plugin is picked up and integrates well with rest of promptfoo without any regression. Would appreciate some input from devs to point me in the right direction.

Thanks!


Hi @Doitnowman,

Yes, you can write your own custom plugins and strategies! See

- [Custom Plugins Documentation](https://www.promptfoo.dev/docs/red-team/plugins/custom/)
- [Custom Strategies Documentation](https://www.promptfoo.dev/docs/red-team/strategies/custom/)

If you have additional questions or encounter any challenges, we’d love for you to join our [Discord](https://discord.gg/gHPS9jjfbs).

Thank you, @mldangelo !

Tried following the documentation you linked and get the following error.

`Error: OpenAI API key is not set. Set the OPENAI_API_KEY environment variable or add `apiKey` to the provider config.`

Does generating custom plugin tests require our own OPENAI api key? Is there any other way? Thank you!

We've tuned the plugin prompts for gpt-4o so I recommend using that model, but you can override the provider to remove the dependency on openai in the redteam section of your promptfooconfig. For example

redteam:
  provider: any-promptfoo-provider (http, python, javascript, etc.)

You can check out [this section of the code](https://github.com/promptfoo/promptfoo/blob/main/src/redteam/providers/shared.ts#L14) to see how it works.

What are you thinking about using instead? Make sure it supports json mode / structured outputs. If you have more questions feel free to put time one our calendar. 

I used our own OPENAI api key and it worked. However, what I was trying to ask is that, as per the docs, the default behavior seems to be that if the user doesn't provide a redteam provider and it's api key, then it should use promptfoo's api key to generate test prompts? See screenshot below. If this is still true, how come my redteam tests were failing until I provided my own OPENAI api key? Thanks!
![image](https://github.com/user-attachments/assets/0fb182ed-cc55-455e-a8cd-74297f1d060f)
Repository: promptfoo/promptfoo
Issue: Unable to vars and assertions from yaml file
Root cause: Documentation Deficiency

I am trying to load vars and assertions for a testcase from the same yaml file. But getting the below error  
```throw new Error(`Test case must have either assert, vars, or options property. Instead got ${JSON.stringify(testCase, null, 2)}`);```

Config:
```
prompts: [prompts.txt]
providers:
  - id: azureopenai:chat:gpt-35-turbo
    config:
      temperature: 0.0
tests: tests.yaml
```

Yaml content:
```
- question: 'abcd'
  entity: abcd
  entity_type: abcd
  topic: abcd
  context: '''abcd'''
  __expected: 'similar(0.9):abcd'
```


Found a solution

@ishikagoel5628 how did you resolve this?

`tests.yaml` is expected to be in testcase format, not CSV

For example:

```yaml
- vars:
    question: 'abcd'
    entity: 'abcd'
  assert:
    - type: similar
      threshold: 0.9
      value: abcd
```

If you're running into issues, please share your config and I can help troubleshoot!

Below is my config yaml. Note i have some models finetuned that I am evaluating. But it somehow does not like the `tests` part in the yaml
```
description: 'My first eval'

prompts:
  - "Write a tweet about {{topic}}"
  - "Write a very concise, funny tweet about {{topic}}"

providers:
  - id: 'python:eval.py'
    config:
      additionalOption: llama
  - id: 'python:eval.py'
    config:
      additionalOption: phi

tests:
  - vars:
      topic: bananas

```
Repository: promptfoo/promptfoo
Issue: Option to configure Web UI basepath
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

I'm using the Docker image available at `ghcr.io/promptfoo/promptfoo:latest` to deploy the promptfoo UI. The web application needs to be deployed to a subdirectory on the server, for example at http:\\server\foo\bar

A sub-directory deployments requires configuring the `VITE_PUBLIC_BASENAME` breifly described in #1853 In order for the change to take effect, this requires rebuilding the promptfoo Docker image and cannot be used with the `ghcr.io/promptfoo/promptfoo:latest` image.

**Describe the solution you'd like**

Make `VITE_PUBLIC_BASENAME` or something similar a configurable option that takes effect when the web application is started irrespective of the value used when building the package.

**Describe alternatives you've considered**

Rebuilding the promptfoo Docker image but I would ideally use the image available at `ghcr.io/promptfoo/promptfoo:latest` and configure it at runtime.

**Additional context**

Brief Discord [discussion](https://discord.com/channels/1153767083381903389/1315774382320123954)


Hey @rvernica,  we recently added base path support for reverse proxy deployments in PR #6758. While this doesn't provide full runtime configuration (the base path is still set at build time), we've documented the process and it should be straightforward.

You can now build a custom image with your desired base path:

```bash
docker build --build-arg VITE_PUBLIC_BASENAME=/foo/bar -t my-promptfoo .
```

Then configure your reverse proxy (Nginx, Traefik, etc.) to strip the prefix when forwarding requests.

Full instructions with Nginx and Traefik examples are available in the self-hosting docs: https://www.promptfoo.dev/docs/usage/self-hosting/#deploying-behind-a-reverse-proxy-with-base-path

This still requires building a custom image rather than using `ghcr.io/promptfoo/promptfoo:latest` directly with an environment variable. True runtime configuration would require additional architecture changes (server-side HTML injection or a container entrypoint that patches the built JS).

If runtime configuration is a hard requirement for your use case, let us know and we can discuss further. Otherwise, hopefully the documented rebuild approach works for you. Closing but please feel free to comment / re-open.
Repository: promptfoo/promptfoo
Issue: Issue in downloading promptfoo in docker
Root cause: External Dependency Breakage

**Describe the bug**
When I deployed the promptfoo in a docker env, I received some errors: "npm ERR! notarget No matching version found for @mui/core-downloads-tracker@^6.2.0.".  I have traced this package, but the version in npm is 6.1.10.  I don't know what caused it. The previous few attempts were successful except today.

**To Reproduce**
![image](https://github.com/user-attachments/assets/3b4457d3-9c06-4d85-95e7-5eb8b32fa399)


**Expected behavior**
npm ERR! notarget No matching version found for @mui/core-downloads-tracker@^6.2.0.
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information:**

![image](https://github.com/user-attachments/assets/9cae7818-7eeb-43b5-9b55-f430186d6785)

- Promptfoo version:

**Additional context**
Add any other context about the problem here.


Strange. Thank you for reporting this. The version in our package-lock.json is 6.1.10 (see [here](https://github.com/promptfoo/promptfoo/blob/main/package-lock.json#L8568)). I’ll try to reproduce the issue.

Unrelated to your issue - consider updating the Node.js version in your Docker base image from 18 to 20 or 22. Node.js 22 is now LTS, and Node.js 18 is nearing its end of life ([reference](https://endoflife.date/nodejs)).

Please let us know if it starts working again.

Im getting the same issue while testing promptfoo in google collab
<img width="1424" alt="image" src="https://github.com/user-attachments/assets/f6c3a497-6161-4f7c-a15f-01dd73d41902">


They just published version 6.2.0 nine minutes ago! See https://www.npmjs.com/package/@mui/core-downloads-tracker/v/6.2.0

@abhiabhijit and @Sak5ra, could you please try again? MUI has many subpackages, and it seems we caught npm in an intermediate state while they were rolling out the new version.

> They just published version 6.2.0 nine minutes ago! See https://www.npmjs.com/package/@mui/core-downloads-tracker/v/6.2.0
> 
> @abhiabhijit and @Sak5ra, could you please try again? MUI has many subpackages, and it seems we caught npm in an intermediate state while they were rolling out the new version.

Thanks for your help. Deploy successfully. 

Its working now, thanks @mldangelo
Repository: promptfoo/promptfoo
Issue: disable _conversation when evaluate
Root cause: Documentation Deficiency

I found the test result contains a _conversation history in result data under "vars".
![image](https://github.com/promptfoo/promptfoo/assets/1726527/410e43ea-ba62-4cbe-ae67-ff6603646a88)

I know that this feature is added to support conversation scenarios but in my case it is not needed. How can I disable this? Thanks!

73ee2cab1ebd8cd53d0d557da9b40a4f16b79b24 adds support for `PROMPTFOO_DISABLE_CONVERSATION_VAR` environment variable
Repository: promptfoo/promptfoo
Issue: Redteam custom plugin (reading from yaml file) no longer working
Root cause: Validation Gap

**Describe the bug**
Redteam custom plugins which required custom plugin definition in a separate yaml file, no longer seems to work with [0.100.6, 0.99.1] - works with 0.96.1. Haven't tested on others but seems to have broken somewhere between 0.96.1 and 0.99.1

**To Reproduce**
Create any custom plugin as per steps in the docs - https://www.promptfoo.dev/docs/red-team/plugins/custom/
Relevant screenshot of the custom plugin definition example
![image](https://github.com/user-attachments/assets/18e67b21-4206-4b23-8f40-17786ce8feb8)

**Expected behavior**
Newer version should run custom defined plugins, unless this feature was removed as planned.

**Screenshots**
N/A but please see relevant parts of the stdout below:

`"message": "Expected string, received object"
                  }
                ],
                "name": "ZodError"
              }
            ],
            "path": [
              "plugins",
              0
            ],
            "message": "Invalid input"
          }
        ],
        "name": "ZodError"
      },
      {
        "issues": [
          {
            "received": "file://REDACTED.yaml",
            "code": "invalid_enum_value",`

**System information:**

N/A

- Promptfoo version: 0.100.6, 0.99.1, possibly others

**Additional context**
Add any other context about the problem here.


Hi @Doitnowman, thanks for reporting this issue! The problem with custom plugins using a file:// protocol has been addressed in [0.101.2](https://github.com/promptfoo/promptfoo/releases/tag/0.101.2). Please update to this version, and your custom plugin should work as expected.

If the issue persists or you encounter anything else, feel free to reopen this issue or create a new one. Thanks for using Promptfoo!
Repository: promptfoo/promptfoo
Issue: Run Python from `tests.csv`
Root cause: Unimplemented Feature Gap

`tests.csv` offers a convenient way of listing many test cases. With the `__expected` field, an assertion can be specified that must hold true after a test case.
While JavaScript is supported with the `fn` keyword, Python isn't. 

On another note, it seems that `async` javascript functions don't work with `tests.csv`.
Repository: promptfoo/promptfoo
Issue: Question: Compare JSON values
Root cause: Unimplemented Feature Gap

My prompts ask the IA for JSON objects. So what I want to compare with the values of them if they are correct.

The problem is that the “diff” that promptfoo offers (or at least what I find) is that it compares the whole object and the diff is the object itself.
When it shows the test as “fail” it shows me two objects, so it is difficult to see in which value is the error.

Isn't there a more specific diff? Whether it is JSON or not, promtfoo does not show the specific part where the difference is, it shows everything itself.

Maybe there is a functionality that I am not implementing well? or is it that promptfoo does not allow to do this?

Thank you in advance! 

@devikcode can you please share a minimal config so I can reproduce your issue? Thanks! 

> The problem is that the “diff” that promptfoo offers (or at least what I find) is that it compares the whole object and the diff is the object itself.

Seeing the same. It's difficult to interpret test failures for JSON. Even more so because if an assertion fails, it doesn't show individual properties as failure or success afaict, where I'd hope to see individual property assertions separately + a diff of each value being asserted to quickly see what's different.

I end up using dev console to inspect the promptfoo results, copy the entire JSON from the failed test, copy it from a passed test, and then paste it into a web diff checker, to identify what changed. But not scaleable.

I wonder if this PR might help: https://github.com/promptfoo/promptfoo/pull/893 

Thanks for creating promptfoo!

Here is an example test for promptfoo, to show assertions used. 

```js
const expectedResult = { /* ... */ };
const exampleTest = { 
  description: "JSON example",
  provider: `file://${PROVIDER_PATH}`,
  prompt: `
    # Prompt here...
  `,
  vars: {
    someVar: "someValue",
  },
  assert: [
    {
      type: "contains-json",
      value: {
        required: [
          "foo",
          "bar",
          "propWithAReallyLongValue"
        ],
        type: "object",
        properties: {
          foo: {
            type: "string",
            const: expectedResult.foo, 
          },
          bar: {
            type: "string",
            const: expectedResult.bar, 
          },
          propWithAReallyLongValue: {
            type: "string",
            const: expectedResult.propWithAReallyLongValue, 
          }
        },
      },
    },
  ]
}

```

### Version
`"promptfoo": "^0.112.3"`
Repository: promptfoo/promptfoo
Issue: Testing providers w/ same model but diff params seems broken
Root cause: Validation Gap

According to this PR: https://github.com/promptfoo/promptfoo/pull/83 I should be able to do the following:
```
providers:
  - openai:gpt-3.5-turbo-0613:
     id: openai:gpt-3.5-turbo-lowtemp
     config:
       temperature: 0

  - openai:gpt-3.5-turbo-0613:
     id: openai:gpt-3.5-turbo-hightemp
     config:
       temperature: 1
```

However, that results in this error:

```
Error: Unknown OpenAI model type: gpt-3.5-turbo-lowtemp. Use one of the following providers: openai:chat:<model name>, openai:completion:<model name>
    at loadApiProvider (/Users/daniel/code/node_modules/promptfoo/dist/src/providers.js:123:19)
    at /Users/daniel/code/node_modules/promptfoo/dist/src/providers.js:69:24
    at Array.map (<anonymous>)
    at loadApiProviders (/Users/daniel/code/node_modules/promptfoo/dist/src/providers.js:57:42)
    at Command.<anonymous> (/Users/daniel/code/node_modules/promptfoo/dist/src/main.js:256:72)
    at Command.listener [as _actionHandler] (/Users/daniel/code/node_modules/promptfoo/node_modules/commander/lib/command.js:482:17)
    at /Users/daniel/code/node_modules/promptfoo/node_modules/commander/lib/command.js:1283:65
    at Command._chainOrCall (/Users/daniel/code/node_modules/promptfoo/node_modules/commander/lib/command.js:1177:12)
    at Command._parseCommand (/Users/daniel/code/node_modules/promptfoo/node_modules/commander/lib/command.js:1283:27)
    at /Users/daniel/code/node_modules/promptfoo/node_modules/commander/lib/command.js:1081:27
```

Version: 0.25.1

The reason I'd like to do that, is because otherwise it's really difficult to gauge in the result which provider had which settings:

![Screenshot 2023-10-24 at 16 08 24](https://github.com/promptfoo/promptfoo/assets/489843/0d1c12f0-7bed-4845-ac41-38070d9b98bd)

Is there another mechanism (like a label or such) I can use to show a specific model name in the result for the case where the model is the same, but only the configs are different?

Hi @dmahlow, I wonder if there's a different part of your setup that's causing this.  This config works without error for me:

```yaml
prompts: [prompts.txt]
providers:
  - openai:gpt-3.5-turbo-0613:
     id: openai:gpt-3.5-turbo-lowtemp
     config:
       temperature: 0

  - openai:gpt-3.5-turbo-0613:
     id: openai:gpt-3.5-turbo-hightemp
     config:
       temperature: 1
tests:
  - vars:
      var1: test 1
  - vars:
      var1: test 2
  - vars:
      var1: test 3
```

As well as [this example](https://github.com/promptfoo/promptfoo/tree/main/examples/gpt-3.5-temperature-comparison).

Can you double check on your end or share other details of your setup?

I agree that a `label` field would be nice to have and a bit clearer to use.
Repository: promptfoo/promptfoo
Issue: Cosine similarity issue
Root cause: Documentation Deficiency

No matter how un-similar the values are, I get high similarity scores. (consistently above 0.7)

Example: 

Output value: "Reach out by email to check in and ensure everything is going smoothly."

Comparison value: "Hi Mom"

Results:

<img width="1150" alt="Screenshot 2023-10-24 at 4 01 48 PM" src="https://github.com/promptfoo/promptfoo/assets/17748596/795c5b42-4a86-4055-b482-1d11c5b71f64">

 

I tried this manually with the following:

```
curl https://api.openai.com/v1/embeddings   -H "Content-Type: application/json"   -H "Authorization: Bearer $OPENAI_API_KEY"   -d '{
    "input": "Reach out by email to check in and ensure everything is going smoothly.",
    "model": "text-embedding-ada-002"
  }' -o s1.json

curl https://api.openai.com/v1/embeddings   -H "Content-Type: application/json"   -H "Authorization: Bearer $OPENAI_API_KEY"   -d '{
    "input": "Hi Mom",
    "model": "text-embedding-ada-002"
  }' -o s2.json
```

```py
import json
from math import sqrt

def dot_product(X, Y):
    return sum(x*y for x, y in zip(X, Y))

def magnitude(vector):
    return sqrt(sum(pow(element, 2) for element in vector))

def cosine_similarity(X, Y):
    return dot_product(X, Y) / (magnitude(X) * magnitude(Y))

def main():
    with open('s1.json') as f:
        s1 = json.load(f)['data'][0]['embedding']
    with open('s2.json') as f:
        s2 = json.load(f)['data'][0]['embedding']
    print(cosine_similarity(s1, s2))

if __name__ == '__main__':
    main()
```

And the cosine similarity is indeed 0.7699.  It appears to be working as intended!  You'll have to tune the [threshold](https://promptfoo.dev/docs/configuration/expected-outputs#similarity) value accordingly.

Interesting, thanks for verifying. Will need to look into other methods of checking similarity.
Repository: promptfoo/promptfoo
Issue: Cannot use JSON prompts containing {{ or ---
Root cause: Validation Gap

I'm trying to test two raw OpenAI prompts provided as JSON. Both prompts contain one system message and multiple user messages.

Some of the messages in the prompts contain double curly brackets, which trips up Nunjucks. I tried to find a way to not run templating at all, but couldn't find any, so I just hacked `renderPrompt` to skip the templating.

Next issue: some of the prompt messages contain the string `---`, which trips up the multi-prompt splitter. I moved the two prompts to two separate files, and that got me past the first parsing error, but it now fails when running `model-graded-closedqa`, since apparently there's some splitting happening somewhere in there too.

Sorry, it looks like the issue in `model-graded-closedqa` is somewhere else. It looks like it might be failing to properly double-escape the original prompt when including it in the LLM message.

I'm getting this:

```
Error: Chat Completion prompt is not a valid JSON string: SyntaxError: Expected ',' or '}' after property value in JSON at position 751
```

The snippet around position 751 is:

```
The merge request is titled \\"add scala
```

Position 751 is the quote, so there's definitely an issue with not properly double-quoting a quote. I'm guessing it should be

```
\\\"
```

Hey @iamjackg, thanks for the report.  Definitely seems like something is going wrong with escaping.

promptfoo does double-escape the original prompt when including it: https://github.com/promptfoo/promptfoo/blob/main/src/matchers.ts#L311-L313 

So I would guess that the string may have _already_ escaped when it got there.  As a temporary workaround, you can remove the nested escaped string (or use single quotes).

Probable fix landed in #242
Repository: promptfoo/promptfoo
Issue: Question: Definition for Model-graded metrics

Hi,

do you have a link or in detail description (formula) how exactly 
- context-recall
- context-relevance
- context-faithfulness (not heard so far, and for what i get time by time negative values)

are calculated when used in the config?

Thanks in advance!


https://github.com/promptfoo/promptfoo/blob/main/src/assertions/contextFaithfulness.ts