zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Trying to OR between multiple asserts
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I have an assertion where I want to check whether the response contains either one set of strings or another set of strings. So, it's basically like an OR between two `icontains-all` assertions. There is an existing way to do this with promptfoo but it's an awkward workaround and easy to make mistakes with.

**Describe the solution you'd like**
```yaml
    assert:
      - type: assert-any
        assert:
          - type: contains
            value: "string A"
          - type: contains
            value: "string B"
```
or something similar.

**Describe alternatives you've considered**
Currently, the docs say:
>Example - if one of two assertions need to pass or 50%:
>```yaml
>tests:
>  - description: 'Test that the output is cheap or fast'
>    vars:
>      example: 'Hello, World!'
>    assert:
>      - type: assert-set
>        threshold: 0.5
>        assert:
>          - type: cost
>            threshold: 0.001
>          - type: latency
>            threshold: 200
>```

So, I would have to do this:
```yaml
    assert:
      - type: assert-set
        threshold: 0.5
        assert:
          - type: contains
            value: "string A"
          - type: contains
            value: "string B"
```

and if I add another assertion, I have to update the threshold by calculating 1/(number of assertions):
```yaml
    assert:
      - type: assert-set
        threshold: 0.333 # rounded to 3 decimal places :)
        assert:
          - type: contains
            value: "string A"
          - type: contains
            value: "string B"
          - type: contains
            value: "string C"
```

It's also not obvious what I'm doing here just by reading the assertion so it's not readable.

**Additional context**
Promptfoo v0.85.0


Hi @mrafnadeem-apimatic, thanks for the suggestion! As a workaround and for your use case, I recommend using a custom JavaScript or Python assertion. For example: 

```yaml
assert:
  - type: javascript
    value: |
      // Check if the output contains either "string A" or "string B"
      const containsA = output.includes("string A");
      const containsB = output.includes("string B");
      return containsA || containsB;
```

You can also save them in a separate file and reference it using the `file://` prefix:

```yaml
assert:
  - type: python
    value: file://path/to/your_assertion.py
```

Check out https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/ and https://www.promptfoo.dev/docs/configuration/expected-outputs/python/ 

I'll close this issue for now. Let us know if the workaround is not sufficient and feel free to open a new issue if you have further questions. Thanks again for a great question.
Repository: promptfoo/promptfoo
Issue: HTTP provider URL as template
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I wanted to use the HTTP provider for setting tests against an HTTP endpoint but ran into a limitation, currently the URL is configured as the `id` of the provider, eg: 
```yaml
providers:
  - id: 'https://example.com/generate'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        myPrompt: '{{prompt}}'
      responseParser: 'json.output'
```
but the `id` is not a template so it cannot be replaced during runtime. The usecase would be that during a CI pipeline execution the endpoint can be changed to a specific environment (qa|dev|etc).

I noticed that https://github.com/promptfoo/promptfoo/issues/1552 allows to use envvars in templated properties, but this does not work for the HTTP provider.

**Describe the solution you'd like**
Something like 
```
providers:
  - id: 'api-endpoint'
    url: 'https://{{ env.MYHOST }}:{{ env.MYPORT }}/query'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        myPrompt: '{{prompt}}'
      responseParser: 'json.output'
```
or simply be able to do
```
providers:
  - id: 'https://{{ env.MYHOST }}:{{ env.MYPORT }}/query'
```    


**Describe alternatives you've considered**
- writing own Custom JS or Python, but seems such a simple change that should be part of the HTTP Provider
- Generating the YAML for the test before execution, via `sed` or similar



Hi @AlexRRR, thanks for raising this issue! We’ve just merged a PR that adds templating support for provider URLs (#1747). This feature will allow you to use templated values in the `id` field, similar to your examples:

```yaml
providers:
  - id: 'https://{{ env.MYHOST }}:{{ env.MYPORT }}/query'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        myPrompt: '{{prompt}}'
      responseParser: 'json.output'
```

This update will be included in the next release (tomorrow). Please let us know if you have any other feedback or run into issues!

Hi @AlexRRR, we  just shipped the templating support for provider URLs in [0.90.0](https://github.com/promptfoo/promptfoo/releases/tag/0.90.0).  Let us know if you have any more feedback or run into any issues. Thanks again for the feature suggestion!
Repository: promptfoo/promptfoo
Issue: Add support for `bedrock:ai21.jamba-*:0` models
Root cause: Unimplemented Feature Gap

May we have support added for AI21 Jambda 1.5 models in the Bedrock provider? Looking to test these without needing to build a custom provider.

Specific model ids:

- `ai21.jamba-1-5-large-v1:0`
- `ai21.jamba-1-5-mini-v1:0`

Thank you!

Hi @thomasgtaylor, I just shipped support for AI21 Jamba 1.5 models in the Bedrock provider in [0.90.0](https://github.com/promptfoo/promptfoo/releases/tag/0.90.0).  Let me know if you encounter any issues or have more feedback. Appreciate your help in making the project better!

This is awesome! Thank you Michael!
Repository: promptfoo/promptfoo
Issue: Transforms are run for every provider, even if they are specified under a single provider
Root cause: Algorithmic Error

**Describe the bug**
I want to transform output from different providers differently, so I've written a provider transform

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

promptfoo provider config:
```
providers:
  - id: "openai:chat:gpt-4o-mini"
    prompts:
      <...>
  - id: "anthropic:messages:claude-3-5-sonnet-20240620"
    transform: file://claude_transforms.py:special_claude_transform
    prompts:
      <...>
```

The transform:
```
def special_claude_transform(output, context):
    print(context['prompt']['label']) # <- will be run for any prompts defined under openai and for anthropic
    return context['prompt']['label'] + output
```

**Expected behavior**
I expect that a transform defined only under one provider will only be applied to prompts defined for that provider

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information (please complete the following information):**

- Promptfoo version: `"promptfoo": "^0.89.4"`

**Additional context**
Add any other context about the problem here.


Thanks for reporting this @bfdykstra. It seems like a real bug, and I’ll look into it. And thank you for the configuration examples you provided. We’ll work on a fix soon. Thanks for helping improve Promptfoo!

Thank you!

I tried reproducing this issue using a similar setup, but the transforms were only applied to the respective providers as expected. Here's the configuration I used:

```yaml
prompts:
  - id: 'Upper case: {{query}}'
    label: prompt1
  - id: 'prefix prompt {{query}}'
    label: prompt2
providers:
  - id: echo
    label: echo1
    transform: output.toUpperCase()
    prompts:
      - prompt1
  - id: echo
    label: echo2
    transform: '`prefix: ${output}`'
    prompts: 
      - prompt2
tests:
  - vars:
      query: injection
```

Could you please provide a bit more detail about your configuration or any additional context that might help us reproduce the behavior you're seeing? Thanks! 

hmm let me check again, may have been solved by #1757 ?

Yup solved there!
Repository: promptfoo/promptfoo
Issue: Allow Turso remote databases
Root cause: Configuration Error

It appears that the `turso` driver is already used, but because the credentials are hard-coded to be read from the file, it is not possible to use Turso hosted databases.

Would it be possible to modify this config to read config from the environment first, and then fallback to the local database?

https://github.com/promptfoo/promptfoo/blob/1b55a15e8019d481cd080ce9d3a151a35ef113d8/drizzle.config.ts#L4-L12

Docs https://orm.drizzle.team/learn/tutorials/drizzle-with-turso

```ts
import { defineConfig } from 'drizzle-kit';

export default defineConfig({
  schema: './src/db/schema.ts',
  out: './migrations',
  dialect: 'sqlite',
  driver: 'turso',
  dbCredentials: {
    url: process.env.TURSO_CONNECTION_URL!,
    authToken: process.env.TURSO_AUTH_TOKEN!,
  },
});
```

So the final change may look like this:

```ts
export default defineConfig({
  dialect: 'sqlite',
  driver: 'turso',
  schema: './src/database/tables.ts',
  out: './drizzle',
  dbCredentials: {
    url: process.env.TURSO_CONNECTION_URL ?? `file:${getDbPath()}`,
    authToken: process.env.TURSO_AUTH_TOKEN,
  },
});
```

Hi @moltar, generically overriding the database path and credentials would be a great addition to promptfoo, and we’d welcome a PR to implement this change. If you decide to proceed, please also update [`src/envars.ts`](https://github.com/promptfoo/promptfoo/blob/main/src/envars.ts) and the [self-hosting documentation](https://www.promptfoo.dev/docs/usage/self-hosting/#eval-storage) to include the new environment variables. Let me know if you have any questions!

@moltar would you like help getting this in? 

Yeah, if you provide a bit more guidance, I can definitely contribute! Basically just where things should go and the expectations on the PR. Thanks.
Repository: promptfoo/promptfoo
Issue: Configuration as TypeScript
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**

YAML is ... not great (politely speaking). 

**Describe the solution you'd like**

Because we already have types for the test file, wouldn't it be great to write the test files in TS directly?

**Describe alternatives you've considered**

Writing my own generator from TS -> YAML.

**Additional context**

Can use Liftoff to support _any_ config file type: https://www.npmjs.com/package/liftoff

Tools like gulp and plopjs use it

And many other projects: https://www.npmjs.com/browse/depended/liftoff

Hi @moltar, thanks for your suggestion! We actually support TypeScript configurations already. You can check out a basic example here: [TypeScript Config Example](https://github.com/promptfoo/promptfoo/tree/main/examples/ts-config). If you have any specific improvements or additional features in mind, I’d love to hear your thoughts. Let us know if you have any questions! I am going to close the issue but feel free to comment on it or re-open if this doesn't answer your question. 

Perfect! Works great. Thank you!

Here's a [`tsx`](https://github.com/privatenumber/tsx) based example too:

```
NODE_OPTIONS="--import tsx" promptfoo eval -c src/promptfooconfig.ts              
```

Thanks for the link to liftoff, looks pretty cool
Repository: promptfoo/promptfoo
Issue: Request: OpenAI provider `OPENAI_API_BASE_URL` over `OPENAI_API_HOST`
Root cause: Architectural Constraint

When doing some hacky stuff (in https://github.com/promptfoo/promptfoo/issues/176), I am hitting an issue where `https://` is hardcoded into the OpenAI provider.

What do you think of making the default OpenAI provider `https://api.openai.com` (including `https://`)?  Then, one can override the protocol (http, https) with `OPENAI_API_BASE_URL` (instead of `OPENAI_API_HOST`).

Fwiw, this is how Ollama, Llama.cpp, and LocalAI providers already work, they include the protocol in the environment variable/default.
Repository: promptfoo/promptfoo
Issue: Unable to generate shareable url for very large evals
Root cause: Architectural Constraint

**Describe the bug**
Unable to generate share URL . Share button keeps showing infinite processing

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run the promptfoo eval 
2. Run the promptfoo view
3. Click the share button to create a shareable URL 
4. Button keeps showing infinite processing

**Expected behavior**
Shareable URL for the eval results

**Screenshots**
![image](https://github.com/user-attachments/assets/399436c6-f5a9-4e47-87f6-0708b351dcb2)


**System information (please complete the following information):**
Request URL:
https://api.promptfoo.dev/api/eval
Request Method:
POST
Status Code:
413 Payload Too Large
- Promptfoo version:
0.87.1


Out of curiosity, how large is the eval you are trying to share?  Are you able to share an example (e.g. produced by `promptfoo eval -o output.json`)

It has 12664 lines. Its size is 263MB. Its taking too long to upload on drive to share with you.

Unfortunately we can't support payloads of that size via public sharing
right now.  You could try self-hosting https://www.promptfoo.dev/docs/usage/self-hosting/ or get in touch for
our paid cloud version: https://www.promptfoo.dev/contact/

One more idea: you can set `PROMPTFOO_LIGHTWEIGHT_RESULTS=1` to strip some information from the eval result and make it much smaller.  This may make it easier to share :)

Closing this out. The public sharing API can't handle payloads that large - 263MB is way over what we support, hence the 413.

@typpo covered the main workarounds above. One addition: if you're hitting 413s on evals that are large but not *that* large, `PROMPTFOO_SHARE_CHUNK_SIZE=10` can help by breaking up the upload into smaller pieces.

The infinite spinner when this fails is bad UX (should show an actual error), but that's separate from the size limitation itself. We'll work on this in a follow up PR. Thank you so much!
Repository: promptfoo/promptfoo
Issue: Is there an option to add a custom provider configuration with variables from a vars file?
Root cause: Documentation Deficiency

It is necessary to pass a variable from the var file to custom provider, the reference to which is in the test, to the provider file
Like that: 

```
providers:
  - id: python:providers/get_existing_aws_file.py
    config:
      s3FileName: context['vars']['AWS_SUMMARY_S3_KEY']
      aws_bucket_name: context['vars']['AWS_BUCKET_NAME']

```
`s3FileName` and `aws_bucket_name` located in the `vars` folder, and specific for each test. 

I made a variable call like I do in the test file, but it didn't work. Any ideas how to do it right and is it even possible?

Hi @Starday2009, check out: https://www.promptfoo.dev/docs/providers/python/#python-script The provider function has access to the vars via context.  If you run into any issues or have more questions, feel free to reach out.

Great! Thank you! It worked, the issue was that I was trying to call the variable through the promptfooconfig.yaml, but I needed to do it inside the provider
Repository: promptfoo/promptfoo
Issue: Additional Database Support - postgreSQL
GitHub labels: Feature Request

**Is your feature request related to a problem? Please describe.**

Not really a problem, just need further customization options to align with the preferred organization infrastructure recommendations.

**Describe the solution you'd like**

I'd like the ability to configure different databases (postgreSQL) to work as the persistence layer.

- ability to configure via command line
- ability to configure via config file

**Describe alternatives you've considered**
No alternatives

**Additional context**
No additional context

Hi @carvana-holwerda, thank you for your suggestion! Support for PostgreSQL via [drizzle](https://orm.drizzle.team/docs/overview) and better data normalization are on our roadmap. We would love to hear more about your specific use case and any other features you'd like to see - feel free to schedule some time on our calendar [here](https://cal.com/team/promptfoo/intro) if you would like to chat.
Repository: promptfoo/promptfoo
Issue: evaluateOptions not working if file name != promptfooconfig
Root cause: Configuration Error

**Describe the bug**

The `evaluateOptions` is not effective if my promptfoo config file is different from  `promptfooconfig.yaml`


**To Reproduce**

1. With a `promptfooconfig.yaml` set `evaluateOptions: { repeat: 2, cache: false }` and write a test
2. Execute `promptfoo eval` or `promptfoo eval -c promptfooconfig.yaml`
3. The test is well repeated 2 times and the cache is disabled
4. Rename `promptfooconfig.yaml` into `promptfooconfig-test.yaml`
5. Execute `promptfoo eval -c promptfooconfig-test.yaml`
6. The test is executed only once, and the cache is enabled

**Expected behavior**

I expect that the evaluateOptions are effective if my promptfoo config file is different from `promptfooconfig.yaml`


**System information (please complete the following information):**

- Promptfoo version: 0.90.1




I also ran into this issue!

Can confirm that this issue still presents on the latest code. I am going to take a swing at this.

Hi @gregoryboue and @eharris128, this has been fixed in #5375 (v0.118.10). Thanks for the report and sorry it took a while!

Verified that `evaluateOptions` now works with custom config filenames like `promptfoo eval -c my-config.yaml`

10/10

Follow through is follow through.

Even it is took a while :)
Repository: promptfoo/promptfoo
Issue: Provider prompts filter not used when running evaluate via Node package
Root cause: Configuration Error

Hi! 👋 
      
Firstly, thanks for your work on this project! 🙂

Today I used [patch-package](https://github.com/ds300/patch-package) to patch `promptfoo@0.90.1` for the project I'm working on.

I'm running Node package and everything is great except `provider.prompts` filter. I have different prompts for different providers, but when running `evaluate` from TS it's not picked up. Everything works correctly when running via CLI. I dug a little bit, and it looks like `providerPromptMap` is never constructed in the `evaluate` function exposed as a Node package.

Here is the diff that solved my problem (generated by `patch-package`):

```diff
diff --git a/node_modules/promptfoo/dist/src/index.js b/node_modules/promptfoo/dist/src/index.js
index 192c8b2..24ba099 100644
--- a/node_modules/promptfoo/dist/src/index.js
+++ b/node_modules/promptfoo/dist/src/index.js
@@ -66,7 +66,7 @@ async function evaluate(testSuite, options = {}) {
             if (typeof promptInput === 'function') {
                 return {
                     raw: promptInput.toString(),
-                    label: promptInput.toString(),
+                    label: promptInput?.name ?? promptInput.toString(),
                     function: promptInput,
                 };
             }
@@ -111,8 +111,14 @@ async function evaluate(testSuite, options = {}) {
     if (options.cache === false || (options.repeat && options.repeat > 1)) {
         cache.disableCache();
     }
+
+    const parsedProviderPromptMap = prompts_1.readProviderPromptMap(testSuite, constructedTestSuite.prompts);
+
     // Run the eval!
-    const ret = await (0, evaluator_1.evaluate)(constructedTestSuite, {
+    const ret = await (0, evaluator_1.evaluate)({
+        ...constructedTestSuite,
+        providerPromptMap: parsedProviderPromptMap
+    }, {
         eventSource: 'library',
         ...options,
     });
```

<em>This issue body was [partially generated by patch-package](https://github.com/ds300/patch-package/issues/296).</em>


Thank you @baransu! We'll get a fix out for this

Hi @baransu, thanks again for raising this issue and for your detailed investigation and patch! @vsauter implemented the fix, and it has been deployed. Please pull the latest version to get the update.

Feel free to reach out if you encounter any further issues.

Thank you!
Repository: promptfoo/promptfoo
Issue: Question: Is there a way to separate test runs depending on a given set?
Root cause: Documentation Deficiency

Question: Is there a way to separate test runs depending on a given set?
For example, I want to run a test set daily for one dataset, and once a week for 10 datasets. How can I separate these runs if everything needs to be stored in one file - promptfooconfig.yaml?

@Starday2009 I would create two config files and then you can specify which one you want to run using `-c` eg:
`promtfoo eval -c weeklyconfig.yaml`
Repository: promptfoo/promptfoo
Issue: Question: Is it possible to have one test and run it on all variables from one folder?
GitHub labels: Open Source

Question: Is it possible to have one test and run it on all variables from one folder?

I’m trying to run it this way — I have one file with the test and several files with different data, but when I run it like this, only one test gets executed:

```
- vars:
    ../../vars/*
  assert:
    - type: python
      value: file://scripts/test_data_in_output.py
      metric: deterministic
```



Yes, the summarization example has an example of this: https://github.com/promptfoo/promptfoo/blob/main/examples/summarization/promptfooconfig.yaml

Hi! Thank you! @typpo 
But when I changed the configuration to this

```
tests:
  - vars:
      data: file://data/*
    assert:
      - type: python
        value: file://scripts/test_data_in_output.py
        metric: deterministic
```

My tests stopped seeing the contents of the file with the variables. The file has the following structure and has .yaml type

```
DOC_ID: "995"
ENVIRONMENT: QA
report_date: "2022"
```

I retrieve the variables in this way:

`context['vars']['ENVIRONMENT']`

Hi @Starday2009, unfortunately, the variables from your YAML file are being loaded as a single variable, which is causing the issue you're seeing. To resolve this, you can use an `inputTransform` to map the variables into an object before running the tests. More details can be found in our documentation: https://www.promptfoo.dev/docs/configuration/guide/#transforming-input-variables (thanks @MrFlounder).

Alternatively, you can handle it in a custom prompt function by using `JSON.parse` or `json.loads` on `context.vars.data`: https://www.promptfoo.dev/docs/configuration/parameters/#prompt-functions.

Happy to hop on a call if you need help implementing this!

Hi @mldangelo! The first option seemed ideal to me. However, for some reason, the transform_vars.py does not recognize the contents of vars. 
Here is what I’m trying to do

```
def get_transform_vars(vars, context):
    return {
        'file_content': vars
    }
```

with 

```
tests:
  - options:
      transformVars: file://scripts/transform_vars.py:get_transform_vars
  - vars:
      file_data: "file://vars/r191/*.yaml"
```

 and what I’m getting:
<img width="953" alt="Screenshot 2024-10-06 at 20 27 07" src="https://github.com/user-attachments/assets/9b17baca-0bbc-47d2-9345-8d542f11024a">

Maybe I'm doing something wrong?
When I tried to do it by reading the file, like this:

```
def get_transform_vars(vars, context):
    folder_path = os.getcwd() 
    file_path = os.path.join(folder_path, vars['file_path']) 

    with open(file_path, 'r') as file:
        file_content = file.read()

    return {
        'file_content': file_content,
    }
```

 it also didn't work, since the full path to the file is expected, and not /*
```

  - vars:
      file_data: "file://vars/r191/*.yaml"
```