zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Replicate official models cannot be used as of promptfoo 0.61
Root cause: External Dependency Breakage

I have been using promptfoo to test prompts against official models provided by Replicate, such as [meta/meta-llama-3-8b-instruct](https://replicate.com/meta/meta-llama-3-8b-instruct).

I attempted to run an eval today for the first time in a while and was greeted with the following error in my evaluation output after updating promptfoo to the latest version.

`[FAIL] API call error: Error: Invalid reference to model version: meta/meta-llama-3-8b-instruct:. Expected format: owner/name or owner/name:version`

I found this strange, as I use the owner/name format in my promptfoo config and have not changed it.

Here is a snippet from the config:

```yaml
  - id: replicate:meta/meta-llama-3-8b-instruct
    config:
      temperature: 0
      # continued...
```

I subsquently sought to specify a version for the model, but the [replicate API](https://replicate.com/docs/reference/http#models.versions.list) returns the following for this model:
`{"status":404,"title":"Not found","detail":"This model is an official model and does not expose a list of versions."}`

After manually walking back the promptfoo version, I'm seeing my setup work fine in `0.60` but not `0.61`. I do not see anything explicit in the changelog for `0.61` that would cause this.



Thanks for flagging, @aaronstillwell!  #1059 fixes this issue.

A fix for this was deployed in [0.68.1](https://github.com/promptfoo/promptfoo/releases/tag/0.68.1) Please let us know if you run into any other issues and thanks again for reporting it!
Repository: promptfoo/promptfoo
Issue: python and javascript prompts in subdirectories not found
Root cause: Algorithmic Error


If you specify a python or javascript prompt in a subdirectory, like so:

```
prompts:
  - file://./prompts/prompt.py
```

Results in the following error, even though the file is indeed present at `./prompts/prompt.py`:

```
> npx promptfoo@latest eval --verbose
Reading prompts from ["file://./prompts/prompt.py"]
node:fs:603
  handleErrorFromBinding(ctx);
  ^

Error: ENOENT: no such file or directory, open 'prompt.py'
    at Object.openSync (node:fs:603:3)
    at Object.readFileSync (node:fs:471:35)
    at processPythonFile (/home/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/prompts/processors/python.js:80:28)
    at processPrompt (/home/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/prompts/index.js:112:47)
    at readPrompts (/home/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/prompts/index.js:134:35)
    at resolveConfigs (/home/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/main.js:130:59)
    at runEvaluation (/home/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/main.js:546:54)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async Command.<anonymous> (/home/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/main.js:705:9) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: 'prompt.py'
}
```

Somewhere, it appears to be stripping the path of all directories and just retaining the filename, then trying to look for it in the current directory. If you move the prompt from `./prompts/prompt.py` to `./prompt.py`, it works. This is problematic though, because having to store all prompts in the current directory can get very messy.

I have tested that this is currently happening for python and javascript prompts - but not for yaml / json.

Version:
```
>  npx promptfoo@latest --version
0.68.0
```


Thanks for the report, we'll get this fixed.  A potential temp workaround may be to revert to version 0.66.0.

A fix for this was deployed in [0.68.1](https://github.com/promptfoo/promptfoo/releases/tag/0.68.1) Please let me know if you run into any other issues and thanks for reporting it!
Repository: promptfoo/promptfoo
Issue: Allow filtering test cases (inside scenarios)
Root cause: Unimplemented Feature Gap

Given the following scenario, I want to only run test on the Spanish language:

```
scenarios:
  - config:
      - vars:
          language: Spanish
          expectedHelloWorld: 'Hola mundo'
      - vars:
          language: French
          expectedHelloWorld: 'Bonjour le monde'

    tests:
      # this isn't templated
      - description: Translated Hello World from {{ language }}
        vars:
          input: 'Hello world'
      - description: Translated Good Morning {{ language }}
        vars:
          input: 'Good morning'
```

I'm unable to filter with `--filter-pattern=Spanish` because the scenarios.tests.description field is not templated. Templating that field would fix this.

Alternatives:
* Add a new --filter-test-case flag which filters scenarios.configs instead


Hi, 
I also would be interested in this feature. From https://github.com/promptfoo/promptfoo/issues/987
I read that however the "normal" filtering without variables  in the description field should work already? Or is there something missing? I tried with this code:  
 
```
scenarios:
  - config:
    - vars:
        prompt: file://./prompt.txt
        format_instructions: file://./format_instructions.txt
    tests:
      - description: 'test'
        vars:
```

but when I execute `promptfoo eval --filter-pattern 'something'` the test is executed nonetheless..
using promtfoo 0.100.5

Thanks for some advice and if the feature including templating is planned to be implemented 

Also looking for a way to filter scenarios. Both `--filter-metadata` and `--filter-pattern` seem to have no effect on tests within scenarios.
Repository: promptfoo/promptfoo
Issue: Relative paths in prompt configuration isn't resolved
Root cause: Algorithmic Error

In previous versions (tested with 0.61.0), the following configurations were accepted -

```
prompts:
  - file://../../some/path/to/a/prompt.txt
```

```
prompts:
  - prompts/prompt.txt
```

However, this is not resolved in the latest version -

```
    return binding.readFileUtf8(path, stringToFlags(options.flag));
                   ^

Error: ENOENT: no such file or directory, open 'prompt.txt'
    at Object.readFileSync (node:fs:453:20)
    at processTxtFile (/opt/homebrew/lib/node_modules/promptfoo/dist/src/prompts/processors/text.js:38:28)
    at processPrompt (/opt/homebrew/lib/node_modules/promptfoo/dist/src/prompts/index.js:117:42)
    at readPrompts (/opt/homebrew/lib/node_modules/promptfoo/dist/src/prompts/index.js:136:35)
    at resolveConfigs (/opt/homebrew/lib/node_modules/promptfoo/dist/src/main.js:129:59)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async runEvaluation (/opt/homebrew/lib/node_modules/promptfoo/dist/src/main.js:545:48)
    at async Command.<anonymous> (/opt/homebrew/lib/node_modules/promptfoo/dist/src/main.js:704:9) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'prompt.txt'
}
```

The [documentation](https://www.promptfoo.dev/docs/configuration/parameters/#prompts-from-file) suggests this should work.
I believe the source of the change was [this commit](https://github.com/promptfoo/promptfoo/commit/9bd03d29e3179f2b665c867d3cd4528b2c056264#diff-0f0b37407b21be6434ac29c4b75a5a1c31ade0668c1afafa267cba4efc13aba9).

Thanks for reporting this issue, @chrisprice. I apologize for the inconvenience caused by this bug (introduced in [PR #994](https://github.com/promptfoo/promptfoo/pull/994)). I've addressed the problem with relative paths in [PR #1066](https://github.com/promptfoo/promptfoo/pull/1066). I'll get a fix deployed later today. Please let me know if it doesn't resolve the issue for you. Thanks for your patience and thanks again for reporting this!

A fix for this was deployed in [0.68.1](https://github.com/promptfoo/promptfoo/releases/tag/0.68.1) Please let me know if you run into any other issues and thanks again for reporting it!
Repository: promptfoo/promptfoo
Issue: prompt does not support '/' in question
Root cause: Validation Gap

C:\Users\AppData\Roaming\npm\node_modules\promptfoo\dist\src\prompts\index.js:136
            throw new Error(`There are no prompts in ${JSON.stringify(prompt.raw)}`);
                  ^

Error: There are no prompts in "How many AR/VR devices xxxxx?"
    at readPrompts (C:\Users\AppData\Roaming\npm\node_modules\promptfoo\dist\src\prompts\index.js:136:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async resolveConfigs (C:\Users\AppData\Roaming\npm\node_modules\promptfoo\dist\src\main.js:129:27)
    at async runEvaluation (C:\Users\AppData\Roaming\npm\node_modules\promptfoo\dist\src\main.js:545:48)
    at async Command.<anonymous> (C:\Users\AppData\Roaming\npm\node_modules\promptfoo\dist\src\main.js:704:9)

Node.js v20.15.0


Thanks for reporting this issue, @landhu. I apologize for the trouble you're experiencing. I believe this is a duplicate of [issue #1063](https://github.com/promptfoo/promptfoo/issues/1063) and is related to your prompt file path. I am addressing the path issue in [PR #1066](https://github.com/promptfoo/promptfoo/pull/1066), which I'll get deployed later today. Please let me know if it doesn't resolve the issue for you. Thanks for your patience and for bringing this to our attention!

A fix for this was deployed in [0.68.1](https://github.com/promptfoo/promptfoo/releases/tag/0.68.1) Please let me know if you run into any other issues and thanks again for reporting it!
Repository: promptfoo/promptfoo
Issue: Yaml ref blocks not working
Root cause: Documentation Deficiency

The YAML references feature described here:
https://www.promptfoo.dev/docs/configuration/guide#yaml-references

Is currently not working. For example:

```
description: "My eval"

prompts:
  - 'What do you know about {{topic}}?'

providers:
  - "openai:gpt-4o"
  

tests:
  - vars:
      topic: new york city
    assert:
      - $ref: '#/assertionTemplates/startsUpperCase'


assertionTemplates:
    startsUpperCase:
      - type: javascript
        value: output[0] === output[0].toUpperCase()
```


Results in:
```
TypeError: Cannot read properties of undefined (reading 'startsWith')
```


Version:
```
npx promptfoo@latest --version                                                                                                                                                                                                
0.68.0
```

It should work if you make `startsUpperCase` not an array:

```yaml
description: "My eval"

prompts:
  - 'What do you know about {{topic}}?'

providers:
  - "openai:gpt-4o"


tests:
  - vars:
      topic: new york city
    assert:
      - $ref: '#/assertionTemplates/startsUpperCase'


assertionTemplates:
    startsUpperCase:
      type: javascript
      value: output[0] === output[0].toUpperCase()
```

I'll fix the example!

Thanks a lot!
Repository: promptfoo/promptfoo
Issue: Chaining Prompts
Root cause: Documentation Deficiency

Hi, I like promptfoo.  It feels natural that I should be able to:

prompts:
  - [ "Tell me a story", "Summarize the following: {{outputs[-1]}}", "Translate the following to Spanish {{outputs[1]}}\n{{outputs[0]}}]


Is there a way to accomplish this already?  would this be difficult to implement?

You may be interested in [`_conversation`](https://promptfoo.dev/docs/configuration/chat/#using-the-conversation-variable) or [`storeOutputAs`](https://promptfoo.dev/docs/configuration/chat/#using-storeoutputas), which should support this use case.
 

hmm, not quite. Unless I can storeOutputAs on the prompt itself. but thanks anyway.  

With `_conversation`, you can access previous outputs like `_conversation[0].output` and `_conversation[1].output`.  The last output is `_conversation[_conversation.length - 1].output`.  As far as I can see from your original example, this covers all the variables you need and you can use them in the prompt.

LMK if you have any more questions about this!
Repository: promptfoo/promptfoo
Issue: [Web UI] Support `scenarios` yaml config option
Root cause: Unimplemented Feature Gap

https://github.com/promptfoo/promptfoo/assets/17098959/38227a4a-9501-48a9-b527-6ce1a65959b5

I have the following YAML config file that defines the eval I want to run. I want to be able to run this via the Web UI:
```yaml
description: 'Test'
prompts:
- file://src/providers/test/prompt.txt
providers:
- id: 'python:src/providers/test/openai/translate.py'
  label: 'GPT-4o'
  config:
    model: 'gpt-4o'
- id: 'python:src/providers/test/openai/translate.py'
  label: 'GPT-3.5-Turbo'
  config:
    model: 'gpt-3.5-turbo'

scenarios:
- config:
  - description: English
    vars:
      language: en (English)
  - description: Spanish
    vars:
      language: es (Spanish)
  - description: French
    vars:
      language: fr (French)
  - description: German
    vars:
      language: de (German)

  tests:
  - vars:
      tone: Expert
  - vars:
      tone: Daring
  - vars:
      tone: Persuasive
  - vars:
      tone: Playful
  - vars:
      tone: Sophisticated
  - vars:
      tone: Supportive
```

I don't think the `scenarios` field gets saved, which would happen [here](https://github.com/promptfoo/promptfoo/blob/b4267a03b392f386ce95be6e43565d12bcab97c8/src/web/nextui/src/app/setup/YamlEditor.tsx#L39-L45)


Hi @chrismaltais, Thanks for reporting this issue and providing detailed information! We've identified the problem with the `scenarios` field not being saved. A fix is currently on the way.  I'll update you once the fix is deployed. Your feedback is invaluable in helping us improve the Web UI. Thanks again!

This is published in 0.68.2.
Repository: promptfoo/promptfoo
Issue: Ways to pass config to custom providers
Root cause: Documentation Deficiency

I have a custom provider for together.ai, I'd like to have multiple instances using different models. 

How can I pass config into multiple different instances of the same provider?

Hi @lox,

Give this a try:

```yaml
providers:
  - customProvider.js:
      id: custom-provider-hightemp
      config:
        temperature: 1.0
  - customProvider.js:
      id: custom-provider-lowtemp
      config:
        temperature: 0
```

customProvider.js:
```js
const fetch = require('node-fetch');

class CustomApiProvider {
  constructor(options) {
    // Provider ID can be overridden by the config file (e.g. when using multiple of the same provider)
    this.providerId = options?.id || 'custom provider';

    // options.config contains any custom options passed to the provider
    this.config = options?.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt) {
    const body = {
      model: 'text-davinci-002',
      prompt,
      max_tokens: parseInt(this.config?.max_tokens, 10) || 1024,
      temperature: parseFloat(this.config?.temperature) || 0,
    };
    // ...
}

module.exports = CustomApiProvider;
```

I'll update the custom provider example to make this clearer.

Many thanks! This is perfect.
Repository: promptfoo/promptfoo
Issue: how to save a session in custom python provider
Root cause: Documentation Deficiency

Our LLM does not support API_KEY
So, I write a custom python(python requests) to login.
But I don't login again and again in eval. I just want to log in once, save the session, and have this session behind it.
How can I do.

Here is my simple yaml:
`providers:
  - id: "python:my_chat.py"
    label: "my_chat"
    config:
`


my_chat.py:

`def call_api(prompt, options, context):
    config = options.get("config", None)
    additional_option = config.get('additionalOption', None)
    user_variable = context['vars'].get('userVariable', 'user_a')
    login(user_variable)
`

Hi @landhu, can you share more about your integration? Which LLM are you using? Have you tried to persist your session in your custom python provider? 

@mldangelo I am a QA, LLM was secondary encapsulation by Dev, we just access it by login(user/password). 
**Have you tried to persist your session in your custom python provider?**
  --  I don't know how to persist. do you have demo? I tried put session in enviroment. like: os.environ['OWNER_A_AUTH']. But it's not working.

Hi @landhu, I'm sorry you're having trouble. `os.environ['OWNER_A_AUTH']` is unlikely to work unless you have something that is specifically checking for that env var. Here are a few general tips:

1. Try searching online for how to persist sessions in Python.
2. Check if there's a header in your request that you can reuse. 
3. Consider converting your username and password into an auth token programmatically.
4. You can also ask a language model like ChatGPT or Claude for help with this. I did and got many code examples. 

Good luck! Please let us know if you have any other issues specific to promptfoo.
Repository: promptfoo/promptfoo
Issue: Error: Please support "Similarity" on the fly with library instead of making a call with AZURE_OPENAI_API_KEY
Root cause: Unimplemented Feature Gap

Team, Our company does not allow us to use AZURE_OPENAI_API_KEY.

When I try it comes back with - Key based authentication is disabled.

If you can do something like what Spacy does it would be great to do from the library itself.

Or I am also fine if you could support something like custom API with JS you do to make calls. E.g.   - id: customApiProvider.js.

Waiting for your response. Your help is highly appreciated as this is needed for our high priority project with a tight deadline.



Hi @chittamitgit, thanks for reaching out. I'm not entirely sure how you want to authenticate without an API key. Could you please provide an example of what you mean with Spacy? 

For custom providers, please check out our [custom provider example](https://github.com/promptfoo/promptfoo/tree/main/examples/custom-provider). Additionally, you can set an API key in the provider config. More details can be found [here](https://www.promptfoo.dev/docs/providers/openai#configuring-parameters). 

Re. supporting different similarity assertions - You can define custom assertions too. See https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript 


I hope this helps! Please feel free to re-open if this doesn't resolve your issue. 

I mean to have required similarity check library available as part of the npm promtfoo install.  And we need not got to Open AI or Azure to help us in making similarity check.

Sharing a Sample code of "spacy".

import spacy_universal_sentence_encoder

nlp = spacy_universal_sentence_encoder.load_model('en_use_lg')

String promptOneTextOutput= nlp ("My Actual Output")

String promptTwoTextOutput= nlp ("My Expected output")

print (promptOneTextOutput.similarity(promptTwoTextOutput))

if text-embedding-3-large model of openAI or Azure can be supported by custom API is also fine where I can use the azure client secret and client ID. Can you please guide if it is possible to use this way? 

I am expecting something like this below where in assert we can inject a provider as custom API to support above grant-type as client credential way.
prompts:
  - prompt.txt

providers:
  - id: customApiProvider.js
    label: 'My custom provider'

tests:
  - description: 'Test if output is equal to the expected value'
    vars:
      example: 'Hello'
    assert:
      - type: similar
        value: 'Hola'
        **_provider: customApiProviderForSimilityCheck.js_**
        threshold: 0.8
Repository: promptfoo/promptfoo
Issue: Is there a way to run 1 eval N times?
Root cause: Documentation Deficiency

There are some evals that I'd like to benchmark at N == 100 on the same prompt.  Is there a cleaner way to do that without having to write an eval file that repeats the eval 100 times?

If you pass `--repeat 100` to `promptfoo eval` it will repeat the eval for you so you don't have to copy it 100x. The output is not exactly the "cleanest" (it's still the same test over and over), but at least your config will be clean.
Repository: promptfoo/promptfoo
Issue: prompts from file different behavior than prompts from config
Root cause: Documentation Deficiency

Loading prompts from a file asks them off the model all at once (even when using separator). 
Loading prompts as listed out in the config file asks them individually.

The first gives short responses, the second gives the full response. I prefer the full response. 

I believe I can get the same results by loading each prompt in a separate file - but I'd prefer to just load them via file and have them all asked separately. (Unless I'm missing something).

Thanks.

Hey @aisonym, I don't fully understand.  Can you give an example of the difference in output?

Thanks for your response. New to promptfoo.

What I thought was two distinct issues is maybe just one (or I'm not fully understanding). 
Described here in more detail - I believe it's related to put a label on a set of prompts instead of this being an issue of loading prompts in a file vs. in the config file.

[](https://github.com/promptfoo/promptfoo/issues/1076)

FWIW - I think fine to close this item.
Repository: promptfoo/promptfoo
Issue: assert type similar should consider custom provider js file
Root cause: Documentation Deficiency

Hi Team, I see there is a example where similar can use file like customprovider.js but when I try to use it comes back with a error OpenAI API key must be set for similarity comparison.

Please advise how to use custom js file.

I basically wants to use client credentials like client ID and secret to get a access token and inject to make a call to end point https://the-***.openai.azure.com/openai/deployments/**textembedding/embeddings?api-version=2024-**

Please guide how to use. 

I see sample example in GIT site is below:

prompts:
  - 'Translate this to French: {{message}}'

providers:
  - openai:chat:gpt-3.5-turbo

tests:
  - vars:
      message: Hello world
    assert:
      - type: similar
        threshold: 0.25
        value: 'Greetings earth'
        provider: ./customProvider.js


My use would be:

prompts:
  - 'Translate this to French: {{message}}'

providers:
  **_- id: customApiProvider.js_**

tests:
  - vars:
      message: Hello world
    assert:
      - type: similar
        threshold: 0.25
        value: 'Greetings earth'
        **_provider: customEmbeddedApiProvider.js_**

Your help is highly appreciated.


Hi @chittamitgit, you can use Azure OpenAI embedding models for the similarity grader by overriding the `embedding` grader.  There's an example [here](https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/#multiple-graders) with details on how to configure the provider with client credentials [here](https://promptfoo.dev/docs/providers/azure/#using-client-credentials).

On the other hand, if you implement a custom provider yourself, you will have to import the appropriate azure packages or logic to handle client credentials in your implementation.

Thank you @typpo . I am able to use custom api to get the response with azure client credentials with below example where customApiProvider has the logic with client credentials.
prompts:
  - 'Translate this to French: {{message}}'

providers:
  - id: customApiProvider.js.

But if I want to use assert type "similar", how to make config file to make it run from custom api js file like above config I do?

Please guide if below config is correct.
assert:
type: similar
threshold: 0.25
value: 'Greetings earth'
**provider: customEmbeddedApiProvider.js**



Team: feel free to close. I am able to use with custom API now. Thank you for your time and help!