zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Allowing `providers` to be string in `promptfooconfig.yaml`
Root cause: Validation Gap

Here is part of my `promptfooconfig.yaml`:

```yaml
---
prompts: [prompts.txt]
providers: [ollama:llama2:13b]
```

[`pre-commit`'s `check-yaml`](https://pre-commit.com/hooks.html) is flagging my `promptfooconfig.yaml`'s `providers` entry:

```
check yaml....................................................................Failed
- hook id: check-yaml
- exit code: 1

while scanning a plain scalar
  in "promptfooconfig.yaml", line 3, column 13
found unexpected ':'
  in "promptfooconfig.yaml", line 3, column 19
```

This can be fixed by making `providers` a string: `providers: "[ollama:llama2:13b]"`.

However, when I do that, now `promptfoo` fails:

```
>  promptfoo eval
node:internal/modules/cjs/loader:1090
  const err = new Error(message);
              ^

Error: Cannot find module '/Users/user/code/repo/[ollama:llama2:13b]'
Require stack:
- /Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/providers.js
- /Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/main.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1090:15)
    at Module._load (node:internal/modules/cjs/loader:934:27)
    at Module.require (node:internal/modules/cjs/loader:1157:19)
    at require (node:internal/modules/helpers:119:18)
    at /Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/providers.js:200:132
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async loadApiProvider (/Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/providers.js:200:32)
    at async loadApiProviders (/Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/providers.js:45:17)
    at async Command.<anonymous> (/Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/main.js:257:33) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/providers.js',
    '/Users/user/.nvm/versions/node/v19.9.0/lib/node_modules/promptfoo/dist/src/main.js'
  ]
}

Node.js v19.9.0
```

It would be cool if `providers` in YAML could be a string

Does it work with an array of strings?

```yaml
providers: ["ollama:llama2:13b"]
```

Oh nice thinking 🚀  yeah that does work! Thanks for thinking outside of the box
Repository: promptfoo/promptfoo
Issue: [Web Page] Display the output texts in other format like Markdown at the evaluation page
Root cause: Documentation Deficiency

**Is your feature request related to a problem? Please describe.**
No

**Describe the solution you'd like**
I'd like an option that I can see the output texts on eval web page in markdown format. Because the return text I tested is in markdown format, it will be more obvious to compare.

**Describe alternatives you've considered**
I am wondering if it is possible to add an option list that can display output texts in plain text, json, or markdown format.

**Additional context**
<img width="1722" alt="image" src="https://github.com/user-attachments/assets/6d579d6d-febd-4ced-8e98-6a71abf118bf">


Hi @tianzhich, have you tried using the Table Settings option in the eval web page? There is an option to Render model outputs as Markdown. 

<img width="970" alt="markdown" src="https://github.com/user-attachments/assets/9b247115-4bd8-4736-a153-6ea091e480e6">

If this doesn’t fully address your request or if you had something slightly different in mind, feel free to clarify, and we’ll be happy to explore it further!

@mldangelo Thank you very much. This is what I want. But the display seems not very good in the table cell. Maybe we can use a smaller font size, smaller margin between paragraphs etc...

Great suggestion! I'll add customizing the default font size / overriding styles to our backlog for future improvements.

That's great, and I'm willing to help in my free time recently.
Repository: promptfoo/promptfoo
Issue: Custom nunjucks filters for the the paramters being passed in the custom api config body
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

I want to pass a list of strings as a variable in the test case but promptfoo does not treat it as a list instead it iterates over it and run test case for each value in the provided list in vars.

**Describe the solution you'd like**

Treating the list as individual test input is now a default promptfoo behaviour, so I want to use the nunjucks filter. So that I can take the input as a string and when passing it inside the custom API config body I can covert it into a list of strings separated by commas 
```
nunjucksFilters:
  splitIntoList: ../utils/splitIntoList.js

providers:
  - id: 'https://localhost:44800/requiresCodeSample-test'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        apiName: '{{apiName}}'
        messages: '{{messages | dump}}'
        endpointNames: '{{ endpointNames | splitIntoList }}'

scenarios:
  - config:
      - vars:
          apiName: Books API
          endpointNames: "books.search,books.get,books.create,books.update,books.delete"
    tests:
      - vars:
          messages:
            - role: 'user'
              content: "Which endpoints don't need a subscription?"             
            - role: 'assistant'
              content: 'You can use the search endpoint without a subscription.'             
            - role: 'user'
              content: 'How do I use it?'                                         
        assert:
        - type: starts-with
          value: 'Yes'   
```
**Describe alternatives you've considered**

```
providers:
  - id: 'https://localhost:44800/requiresCodeSample-test'
    config:
      method: 'POST'
      headers:
        'Content-Type': 'application/json'
      body:
        apiName: '{{apiName}}'
        messages: '{{messages | dump}}'
        endpointNames: '{{ endpointNames.names | dump }}'
      responseParser: 'json.requiresCodeSample'

scenarios:
  - config:
      - vars:
          apiName: Books API
          endpointNames:
            names: [ books.search,books.get,books.create,books.update,books.delete]
```
**Additional context**
Add any other context or screenshots about the feature request here.


Hi @mehnoorsiddiqui,

Thanks for the detailed explanation! Try setting the environment variable `PROMPTFOO_DISABLE_VAR_EXPANSION=1` to prevent the list from being expanded and treated as separate items.

Let me know if that works. If not, we can look into other options.

Hi @mehnoorsiddiqui, I'm going to close this issue due to inactivity. If the suggested solution (`PROMPTFOO_DISABLE_VAR_EXPANSION=1`) didn’t resolve your problem or you still need help with this, feel free to reopen the issue with more details, and we’ll be happy to assist. Thanks again for your feedback!
Repository: promptfoo/promptfoo
Issue: Add Pricing Information for Azure OpenAI
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**

The pricing information for Azure OpenAI is currently missing. Also, OpenAI and Azure OpenAI have the same pricing structure for the East US region. 

**Describe the solution you'd like**

Can you please add the pricing details for Azure OpenAI . Even if you include only the East US region. itwould be helpful. 

Here are the pricing pages : 
- [link to Azure OpenAI pricing](https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/) 
- [link to OpenAI pricing](https://openai.com/api/pricing/)



Hi @AmineDjeghri, we’ve just shipped [0.88.0](https://github.com/promptfoo/promptfoo/releases/tag/0.88.0), which includes the pricing information for Azure OpenAI models in East US. Thanks again for bringing this up - your feedback was really helpful! I'm going to work to automate price updates more in the near future.  Feel free to check it out and let us know if anything else comes up. Thanks again!
Repository: promptfoo/promptfoo
Issue: Referencing a function call from an external file to be reusable across prompts
Root cause: Configuration Error

Is there a way to reference a tool or function call definition from an external json / yaml file for openAI?

https://github.com/promptfoo/promptfoo/blob/846993f7969a0c49926a01939835b08c44cdaf13/examples/openai-function-call/promptfooconfig.yaml#L8

Trying to move the function call schema outside of promptfooconfig.yaml

Hi @albertlieyingadrian, great question!

We supported this, but there was a bug that prevented it from working properly for the first test case. I've just pushed a fix for this in 333e61958f. I've also updated the OpenAI function call example to demonstrate how to define them externally. This will be in the next release (by Monday at the latest).

Thanks for bringing this up! Please let us know if you run into any other issues.

We just released [0.88.0](https://github.com/promptfoo/promptfoo/releases/tag/0.88.0), which includes the fix for referencing function call schemas from external files. Thanks again @albertlieyingadrian for reporting this issue! Please let us know if you run into any other bugs.
Repository: promptfoo/promptfoo
Issue: Latency assertion doesn't work for custom provider
Root cause: Validation Gap

I tried adding a custom provider and tried asserting the latency but I kept getting an error even though I already run the eval with `--no-cache`

Here is the error
```shell
Error: Latency assertion does not support cached results. Rerun the eval with --no-cache

Error: Latency assertion does not support cached results. Rerun the eval with --no-cache
    at runAssertion 
```

Here is the custom provider
```javascript
const { distance, closest } = require('fastest-levenshtein')

class CustomApiProvider {
  constructor(options) {
    // Provider ID can be overridden by the config file (e.g. when using multiple of the same provider)
    this.providerId = options.id || 'custom provider';

    // options.config contains any custom options passed to the provider
    this.config = options.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt, context) {
    const yesThreshold = this.config.threshold_yes || 4;

    const currentAddress = context.vars.current_address;
    const targetAddress = context.vars.target_address;

    const levDistance = distance(currentAddress, targetAddress);

    const result = levDistance < yesThreshold ? "Yes" : "No";

    return {
      output: JSON.stringify({
        result: result,
        reason: `The distance is ${levDistance}, so the result is ${result}.`
      }),
      tokenUsage: {
        total: 0,
        prompt: 0,
        completion: 0,
        cached: 0
      },
      cached: false,
      cost: 0
    };
  }
}

module.exports = CustomApiProvider;
```

My work around is to add delay in my processor
```javascript
    await new Promise(resolve => setTimeout(resolve, 100)); // 100ms delay
    const levDistance = distance(currentAddress, targetAddress);
```

So the latency will never be 0

Hi @albertlieyingadrian, thank you so much for the fix! I just merged your PR and really appreciate your quick work on it. I’ll ensure it gets released for you tomorrow. Please keep the bug reports and PRs coming! Thanks again for contributing and being such a great part of the community!

Thanks! Saw that https://github.com/promptfoo/promptfoo/pull/1668

Should I close this issue now? @mldangelo 

We usually wait to close them until the release is published, but by all means, you can close it whenever you like. Thanks again for the fix! 

Hi @albertlieyingadrian, I'm sure you already saw this, but commenting for visibility to others. I just released [0.88.0](https://github.com/promptfoo/promptfoo/releases/tag/0.88.0), which includes your fix. Thanks again for your contribution and for being the Promptfoo MVP of the week!
Repository: promptfoo/promptfoo
Issue: How to specify `prompt.py` in `prompfooconfig.yaml`
Root cause: Documentation Deficiency

I have a Python file just like https://promptfoo.dev/docs/configuration/parameters#prompt-functions called `prompt.py`.

In `promptfooconfig.yaml`, I have `prompts: [prompt.py]`.

However, this seems to be actually passing the source code as the prompt, not running the Python file to print the prompt.

How can I get `prompt.py` to run, when specified as a `prompts` source?

That _should_ work 🤔  What version are you using?  `promptfoo --version`

I am using the latest release, `0.24.0`

Just to share what I am seeing, with this `prompt.py`:

```python
import json
import sys


def generate_prompt(context: dict) -> str:
    return "Hello"


if __name__ == "__main__":
    print(generate_prompt(json.loads(sys.argv[1])))
```

This output (table has source code, not the printed stdout):

<img width="556" alt="screenshot of printed eval" src="https://github.com/promptfoo/promptfoo/assets/8990777/ca3b7dd6-ac85-4302-864c-10a544c8ebcb">




Oh, I see what you're saying.  The actual prompt text will vary by test case, presumably, so the column header will just have the source code itself because there isn't necessarily a generic string to represent the entire column.

Ah okay, gotchu.  Is there some way I can show the final message being passed to the LLM, to verify it's working as intended?

I also spent some time yesterday trying to solve this problem, but I couldn't. I need to use Python files instead of txt, so it's a problem for me. It seems to expect JSON, not just any YAML! You can provide JSON when YAML is expected, because JSON is YAML, but not vice-versa.


I get this: `'{vars:{question: question,document: document}}'`

However, It should be:

```
{
  "vars": {
    "question": "question",
    "document": "document"
  }
}

```


@jamesbraza The final prompt should be visible by clicking on the 🔍  icon on each cell.

@maciejbrasewicz Thanks for opening the other issue.  I'll investigate there 🤔 

Thanks @typpo, appreciate the answer! I made a docs PR to upstream that to the docs
Repository: promptfoo/promptfoo
Issue: Request: supporting semantic linefeeds or markdown as prompt input
Root cause: Unimplemented Feature Gap

I have a few prompts that are really long (500+ chars).  Dealing with that in my IDE and `prompts.txt` (without soft wrapping enabled) is a pain.  If I were to split the long line up with a single newline:

```txt
A long
long line.
```

promptfoo preserves the newline when passing it to the model.  This is a sensible behavior.

I am hoping to use newlines in a [semantic linefeed](https://rhodesmill.org/brandon/2012/one-sentence-per-line/) sense.  That way I can have a multiline prompt in `prompts.txt`, but it's passed to the model as one line.  When there's two newlines in `prompts.txt`, it'd be interpreted as one newline when passed to the model.  This works well, because the inter-prompt delimiter is `---`.

There's two ways we can support this:
- Support semantic linefeeds in `prompts.txt` with an opt-in `promptfooconfig` parameter `use-semantic-linefeeds: true`
- Support Markdown (`.md`) as an input type, and pass the _rendered` Markdown to the model
    - Aside: this would enable support of more complex text structures, than just plain text

Thank you for your consideration!

I think this will be hard to implement, and a bit of a niche use case.  I also think most people would expect markdown to be passed directly to the LLM (as that is very common), rather than rendered beforehand.  So, I'm going to close this unless there is further interest.
Repository: promptfoo/promptfoo
Issue: Can rubricPrompt reference a prompt text from an external file?
Root cause: Unimplemented Feature Gap

https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/#customizing-the-rubric-prompt

```yaml
defaultTest:
  options:
  rubricPrompt: >
    [
        {
            "role": "system",
            "content": "Evaluate the following output based on these criteria:\n1. Clarity of explanation\n2. Accuracy of information\n3. Relevance to the topic\n\nProvide a score out of 10 for each criterion and an overall assessment."
        },
        {
            "role": "user",
            "content": "Output to evaluate: {{output}}\n\nRubric: {{rubric}}"
        }
    ]
```

Trying to store the prompt in a separate prompt / json file. Wasn't able to find the reference from the docs.

Hey @albertlieyingadrian, first off, thank you so much for filing and fixing so many issues recently—it’s been awesome to see how quickly you've become a power user. Your contributions are really helping shape Promptfoo, and we're super grateful for your dedication!

Regarding your question: **No, referencing a prompt text from an external file isn't supported yet, but this would be easy to implement.** You can check out [this section of the code](https://github.com/promptfoo/promptfoo/blob/main/src/matchers.ts#L372-L373), where it would need to be integrated. We could likely use [this utility](https://github.com/promptfoo/promptfoo/blob/521/src/util/index.ts#L1140) to handle the external file loading. It might be as simple as a quick conversation with claude 3.5 in cursor (plus a few unit tests and a sentence or two in the docs). If you want to take the lead on this, I'd love the help - if not, I can get to it this week.

On a related note: many of these metrics were originally ported from [RAGAS](https://github.com/explodinggradients/ragas/tree/main/src/ragas/metrics). Since then, RAGAS has updated several metrics and added new ones. I'd love help updating Promptfoo to support all the current RAGAS metrics if you're interested.

I’m happy to help scope this out or chat further on a call if you'd like. Thanks again for all your contributions!

> if not, I can get to it this week.

@mldangelo This will be extremely useful to load external prompts!


@mldangelo Actually, I went ahead with this PR https://github.com/promptfoo/promptfoo/pull/1698
Repository: promptfoo/promptfoo
Issue: Possible to pass env variables for external prompt's path?
Root cause: Unimplemented Feature Gap

For example, I'd love to pass `env.ROOT_PATH = ${pwd}` here
```yaml
prompts:
  - file://{{ env.ROOT_PATH }}/something.txt

providers:
  - label: 'gpt 3.5 low temperature'
    id: openai:gpt-3.5-turbo
    config:
      temperature: 0.1
```


hi @albertlieyingadrian, this isn't supported yet, but you could add support for it by calling Nunjucks around [here](https://github.com/promptfoo/promptfoo/blob/main/src/util/index.ts#L1154). We’d gladly accept a PR if you’re interested in working on it. 

Here you go @mldangelo https://github.com/promptfoo/promptfoo/pull/1688

Hi @albertlieyingadrian, thank you so much for your contribution! This was just released in version [0.89.0](https://github.com/promptfoo/promptfoo/releases/tag/0.89.0). I appreciate your help in making promptfoo better. Please keep the issues and PRs coming.

Thanks! @mldangelo I also added a request for other config's keys as well https://github.com/promptfoo/promptfoo/issues/1700
Repository: promptfoo/promptfoo
Issue: Docs request: adding Reference table for `scenarios`

https://promptfoo.dev/docs/configuration/scenarios#example documents an example for `scenarios`.

My docs request is to add a [Reference](https://promptfoo.dev/docs/configuration/reference) table for `scenarios`

Update, nevermind, there's a table for it here: https://promptfoo.dev/docs/configuration/scenarios#configuration
Repository: promptfoo/promptfoo
Issue: Using PROMPTFOO_LIGHTWEIGHT_RESULTS doesn't save results .json file
Root cause: Configuration Error

**Describe the bug**
When `PROMPTFOO_LIGHTWEIGHT_RESULTS=1`, the results aren't written to the output.

**To Reproduce**

1. Run with PROMPTFOO_LIGHTWEIGHT_RESULTS=1
2. See that the output isn't written

In the new commit here: https://github.com/promptfoo/promptfoo/commit/ecfc650b03ace4fad2d7541ea6771cb2ea0edc5f#diff-7529f5d4cc084a155324df776df361b69da340c5e2d6c8da5376d9ee1eee9049R171

We are clearing the content of the config object, and below is where the `outputPath` is read.

Thanks for the report, we'll get a fix in

#1737 should fix

Hi @lone-star, We’ve shipped a fix for this issue. Can you try [0.90.0](https://github.com/promptfoo/promptfoo/releases/tag/0.90.0) and try again with `PROMPTFOO_LIGHTWEIGHT_RESULTS=1`? 

Let us know if you encounter any further issues or have additional feedback. Thanks for your help in improve promptfoo!
Repository: promptfoo/promptfoo
Issue: [Eval grouping] Is it possible organize the eval files into a folder instead of dumping them all in the root folder?
Root cause: Documentation Deficiency


@mldangelo Do you by chance know if this is possible?

@albertlieyingadrian do you mean promptfoo configs or output files? It should be possible - book time on my calendar and we can look at it together https://cal.com/michael-dangelo/30min or fix it together. 

actually I figured it out. Will close this. Thanks!
Repository: promptfoo/promptfoo
Issue: Adding config for custom assertion
Root cause: Unimplemented Feature Gap

### Question
Is it possible to add "config" for custom assertion? eg.

```yaml
assert:
  - type: javascript
    value: file://relative/path/to/script.js
```

I would love to pass certain parameters to the respective script as 3rd argument after `context`

```javascript
module.exports = (output, context, customArgs?) => {
  return output.length > 10;
};
```

### Reference
- https://www.promptfoo.dev/docs/configuration/expected-outputs/javascript/#external-script

My workaround now
```yaml
assert:
  - type: javascript
    metric: YOUR_METRIC_NAME
    value: file://relative/path/to/script.js
    config:
      argA: valueA
```


```javascript
module.exports = (output, context) => {
  // Log the assert object
  const assertionMetricName = 'YOUR_METRIC_NAME'
  const assertion = context.test?.assert?.find(assertion => assertion.metric === assertionMetricName);

  const assertionConfig = assertion?.config

  // ... 
}

```