zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Does 'providers' support customized LLM Model service?
Root cause: Documentation Deficiency

Hi team,

I'm trying to use promptfoo to test my own LLM Model for prompts, but I failed to find the example/cases or userguide how to setup `promptfooconfig.yaml`. For example, my LLM Model service provide api like https://${host}:${port}/${api}, is it possible?

Thanks so much!

For starters, what is your LLM service API, is it custom?  Does it match one of these providers: https://promptfoo.dev/docs/providers/?

https://promptfoo.dev/docs/providers/custom-api details how to make a custom provider.

From there, check `promptfoo init` to create the initial config: https://promptfoo.dev/docs/getting-started

Thanks @jamesbraza!  @ooooona the Custom APIs documentation is the way to go - feel free to follow up if anything is unclear.

Thanks @jamesbraza and @typpo, I want to test prompt with my own finetune Code-Llama Model, which I wrapped with my python code, providing v2 inference api like: _http://localhost:8080/v2/models/{model_name}/infer_. 

[current providers](https://promptfoo.dev/docs/providers/) do not match my case, seems that I have to use custom-api?

I need to implement call my v2 inference api, like _http://localhost:8080/v2/models/{model_name}/infer_, in `callApi: (prompt: string, context: { vars: Record<string, any> }) => Promise<ProviderResult>`? 

I will try this way, and will let you know if it works or not.

That is correct.  You could either use a custom API module, or if you're willing to conform to a predefined format you can use a [webhook provider](https://promptfoo.dev/docs/providers/webhook), which sends an HTTP POST request `{prompt: string}` and expects `{output: string}` in return from your server.
Repository: promptfoo/promptfoo
Issue: output context truncated in python assertions
Root cause: Configuration Error

**Describe the bug**
It looks like all output may be truncated, even when running something like 'promptfoo eval -o output.txt'

**To Reproduce**
config:
```
description: "My eval"
prompts:
  - "Create a numbered list of 200 birds that are native to North America."
providers:
  - "openai:gpt-4o-mini"
tests:
  - assert:
    - type: python
      value: file://assert.py
      config: 
        foo: 10
```

assert.py
```
from typing import Union, Dict, Any

def get_assert(output: str, context) -> Union[bool, float, Dict[str, Any]]:
    with open('/Users/robertlaabs/Desktop/output_len_test.txt', 'wt') as f:
        f.write(str(len(output)))
    with open('/Users/robertlaabs/Desktop/output_test.txt', 'wt') as f:
        f.write(output)
    return True

```


**System information (please complete the following information):**

- Promptfoo version: 0.91.1


Solved by settings max_tokens in the config
Repository: promptfoo/promptfoo
Issue: Cannot set custom provider for LLM grader override
Root cause: Validation Gap

As per the docs https://www.promptfoo.dev/docs/configuration/expected-outputs#overriding-the-llm-grader
We should be able to use custom providers for the LLM grading.
However, when setting custom provider via any of the methods (defaultTest.options and assertion.provider), the following error is thrown

`Error: Invariant failed: Provider supplied to llm-rubric must have an id
at invariant (/usr/local/share/npm-global/lib/node_modules/promptfoo/node_modules/tiny-invariant/dist/tiny-invariant.cjs.js:14:11)
at getGradingProvider (/usr/local/share/npm-global/lib/node_modules/promptfoo/dist/src/matchers.js:50:38)
at matchesLlmRubric (/usr/local/share/npm-global/lib/node_modules/promptfoo/dist/src/matchers.js:118:31)
at runAssertion (/usr/local/share/npm-global/lib/node_modules/promptfoo/dist/src/assertions.js:383:55)
at runAssertions (/usr/local/share/npm-global/lib/node_modules/promptfoo/dist/src/assertions.js:49:30)
at Evaluator.runEval (/usr/local/share/npm-global/lib/node_modules/promptfoo/dist/src/evaluator.js:182:74)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async /usr/local/share/npm-global/lib/node_modules/promptfoo/dist/src/evaluator.js:418:25`

Providing the id, as suggested by the error does not matter.
The same custom provider works fine when set as the configuration file's "providers" property. 

Below is the promptfooconfig.yaml file that reproduces the issue for me.

```yaml
prompts: [prompt.yaml]
providers: [../../providers/custom.js]
defaultTest:
  assert:
    - type: llm-rubric
  options:
    provider: [../../providers/custom.js]
    rubricPrompt:
      - role: system
        content: >-
          Grade the output by the following specifications, keeping track of the points scored:

          Did the output include code? +1 point
          Was the output in markdown format? +1 point
          Is the output relevant to the {{ query }}? +1 point

          Calculate the score and pass the test if total points is greater than 2.5.
          Output must be in the following JSON format:
          {pass: true, score: number, reason: string}
      - role: user
        content: 'Output: {{ output }}'
tests: tests/user-prompts.csv
```

The workaround I have found is to set provider as a string.
Setting the provider as object and providing id as below, does not work.

```yaml
defaultTest:
  assert:
    - type: llm-rubric
  options:
    provider: 
      ../../providers/custom.js:
        id: 'evaluator'
```

Hi @AnkitTheCoder, a bit late to this issue but the correct setup would be to set `provider: ../../providers/custom.js`, as you found.  The `provider` property on [Assertion](https://promptfoo.dev/docs/configuration/reference#assertion) is a string.
Repository: promptfoo/promptfoo
Issue: Output provider labels do not match up with the config
Root cause: Algorithmic Error

**Describe the bug**

In these images you can see that the label does not match the config

![image](https://github.com/user-attachments/assets/d5c8203c-7805-4d58-8c02-6d913be470bc)

![image](https://github.com/user-attachments/assets/e8a837e3-5f2f-47e3-8aeb-a3971b055f88)


**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

Here's my provider config:
```yaml
providers:
  - id: file://provider_structured_response.py
    label: claude-3-5-sonnet
    config:
      provider: anthropic
      model: claude-3-5-sonnet-20240620
      model_args:
        temperature: 0.0
  - id: file://provider_structured_response.py
    label: openai_gpt-4o
    config:
      provider: openai
      model: gpt-4o
      model_args:
        temperature: 0.0
```

Run it a few times. About half of the time it is correct and half the time wrong



**System information (please complete the following information):**

- Promptfoo version: 0.92.0

**Additional context**
Before I added the labels, both of the results were merged into a single output column


This is worse than I realised - not only do the label no match up with the provider but the pass rate is shuffled compared to the metric

<img width="870" alt="image" src="https://github.com/user-attachments/assets/12f142a7-fb76-4d6e-a431-8aeb66cd979e">

I don't know which result actually matches up with which model

I've just checked and this doesn't happen in version 0.91.3

Maybe due to https://github.com/promptfoo/promptfoo/pull/1776 @sklein12?

I'll take a look

@brendanator I found the bug, thanks for reporting! It was just a display issue, so all your stats and results from previous runs will still be valid.

Fixed in release https://github.com/promptfoo/promptfoo/releases/tag/0.92.1

It works! Thanks @sklein12 ⭐
Repository: promptfoo/promptfoo
Issue: Provider for instructor?
Root cause: Documentation Deficiency

https://js.useinstructor.com/

You can use Instructor with promptfoo via a custom JavaScript provider. Here's a minimal example:

**1. Install dependencies:**
```bash
npm install @instructor-ai/instructor openai zod
```

**2. Create `instructor_provider.js`:**
```javascript
const Instructor = require("@instructor-ai/instructor").default;
const OpenAI = require("openai").default;
const { z } = require("zod");

const schemas = {
  UserInfo: z.object({
    name: z.string().describe("The user's full name"),
    age: z.number().describe("The user's age"),
    occupation: z.string().optional(),
  }),
};

class InstructorProvider {
  constructor(options) {
    this.providerId = options.id || "instructor-provider";
    this.config = options.config || {};
    this.schemaName = this.config.schema || "UserInfo";

    const oai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
    this.client = Instructor({ client: oai, mode: "TOOLS" });
  }

  id() { return this.providerId; }

  async callApi(prompt) {
    const schema = schemas[this.schemaName];
    const result = await this.client.chat.completions.create({
      model: this.config.model || "gpt-4o-mini",
      messages: [{ role: "user", content: prompt }],
      response_model: { schema, name: this.schemaName },
    });
    return { output: JSON.stringify(result, null, 2) };
  }
}

module.exports = InstructorProvider;
```

**3. Create `promptfooconfig.yaml`:**
```yaml
providers:
  - id: file://instructor_provider.js
    config:
      model: gpt-4o-mini
      schema: UserInfo

prompts:
  - "{{input}}"

tests:
  - vars:
      input: "Sarah Johnson is a 28-year-old data scientist."
    assert:
      - type: is-json
      - type: javascript
        value: |
          const data = JSON.parse(output);
          return data.name === "Sarah Johnson" && data.age === 28;
```

**4. Run:**
```bash
npx promptfoo eval
```

Since Instructor works well as a custom provider and doesn't require a dedicated built-in provider, I'm closing this issue. Please feel free to comment or re-open if you have any follow ups!
Repository: promptfoo/promptfoo
Issue: Question: how to use `vars` within `assert`'s `value`
Root cause: Documentation Deficiency

Here is a subset of my `promptfooconfig`:

```yaml
    - vars:
          value: foo
          result: bar
      assert:
          - type: similar
            value: The answer is {{ result }}.
```

Is it possible to use `vars` in an assertion's `value`?

The details pages suggests the substitution didn't take place:

![screenshot of details pane](https://github.com/promptfoo/promptfoo/assets/8990777/4f038c3b-cafe-40e7-ba90-b061da2028a7)

It looks like [Scenarios](https://promptfoo.dev/docs/configuration/scenarios) get closer, but the expectedXYZ fields in the example also aren't using templating from the `vars` like `language`.

Realizing this may be enabled by https://github.com/promptfoo/promptfoo/issues/150

Okay @typpo just to follow up on this, use of `vars` is now possible in assertions via code files, and thank you for that 👍.

Is it possible to use `vars` in an assertion in a vanilla `promptfooconfig.yaml` (no external `.py`/`.js` code file, and not using `scenarios`)?

```yaml
    - vars:
          value: foo
          result: bar
      assert:
          - type: similar
            value: The answer is {{ result }}.
```

I'm pretty sure this is already supported, as it is a test case.  It may be that the webview is not showing the substituted variable, even though the assert is run correctly.

Okay I was also wondering that.  Can you add something (if not already present) to the web view (`promptfoo view`) to show the finalized assertion string? That can give me confidence it works.

Another idea is to add `debug` logging that prints the actual finalized `value` used in the assertion

---

Testing with `--output file.yaml`, it doesn't contain the finalized assertion `value`, but it does contain the finalized prompt under `results.table.body.outputs[x].prompt`.

It would be cool if both of these had the finalized assertion string:
- `promptfoo view`
- `--output file.xyz`

@typpo I see in 0.25.0 there is a bunch of UI improvements 🥳 love it!

Any chance you could add support to the web view to also view finalized expected strings?

Haven't lost track of this.  Just requires a bit of extra plumbing with the way it's currently implemented :thinking:

@jamesbraza! This works now.  You can try it out in recent versions. Your feedback and ideas have been super helpful! Thank you!
Repository: promptfoo/promptfoo
Issue: Promptfoo doesn't start without installing @fal-ai/serverless-client
Root cause: Validation Gap

**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
1. `mkdir tmp && cd tmp`
2. `yarn add promptfoo`
3. `yarn promptfoo`
4. See error

```
$ ./tmp/node_modules/.bin/promptfoo
node:internal/modules/cjs/loader:1078
  throw err;
  ^

Error: Cannot find module '@fal-ai/serverless-client'
Require stack:
- ./tmp/node_modules/promptfoo/dist/src/providers/fal.js
- ./tmp/node_modules/promptfoo/dist/src/providers.js
- ./tmp/node_modules/promptfoo/dist/src/matchers.js
- ./tmp/node_modules/promptfoo/dist/src/assertions.js
- ./tmp/node_modules/promptfoo/dist/src/evaluator.js
- ./tmp/node_modules/promptfoo/dist/src/commands/eval.js
- ./tmp/node_modules/promptfoo/dist/src/main.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:1075:15)
    at Module._load (node:internal/modules/cjs/loader:920:27)
    at Module.require (node:internal/modules/cjs/loader:1141:19)
    at require (node:internal/modules/cjs/helpers:110:18)
    at Object.<anonymous> (./tmp/node_modules/promptfoo/dist/src/providers/fal.js:30:26)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Module._load (node:internal/modules/cjs/loader:958:12)
    at Module.require (node:internal/modules/cjs/loader:1141:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    './tmp/node_modules/promptfoo/dist/src/providers/fal.js',
    './tmp/node_modules/promptfoo/dist/src/providers.js',
    './tmp/node_modules/promptfoo/dist/src/matchers.js',
    './tmp/node_modules/promptfoo/dist/src/assertions.js',
    './tmp/node_modules/promptfoo/dist/src/evaluator.js',
    './tmp/node_modules/promptfoo/dist/src/commands/eval.js',
    './tmp/node_modules/promptfoo/dist/src/main.js'
  ]
}

Node.js v18.15.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
```

**Expected behavior**
Promptfoo shows CLI help info

- Promptfoo version: 0.92.0, 0.92.1

**Additional context**
A workaround is to explicitly install `@fal-ai/serverless-client`

Thank you for reporting this issue @tomquist! I’ve identified the problem and released a fix in version [0.92.2](https://github.com/promptfoo/promptfoo/releases/tag/0.92.2).

Please update to the latest version, and feel free to reach out if you encounter any further issues. Thanks again for bringing this to our attention!
Repository: promptfoo/promptfoo
Issue: Use cost as a metric
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I'd like to know the average cost per test case

**Describe the solution you'd like**
Use cost as a metric without a threshold

**Describe alternatives you've considered**
I've tried both of these and neither seem to work:
```
  # this expects a threshold - Error: Cost assertion must have a threshold
  - metric: cost
    weight: 0
    type: cost
  # this shows a cost of zero - probably the cost field is undefined
  - metric: cost_js
    weight: 0
    type: javascript
    value: cost
```

**Additional context**
Add any other context or screenshots about the feature request here.

It might also be good to allow token counts to be metrics too

Hi @brendanator, we actually show average cost per test case already!  It's tucked away in that gray text at the bottom of the header

<img width="483" alt="image" src="https://github.com/user-attachments/assets/6b522510-2d6c-4ce2-a400-c3f9b7d380d2">


I do get the cost reported but it doesn't seem to be the average. More like the total costs. In my example here, the cost per completion is ~0.00095. The number reported on top is roughly 53*0.00095, which is about $0.051. Personally, I would prefer to see the average cost but both could be useful.

<img width="424" alt="Screenshot 2024-12-01 at 8 31 10 PM" src="https://github.com/user-attachments/assets/2e708b9f-53c0-4b1a-8b51-8a28504cc622">
Repository: promptfoo/promptfoo
Issue: Bug: OpenAI No embedding returned
Root cause: Validation Gap

https://platform.openai.com/docs/guides/embeddings/what-are-embeddings has an example JSON blob structure from OpenAI `/embeddings`.

Continuing my LocalAI hacking from https://github.com/promptfoo/promptfoo/issues/176 and https://github.com/promptfoo/promptfoo/issues/177, here is an example cache entry made by `promptfoo` from LocalAI's `/v1/embeddings`:

```json
{
    "expireTime": 1696613448918,
    "key": "fetch:http://localhost:8080/v1/embeddings:{\"method\":\"POST\",\"body\":\"{\\\"input\\\":\\\"<redacted by James>.\\\\n\\\",\\\"model\\\":\\\"bert-embeddings\\\"}\"}",
    "val": "{\"object\":\"list\",\"model\":\"bert-embeddings\",\"data\":[{\"embedding\":[-0.058698192,0.019647375,...,-0.0041024373],\"index\":0,\"object\":\"embedding\"}],\"usage\":{\"prompt_tokens\":0,\"completion_tokens\":0,\"total_tokens\":0}}"
}
```

Currently, OpenAI provider has [`data?.data?.[0]?.embedding`](https://github.com/promptfoo/promptfoo/blob/0.24.1/src/providers/openai.ts#L123C25-L123C51).  Running this, I get:

```
[FAIL] API response error: Error: No embedding returned: {"error":{"code":500,"message":"rpc error: code = Unavailable desc = error reading from server: EOF","type":""}}
```

Changing the source to use `data.data[0].embedding`, it works fine, so I think it's a JSON traversal issue.  Why are the `?` there in the source now?

Strangely, I can't repro this any more.  I opened https://github.com/promptfoo/promptfoo/pull/190 to make it easier to read and report issues like this in the future.
Repository: promptfoo/promptfoo
Issue: Add support for Plug'n'Play
Root cause: External Dependency Breakage

**Is your feature request related to a problem? Please describe.**
After the release `0.72.0` promptfoo stopped supporting Plugnplay. It fails to start with the error message:
```
Error: promptfoo tried to access @inquirer/editor, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.
```

**Describe the solution you'd like**
Install promptfoo via yarn using pnp - which is the direction the ecosystem is going.

**Describe alternatives you've considered**
The problem can be circumvented by using the traditional node_modules approach, but that breaks other pieces of my project and could introduce potential errors in the future with ghost dependencies.



Thank you, @pedrooa, for your quick solution on Discord! For anyone else facing the issue, here’s the solution:

To resolve the problem with Plug'n'Play (PnP) and `promptfoo`:

1. Update your `yarn` configuration to use `nodeLinker`:

   ```bash
   yarn config set nodeLinker node-modules
   yarn
   ```

This will switch from using Plug'n'Play to the traditional `node_modules` approach, which resolves the issue with missing dependencies for `@inquirer/editor`.

I’m closing this issue since we have a path forward. Thanks again for the solution! Thanks again @pedrooa
Repository: promptfoo/promptfoo
Issue: transformVars for input vars in python won't work?
Root cause: Documentation Deficiency

**Describe the bug**
Unable to get my transform_vars python function to run proparly.

**To Reproduce**

**promptfooconfig.yaml:**
```
defaultTest:
  options:
    transformVars: file://transform_vars.py
```

**transform_vars.py:**
```
def get_transform_vars(vars, context):
    print(vars)
```

**Exception:**
```
Traceback (most recent call last):
  File "/.nvm/versions/node/v18.20.3/lib/node_modules/promptfoo/dist/src/python/wrapper.py", line 39, in <module>
    result = call_method(script_path, method_name, *data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/.nvm/versions/node/v18.20.3/lib/node_modules/promptfoo/dist/src/python/wrapper.py", line 24, in call_method
    method_to_call = getattr(script_module, method_name)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'transform_vars' has no attribute 'get_transform'. Did you mean: 'get_transform_vars'?
```

- Promptfoo version: 0.92



Hi @danpe 

By default it looks for `get_transform` method defined in your python file. If you want to use a different method name, then you can give the transform method name like this:
`transformVars: file://transform.py:transform`

@MrFlounder You are actually correct, just need to update the docs then :) 
This example is wrong: https://github.com/promptfoo/promptfoo/blob/207469a6a54b3a4123c2f8418d51affe9b04d40d/site/docs/configuration/guide.md?plain=1#L716 

Thanks @danpe for bringing this to our attention. We've updated the documentation to correct the function name in the `transformVars` example in https://github.com/promptfoo/promptfoo/commit/6a9ed383a7ddf801aa8dbbc36af0ee8b237faa4a. 

Please don't hesitate to report any other issues you come across or suggest improvements. Every interaction helps make promptfoo better!
Repository: promptfoo/promptfoo
Issue: Docs request: including the `prompts.txt` in Scenarios
Root cause: Documentation Deficiency

Can we include the `prompts.txt` used in the example [Scenario](https://promptfoo.dev/docs/configuration/scenarios)?  It seems to be relevant in understanding how Scenarios work.

I can open a PR for this, if you share what the `prompts.txt` looked like

Here's the full example: https://github.com/promptfoo/promptfoo/tree/main/examples/multiple-translations-scenarios

Thank you!  Opened https://github.com/promptfoo/promptfoo-docs/pull/15 for this, closing this issue out