zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Can't run JS function prompts when using node package
Root cause: Unimplemented Feature Gap

I'm following [Using the node package](https://www.promptfoo.dev/docs/usage/node-package/) and [Input and output files](https://www.promptfoo.dev/docs/configuration/parameters), but I can't figure out how to get promptfoo to run my JS function.

My `test1.js`:

```
module.exports = async function ({ topic }) {
  return [
    {
      role: "system",
      content: `You're an angry pirate. Be concise and stay in character.`,
    },
    {
      role: "user",
      content: `Tell me about ${topic}`,
    },
  ];
};
```

My `eval.js`:

```
import promptfoo from "promptfoo";

const results = await promptfoo.evaluate(
  {
    prompts: ["file://./test1.js"],
    providers: ["openai:gpt-3.5-turbo"],
    tests: [
      {
        vars: {
          topic: "Hello world",
        },
      },
      {
        vars: {
          topic: "I'm hungry",
        },
      },
    ],
  },
  {
    maxConcurrency: 2,
  }
);

console.log(JSON.stringify(results, null, 2));
```

The output:

```
{
  "version": 2,
  "results": [
    {
      "provider": {
        "id": "openai:gpt-3.5-turbo"
      },
      "prompt": {
        "raw": "file://./test1.js",
        "display": "file://./test1.js"
      },
      "vars": {
        "topic": "I'm hungry"
      },
      "response": {
        "output": "I'm sorry, but I am unable to access or open local files. Is there anything specific you would like to know or discuss about the file \"test1.js\"?",
...
```

Of course, what I actually want to do is put the prompt function directly in the call, like:

```
const results = await promptfoo.evaluate(
  {
    prompts: [
      async function ({ topic }) {
        return [
          {
            role: "system",
            content: `You're an angry pirate. Be concise and stay in character.`,
          },
          {
            role: "user",
            content: `Tell me about ${topic}`,
          },
        ];
      }
    ],
    providers: ["openai:gpt-3.5-turbo"],
```

But this gives me a type error, plus a runtime error:

```
TypeError [ERR_INVALID_ARG_TYPE]: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined
```

How can I use a JS function as the prompt when using the node package?

Hi @jameshfisher,

> Of course, what I actually want to do is put the prompt function directly in the call

Prompt functions are actually not supported via the [node package](https://promptfoo.dev/docs/usage/node-package) yet.  #315 adds this functionality!  It will be published in the next release.

Cool, thanks! ❤️ 

This fix is included in 0.31.0
Repository: promptfoo/promptfoo
Issue: Failed to create shareable URL: Internal Server Error
Root cause: External Dependency Breakage

**Describe the bug**
A clear and concise description of what the bug is.
When running `npx promptfoo@latest share -y`, it throws the error: 
```
Failed to create shareable URL: Internal Server Error
View results: null
```

**To Reproduce**
Steps to reproduce the behavior, including example Promptfoo configurations if applicable:

1. Run `npx promptfoo@latest share -y`
2. See error

**Expected behavior**
A clear and concise description of what you expected to happen.
It's expected to generate a shareable URL.

**Screenshots**
If applicable, add screenshots to help explain your problem.

<img width="578" alt="Image" src="https://github.com/user-attachments/assets/64bd383e-65fa-473d-8fd6-c72f3f98ff73" />

**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.
```
{
  "version": "0.104.4",
  "platform": {
    "os": "darwin",
    "release": "24.3.0",
    "arch": "arm64",
    "nodeVersion": "v20.18.2"
  },
  "env": {},
  "configInfo": {
    "configExists": false,
    "configContent": null
  }
}
```


- Promptfoo version: 0.104.4

**Additional context**
Add any other context about the problem here.


Hey @feiran-ws I am able replicate this issue. We're working on getting a fix out (tomorrow). If you need sharing in the mean time, send me an email at michael@promptfoo.dev 

@mldangelo Thanks for the quick response. I don’t need sharing at the moment, but I really appreciate the offer.

Thanks for the report!  Sharing to `app.promptfoo.dev` should be fixed now
Repository: promptfoo/promptfoo
Issue: How are derived metrics defined
Root cause: Documentation Deficiency

Hi I want to create a derived metric using a mathematical expression but its not clear to me how to average the scores between tests. What language is used for the mathematical expressions and can things like the number of tests be used e.g. sum(tests) / len(tests) ?

Promptfoo uses mathjs to evaluate expressions in derived metrics. You can use standard math operators for operations like averaging. For details, please see the [mathjs syntax documentation](https://mathjs.org/docs/expressions/syntax.html).

For example, to calculate an average score you might write:

```yaml
derivedMetrics:
  - name: 'AverageScore'
    value: '(metric1 + metric2 + metric3) / 3'
```

I just opened https://github.com/promptfoo/promptfoo/pull/3157 to try to explain them better! Would love your thoughts on it. Let us know if you have any other questions.

Thanks, yes that explains it much better! Two related questions:

- It doesn't seem possible to add an assertion to derived metrics so that they can pass/fail, is that correct?
- I would like to create a derived metric that calculates the average latency across multiple input vars. However it seems like the derived metric can only access the pass/fail value rather than the latency milliseconds

Also the javascript example was great and I think it might be able to be used for the average latency use case, but it would help if we could say how all the metrics such as accuracy, speed and difficulty  were defined in the YAML

@mldangelo tagging in case you didn't get a notification for this. Apologies if you did

Hi @nfrancis-esure, thanks for the question!

Derived metrics use [mathjs](https://mathjs.org/) syntax for mathematical expressions. The documentation has been expanded since this issue was opened and now covers this in detail:

**[Derived Metrics Documentation](https://www.promptfoo.dev/docs/configuration/expected-outputs/#creating-derived-metrics)**

### Quick Answer

For your specific question about averaging - derived metrics operate **per prompt**, not across all tests. Each named metric accumulates pass rates automatically. For example:

```yaml
defaultTest:
  assert:
    - type: contains
      value: "expected"
      metric: accuracy  # This creates a named metric

derivedMetrics:
  - name: weighted_score
    value: 'accuracy * 0.6 + relevance * 0.4'  # mathjs syntax
```

The `accuracy` metric will automatically show as a percentage (e.g., "85.00% (17/20)") in the results.

For more complex logic, you can use JavaScript functions:

```yaml
derivedMetrics:
  - name: custom_avg
    value: |
      function(namedScores, evalStep) {
        const { metric1 = 0, metric2 = 0 } = namedScores;
        return (metric1 + metric2) / 2;
      }
```

See also:
- [Named metrics example](https://github.com/promptfoo/promptfoo/tree/main/examples/named-metrics)
- [F-score example](https://github.com/promptfoo/promptfoo/tree/main/examples/f-score)

Closing as this is now documented. Feel free to reopen if you have follow-up questions!
Repository: promptfoo/promptfoo
Issue: Can't use 'rouge_score' in custom metrics due to circular import
Root cause: Documentation Deficiency

**Describe the bug**
I'm creating a bunch of custom metrics and they all work very well except for one based one the "rouge_score" library (https://pypi.org/project/rouge-score/).

When I try to import this specific library to a custom metric script (even a working one), I get the following error : 
`ImportError: cannot import name 'rouge_scorer' from partially initialized module 'rouge_score' (most likely due to a circular import)`

**To Reproduce**

1. Install the rouge_score library
2. Select an existing and working custom metric script
3. Add `from rouge_score import rouge_scorer` with the other imports 
4. Run the evaluation and see the error

If I run the very same custom script directly (without promptfoo, but using a main entrypoint) it works perfectly.

- Promptfoo version: 0.104.4

**Additional infos**
I know you added the `rouge-n` score to promptfoo, so it might come from here. However, I have also implemented a custom BLUE score metric (using the sacrebleu library) and I don't have similar problem (And I get different scores with your `blue` metric and my custom blue metric, but I will open a different issue to discuss that).


Hey @Valkea, I am sorry you're running into this issue. Can you help with a minimal example so I can try to reproduce and resolve your issue? A short promptfoo config such as:

```yaml
# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json
provider:
  - echo
prompts: '{{prompt}}'
tests:
  - vars:
      prompt: ....
    assert:
      - type: ...
```

along with your python script would be really helpful 

So, I lost 4 hours debugging this problem, and that was... obvious and totally my fault!
(the name of my custom metric script was 'rouge_score.py')

Sorry for the false alert, and thank you for the fast response.
Repository: promptfoo/promptfoo
Issue: PIP Python package for promptfoo
Root cause: Unimplemented Feature Gap

Hi! I've been an avid user of promptfoo, and have been building apps around it, really like the interface and extremely intuitive support.
The only issue i keep running into is setting up npm and using something like mise to actually install and manage the packages when using it in python.

Could there be a way to provide a pip package that is a wrapper around the current promptfoo package? it would be EXTREMELY helpful to use promptfoo locally and in production with python. 

something we could `pip install promptfoo` and just keep working with!
Thanks!

@sert121 this is a great idea! We already own [promptfoo on PyPI](https://pypi.org/project/promptfoo/) and have thought about generating mypy or even pydantic types from our schema using tools like the [datamodel code generator](https://docs.pydantic.dev/latest/integrations/). Publishing a python package has been on our backlog for a while, but we will likely not prioritize it in the near term. We need to carefully consider how pip install promptfoo will work. It might involve either downloading a large compiled binary to support multiple OS and architectures or installing Node and all our subdependencies. Optional and peer dependencies add additional complexity.

I would love to see improved Python support in promptfoo and am actively working towards this ([here](https://github.com/promptfoo/promptfoo/pull/2993) is a recent PR that allows you to define test cases in python, but without any types). If you have ideas on areas that need improvement, please share. We are also happy to accept PRs if you have the bandwidth to drive features that benefit your workflow.

Greetings!
Is this still happening? :)
Sunny regards
Repository: promptfoo/promptfoo
Issue: Feat: import custom providers as a list/file
Root cause: Unimplemented Feature Gap

Currently promptfoo allows us to read providers from a file, but this only limits us to one-provider per-file.
I was wondering if we could possibly have multiple providers from the same file, just like we have it in tests. 
That'd be super helpful in just keeping one providers.yaml file.



Great idea @sert121 

This will be in the next release!
Repository: promptfoo/promptfoo
Issue: add vertex:gemini-2.0-flash-001 to docs

**Is your feature request related to a problem? Please describe.**
Docs are missing gemini-2.0-flash-001 version

**Describe the solution you'd like**
Add vertex:gemini-2.0-flash-001 to the docs

**Describe alternatives you've considered**
N/A

**Additional context**
N/A
Repository: promptfoo/promptfoo
Issue: Web viewer cant load eval data
Root cause: Environment Incompatibility

Hi promptfoo team, thank you guys for great framework, really love that. Recently, I have ran an evaluation and try to view that using promptfoo view but the ui just show loading eval data for all the time (image below). Can you guys help me with this?
![image](https://github.com/promptfoo/promptfoo/assets/33374938/620de504-f5f8-4e74-bab0-38e668684b3f)


Double checking, are you running `promptfoo view` in the background?

> Double checking, are you running `promptfoo view` in the background?

No, im running that command directly on my azure VM

Remote view server is currently not supported, but #321 adds this functionality.  It'll be included in the next release :)

Hi @LeDuySon, `promptfoo view` should support hosting on hostnames besides localhost now.  Please give it a try with the latest version

hi @typpo , everything work fine now. Thank you so much!
Repository: promptfoo/promptfoo
Issue: Using prompt.py broke since v0.30.0
Root cause: External Dependency Breakage

Hello 👋,

I've encountered an issue starting from version v0.30.0 where the functionality to execute prompts from Python seems to be malfunctioning. Instead of processing the intended output, the model is responding with the raw contents of the prompt's codebase.


My configuration:
```yaml
description: description
providers: openai:gpt-4
prompts: [prompt.py]
tests: tests_prompt.csv
```

My prompt.py file:
```python
def generate_prompt(context) -> str:
    return get_prompt_from(
        location=os.path.dirname(os.path.dirname(os.path.realpath(__file__))),
        file_name="prompt.yaml",
    ) # return a str


if __name__ == "__main__":
    print(generate_prompt(json.loads(sys.argv[1])))
```

However, when I execute the evaluation, I encounter an error. The response from the language model pertains to the prompt.py code itself rather than the expected output from the script.

Thank you!


Hi @fvaleye,

Can you try with 0.31.0 and see if that solves the issue?  There were some fixes between then and now.

Here's my standalone test case, which seems to be working fine:

promptfooconfig.yaml:
```yaml
providers: openai:gpt-4
prompts: [prompt.py]
tests: tests.csv
```
prompt.py:
```py
import sys
import json

def generate_prompt(context) -> str:
    return 'write a sentence about ' + context['vars']['topic']

if __name__ == "__main__":
    print(generate_prompt(json.loads(sys.argv[1])))
```
tests.csv:
```
topic
bananas
avocado toast
new york city
```

Hey @typpo 👋,

This doesn't work either!

my test.csv file:
```csv
question,context,chat_history,__expected
Who are you?,,,llm-rubric:a chatbot
```



Ah, are you by any chance using multiple configs in a single `eval` command?  I'm able to repro that way.  Will look into a fix.

Should be fixed with 0.31.1 - please give it a try

It works now, thanks @typpo 🙏
Repository: promptfoo/promptfoo
Issue: Empty output when tool is being called by OpenRouter provider
Root cause: Interface Contract Mismatch

**Describe the bug**
We're unable to make assertions on tool calls using providers other than OpenAI, eg OpenRouter or Google. Somehow output seems to be empty string when tool is being called by one of these providers. 

We want to switch from gpt-4o-mini to gemini 2.0, but sadly we're not able to run out test suite :( 

**To Reproduce**
Running this config causes error, while switching to openai:gpt-4o-mini works just fine

```yaml
prompts:
  - 'What is the weather like in {{city}}?'

providers:
  - id: openrouter:openai/gpt-4o-mini
    config:
      tools: [
        {
          "type": "function",
          "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
              "type": "object",
              "properties": {
                "location": {
                  "type": "string",
                  "description": "The city and state, e.g. San Francisco, CA"
                },
                "unit": {
                  "type": "string",
                  "enum": ["celsius", "fahrenheit"]
                }
              },
              "required": ["location"]
            }
          }
        }
      ]

tests:
  - vars:
      city: Boston
    assert:
      # just to debug output value
      - type: javascript
        value: "output"
      - type: is-valid-openai-tools-call
      - type: javascript
        value: output[0].function.name === 'get_current_weather'
      - type: javascript
        value: JSON.parse(output[0].function.arguments).location === 'Boston, MA'
```

**Expected behavior**
Test should pass

**Screenshots**

<img width="1203" alt="Image" src="https://github.com/user-attachments/assets/23714fe6-f95f-4114-8750-c6dfd8e0ba2e" />

**System information:**

```
  "version": "0.104.4",
  "platform": {
    "os": "darwin",
    "release": "23.5.0",
    "arch": "arm64",
    "nodeVersion": "v22.8.0"
  },
```

Thanks for flagging, looks like OpenRouter differs from the OpenAI API by returning an empty string in the `content` field, rather than null or undefined. #3205 will fix in the next release.
Repository: promptfoo/promptfoo
Issue: Chore: Move renderVarsInObject into Tools specific function
GitHub labels: Open Source

Half the providers render vars and env variable sin path names, half do not.

https://github.com/promptfoo/promptfoo/blob/fb8d42c1c2122041f3af0b1a9adac1b0632cc98c/src/providers/azure.ts#L605

Make this consistent and hide this function in something like `maybeLoadToolsFromExternalFile`
Repository: promptfoo/promptfoo
Issue: "Error: OpenAI API key must be set for similarity comparison" comes up after using AZURE_OPENAI_API_KEY
Root cause: Documentation Deficiency

I'm using the Azure integration to run a similarity check for my prompts but this error comes up which asks me for my OPENAI_API_KEY instead. I've tried the overrides as well but that doesn't seem to work either.

I exported the AZURE_OPENAI_API_KEY separately in the command line.

`promptfooconfig.yaml`
```
prompts:
  - "Generate one very interesting fact about {{topic}}"

providers:
    - id: azureopenai:chat:gpt-35-turbo-16k
      config:
        apiHost: 'llm.openai.azure.com'

tests:
  - vars:
      topic: bananas
    assert:
    - type: similar
      value: Bananas are naturally radioactive.
      provider: azureopenai:chat:gpt-35-turbo-16k
```

Hi @anoura12,

There might be something wrong with your setup as I am able to override the assertion provider with this config.  Maybe you need to set the api host for the provider:
```yaml
prompts:
  - "Generate one very interesting fact about {{topic}}"

providers:
    - id: azureopenai:chat:gpt-35-turbo-16k
      config:
        apiHost: 'xxxx.openai.azure.com'

tests:
  - vars:
      topic: bananas
    assert:
      - type: similar
        value: Bananas are naturally radioactive.
        provider:
          id: azureopenai:chat:gpt-35-turbo-16k
          config:
            apiHost: 'xxxx.openai.azure.com'
```

Let me know if you're able to reproduce the issue with a minimal example.

Hi @typpo 

I tried adding the apiHost and the apiKey in the provider this time and it seems that I'm getting the same error again.

```
prompts:
  - "Generate one very interesting fact about {{topic}}"

providers:
    - id: azureopenai:chat:gpt-35-turbo-16k
      config:
        apiHost: 'xxxx.openai.azure.com'
        apiKey: abc123

tests:
  - vars:
      topic: bananas
    assert:
      - type: similar
        value: Bananas are naturally radioactive.
        provider:
          id: azureopenai:chat:gpt-35-turbo-16k
          config:
            apiHost: 'xxxx.openai.azure.com'
            apiKey: abc123
    
```

Here is the error
![Screenshot from 2023-12-06 11-30-47](https://github.com/promptfoo/promptfoo/assets/73518403/29370e20-0809-42f6-8fea-909723579cbe)

Is there something I'm missing? Any idea what I should look into the Azure setup?

Ah I see, the `similar` assertion requires an embedding API and you are passing a chat API.  

You'll have to create a deployment that uses the `text-embedding-ada-002` model and give it `azureopenai:embeddings:text-embedding-ada-002-deploymentname` as the assertion provider.

Please give that a try and let me know how it works!

Oh wow, this worked like a charm

Thanks so much!

No problem.  I've updated the documentation and improved the error message, and will think about other ways to make Azure setup smoother.
Repository: promptfoo/promptfoo
Issue: Support returning configuration from prompt functions
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Promptfoo supports configuring provider parameters for prompts in the YAML configuration:

```yaml
prompts:
  - text: "What is 2+2?"
    config:
      temperature: 0.7
```

However, prompt functions can only ["return a string or an object that represents the prompt"](https://www.promptfoo.dev/docs/configuration/parameters/#prompt-functions).

We use prompts and structured outputs that depend on dynamic configurations so `response_format` cannot be defined statically. Since the prompt and provider objects are merged, I think it would make sense that the prompt function can also return a config?

**Describe the solution you'd like**
Allow prompt functions to return both prompt content and provider configuration:
```python
def my_prompt_function(context: dict) -> dict:
    messages = [
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": f"Tell me about {context['vars']['topic']}"}
    ]
    
    # Dynamic configuration based on context
    config = {
        "response_format": {
            "type": "json_schema",
            "schema": generate_schema_for_topic(context['vars']['topic'])
        }
    }
    
    return {
        "prompt": messages,
        "config": config
    }
```

**Describe alternatives you've considered**
We work around this by using different promptfoo scenarios and use prompt functions that only return the dynamic prompts, and writes the json schema to files as a side-effect that we then reference in the YAML.