zhiminy/EvalEng huggingface dataset:
Repository: promptfoo/promptfoo
Issue: Python error at promptfoo eval
Root cause: Environment Incompatibility

**Describe the bug**
I successfully runned `promptfoo init` , however,  `promptfoo eval` gives the error

```
node:events:497
      throw er; // Unhandled 'error' event
      ^
      
Error: spawn python EACCES
    at ChildProcess._handle.onexit (node:internal/child_process:286:19)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21)
Emitted 'error' event on PythonShell instance at:
    at ChildProcess.<anonymous> (/home/user/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/node_modules/python-shell/index.js:180:18)
    at ChildProcess.emit (node:events:519:28)
    at ChildProcess._handle.onexit (node:internal/child_process:292:12)
    at onErrorNT (node:internal/child_process:484:16)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -13,
  code: 'EACCES',
  syscall: 'spawn python',
  path: 'python',
  spawnargs: [
    '/home/user/.nvm/versions/node/v20.17.0/lib/node_modules/promptfoo/dist/src/python/wrapper.py',
    '/my_code_path/evaluation/prompts.py',
    'prd_prompt',
    '/tmp/promptfoo-python-input-json-1726615914161-40caad32f06a3.json',
    '/tmp/promptfoo-python-output-json-1726615914161-a9d46bdb5ba16.json'
  ]
}
Node.js v20.17.0
```

**System information (please complete the following information):**

- node v20.17.0
- npm 10.8.2
- Ubuntu 22.04

- Promptfoo version: 0.89.0




Hi @BrenosmMaia, thanks for reporting this issue! We shipped updates to python executable discovery today so this is probably my fault. It looks like the error you're encountering (`spawn python EACCES`) is related to permissions or Python not being available in your system's `PATH`.

Could you please try the following:

1. **Check if Python is in your PATH**: Run `python --version` or `python3 --version` in your terminal to confirm that Python is installed and accessible.
   
2. **Set the Python path manually**: If Python is installed but not accessible, try setting the `PROMPTFOO_PYTHON` environment variable to the path of your Python installation. You can do this by running:

   ```bash
   export PROMPTFOO_PYTHON=/path/to/your/python
   ```

   Replace `/path/to/your/python` with the actual path to your Python executable. You can find this path by running `which python` or `which python3`.

Let me know if this resolves the issue, and feel free to share any additional details if the error persists. 

Hello @mldangelo, thanks for the answer.

I did the following, and it worked!

```
$ python3 --version
Python 3.10.12

$ which python3
/usr/bin/python3

$ export PROMPTFOO_PYTHON=/usr/bin/python3
```




Hi @BrenosmMaia, I'm glad to hear that setting the `PROMPTFOO_PYTHON` environment variable resolved the issue!  Since it looks like the problem has been addressed, I'll go ahead and close this issue. If you run into any more issues or have further questions, feel free to open a new one. Thanks again for your report and for helping improve promptfoo!
Repository: promptfoo/promptfoo
Issue: Using CSV tests with multiple columns
Root cause: Documentation Deficiency

I was checking the [google sheets example](https://github.com/promptfoo/promptfoo/tree/main/examples/google-sheets) and saw that the CSV uses two input columns to the prompt file.

Can this be done with prompts from a python file?

I was only able to successfully run `promptfoo eval` with a single argument function, like
```
def challenging_prompt(deciding_rule) -> str:
    
    return f"Describe the rule {deciding_rule}" 
```

When I added a new argument, like

```
def challenging_prompt(deciding_rule, extra) -> str:
    
    return f"Describe the rule {deciding_rule}, considering {extra}" 
```

Got error
```
 Error running Python script: TypeError: challenging_prompt() missing 1 required positional argument: 'ex'
Stack Trace: Error: TypeError: challenging_prompt() missing 1 required positional argument: 'extra'
```


Using the CSV
| deciding_rule | extra | __expected |
|---------------|-------|------------|
| too_many_tries | new_user | dummy |



I'm not sure whether this is currently a limitation or if I'm not using it correctly.



Hi @BrenosmMaia,

Good question! The issue you're encountering is due to how promptfoo expects Python prompt functions to be structured. When using a Python prompt function with promptfoo, the function should accept a single `context` dictionary as its parameter. This `context` contains your test case variables and other useful information.

Here's how you can modify your function to work with multiple input variables:

```python
def challenging_prompt(context: dict) -> str:
    deciding_rule = context['vars']['deciding_rule']
    extra = context['vars']['extra']
    return f"Describe the rule {deciding_rule}, considering {extra}"
```

Your CSV file should have columns matching the variable names:

```
deciding_rule,extra,__expected
too_many_tries,new_user,dummy
```

For more details, please refer to the [promptfoo documentation on prompt functions](https://www.promptfoo.dev/docs/configuration/parameters/#prompt-functions). Let me know if this resolves your issue or if you have any more questions!

@mldangelo 

Thanks! Issue solved.
Repository: promptfoo/promptfoo
Issue: OpenAI provider error when referencing external file for `response_format`
Root cause: Configuration Error

I got an error when referencing an external file following what is mentioned in the docs https://www.promptfoo.dev/docs/providers/openai/#external-file-references

```shell
API error: Invalid type for 'response_format': expected an object, but got a string instead., Type: invalid_request_error, Code: invalid_type

{
  "error": {
    "message": "Invalid type for 'response_format': expected an object, but got a string instead.",
    "type": "invalid_request_error",
    "param": "response_format",
    "code": "invalid_type"
  }
}
```

Files:
- sample.yaml
```yaml
prompts:
  - 'You are a helpful math tutor. Solve {{ problem }}'

providers:
  - id: openai:chat:gpt-4o-mini
    config:
      response_format: file://sample_schema.yaml
      # response_format: 
      #   type: json_schema
      #   json_schema:
      #     name: math_response
      #     schema:
      #       type: object
      #       properties:
      #         answer:
      #           type: string

tests:
  - vars:
      problem: 1+2
  - vars:
      problem: 4+8
```

- sample_schema.yaml
```yaml
type: json_schema
json_schema:
  name: math_response
  schema:
    type: object
    properties:
      answer:
        type: string
```

Do we need to add load from external file as well here? https://github.com/promptfoo/promptfoo/blob/0dfb305706f52efc5853cabdf030b27abb391c5f/src/providers/openai.ts#L525

Like https://github.com/promptfoo/promptfoo/blob/0dfb305706f52efc5853cabdf030b27abb391c5f/src/providers/openai.ts#L521-L523

Created a PR https://github.com/promptfoo/promptfoo/pull/1711 @mldangelo 

@albertlieyingadrian We just issued a release that includes your change. You can find the details in the latest version [0.89.1](https://github.com/promptfoo/promptfoo/releases/tag/0.89.1).

Thanks so much for helping get this in! Let us know if you run into any further issues, and feel free to reopen this issue if needed!
Repository: promptfoo/promptfoo
Issue: Python prompts not working for newer versions
Root cause: Configuration Error

**Describe the bug**
When defining prompts as 

```
prompts:
  - src/prompts.py:regular_prompt
  - src/prompts.py:augmented_prompt
```

The input prompt is the actual code as opposed to the code being run. This issue was not present in version `0.85.2` and is now present.

**To Reproduce**
Install latest version of promptfoo, define a prompt as a python function, and try running.

**Expected behavior**
Expected model to receive user prompt as returned by function instead of the actual python code.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information (please complete the following information):**

- Promptfoo version:
latest version does not work.
Used to work fine for `0.85.2`.


Hi @diego-delarue, thank you for bringing this to our attention. Could you please try adding the `file://` prefix to the file paths in your prompts definition, like this, and see if it resolves your issue?

```yaml
prompts:
  - file://src/prompts.py:regular_prompt
  - file://src/prompts.py:augmented_prompt
```

We prefer this syntax and have updated our documentation accordingly to exclude file paths without the `file://` prefix. Please let me know if this resolves the issue, and if there's anything else we can do to assist. I’ll also review our parsing logic and work on addressing the change that caused this error. 

Edit: this commit is the culprit: https://github.com/promptfoo/promptfoo/commit/92ee2bbb04115586c5f1c98161ee4e4ea6a48987

Using version `0.89.4` and adding the `file://` prefix to my python prompts in my `config.yaml` file, the issue is is resolved. Thanks!
Repository: promptfoo/promptfoo
Issue: prompt from external JS file not working
Root cause: Algorithmic Error

**Describe the bug**
When using prompt from external file
```
prompts:
  - file://prompt.js

providers:
  - label: "fun"
    id: "openai:gpt-4o-mini"

tests:
  - vars:
      topic: bananas
```

with `prompt.js` written like
```
module.exports = async function ({ vars }) {
    return `Write a tweet about ${vars.topic}`;
};
```

The input prompt into openAI provider seems to be an actual code
```
[
  {
    role: 'user',
    content: 'async function ({ vars }) {\n' +
      '    return `Write a tweet about ${vars.prompt}`;\n' +
      '  }'
  }
]
```

**To Reproduce**
Install latest version of promptfoo, define a prompt as a JS file, and try running.

**Expected behavior**
JS file to run and resolved to be a string as returned

**Screenshots**
If applicable, add screenshots to help explain your problem.

**System information (please complete the following information):**

- Promptfoo version:
0.89.0

**Additional context**
Previously working fine on version 0.87.1
might be similar issue as #1717 and #1716 as well

Seems like it started breaking after this release https://github.com/promptfoo/promptfoo/compare/0.87.1...0.88.0

The exact line that is problematic https://github.com/promptfoo/promptfoo/issues/1716#issuecomment-2362639691

fixed with #1724 , thanks @albertlieyingadrian
Repository: promptfoo/promptfoo
Issue: Incorrect JSON data format passed to prompts.py file
Root cause: Environment Incompatibility

 I encountered an issue with the data being passed to the `prompts.py` script. The expectation was that the data would be in a proper JSON format, but the received data showed formatting errors.

**Steps to Reproduce:**

**1. Creating prompt_generator file:**

```
%%writefile prompt_generator.py

import sys
import json

def generate_prompt(context):
    document = context["vars"]["document"]
    question = context["vars"]["question"]
    
    return f"""Document.
{document}
Question: {question}
"RELEVANT" or "NOT_RELEVANT":"""

if __name__ == '__main__':
    data = sys.argv[1] if len(sys.argv) > 1 else None
    with open("log_data.txt", "a") as f:
        f.write(f"Log data: {data}\n")
    print(generate_prompt(json.loads(data)))
```

**2. Creating prompfooconfig.yaml file:**

```
%%writefile promptfooconfig.yaml
prompts:
  - 'prompt_generator.py:generate_prompt'
providers: 
  - openai:gpt-3.5-turbo
tests:
  - vars:
      question: "question1"
      document: "document1"
    assert:
      - type: equals
        value: 'NOT_RELEVANT'
  - vars:
      question: "question2"
      document: "document2"
    assert:
      - type: equals
        value: 'RELEVANT'
```
        
**Log data file:**       

```
Log data: '{vars:{question:question1,document:document1}}'
Log data: '{vars:{question:question2,document:document2}}'
```

It should be something like:

```
{
  "vars": {
    "question": "question1",
    "document": "document1"
  }
}
```

Hi @maciejbrasewicz,

Interestingly, I'm unable to repro with the example you provided:

```
$ cat log_data.txt
Log data: {"vars":{"question":"question1","document":"document1"}}
Log data: {"vars":{"question":"question2","document":"document2"}}
```

Are you able to share more information about your environment? (promptfoo version, python version, OS, ...)

Promptfoo `0.24.1`
Python: `3.11.5`
Node: `v18.18.0`
OS: Windows 11

Thanks for the info, I think this is a Windows issue.  Potential fix in #179, though I do not have a Windows machine to confirm.

@maciejbrasewicz I've published a possible fix in version 0.24.2 - please give it a try and let me know if you're still not seeing the right args passed to the script.
Repository: promptfoo/promptfoo
Issue: Is there a way to see the time taken to run the entire test cases of a test suite from the UI?
Root cause: Unimplemented Feature Gap

Or could it be added via `hooks`? e.g.

```javascript
const { logger } = require('../helper/logger');

async function extensionHook(hookName, context) {
  if (hookName === 'beforeAll') {
    context.suite.startTime = new Date();
  } else if (hookName === 'afterAll') {
    const endTime = new Date();
    context.suite.totalTestTime = endTime - context.suite.startTime;

    logger.info(`Total tests: ${context.results.length}`);
    logger.info(`Total test time: ${context.suite.totalTestTime} ms`);
    // Perform any necessary teardown or reporting
  }
}

module.exports = extensionHook;
```

I don't think we display this anywhere, but in theory we have the timing of every test case so could add it up.  If you want an informal timer I usually just run with `time promptfoo eval` on the command line.  Doing it with hooks is a clever idea
Repository: promptfoo/promptfoo
Issue: Infinite loading when sharing promptfoo's url and wasn't able to see the eval result
Root cause: Environment Incompatibility

<img width="666" alt="Screenshot 2024-09-21 at 1 02 50 PM" src="https://github.com/user-attachments/assets/9ee0e703-faaa-4de7-b9dc-fbbc36c6e075">
<img width="606" alt="Screenshot 2024-09-21 at 1 02 52 PM" src="https://github.com/user-attachments/assets/6a6863b0-3d9e-4518-8ff8-cc573bb98eb6">


I _think_ this is caused by the app.promptfoo.dev instance running an older version that uses localStorage instead of indexDb.  We're in the process of improving the cloud deployment so hopefully this will be resolved once that is done.

Is there an ETA for fixing this issue? @typpo 

Got a similar error again just now
<img width="687" alt="Screenshot 2024-09-23 at 10 55 03 PM" src="https://github.com/user-attachments/assets/9ef4823a-1a3a-4f80-a68e-f48da2d057b6">


Any updates on this issue? :)

@sklein12 @mldangelo 

@albertlieyingadrian how large is your eval? If you put some time on my calendar https://cal.com/michael-dangelo/30min - we can move you over to our enterprise version with private sharing 

@mldangelo Not huge actually. Only ~100 test cases per eval

Hey @albertlieyingadrian and everyone else following along - thanks for your patience on this one. We've completely rebuilt our sharing infrastructure with the new promptfoo.app backend. The old app.promptfoo.dev instance that was causing these issues has been retired.

Sharing should work much more reliably now. If you run into any issues with the new system, please open a fresh issue and we'll take a look.

Closing this one out - thanks again for the report!
Repository: promptfoo/promptfoo
Issue: `outputPath` is not working
Root cause: Configuration Error

Tried setting up outputPath in the config as both string and array of string following this guide https://www.promptfoo.dev/docs/configuration/parameters/#output-file. However, both didn't work 

```yaml
tests:
  - file://datasets/customer/mini_default_test.yaml

outputPath: 
  - reports/customer_output.html
# outputPath: 'reports/customer.html'
```

Suspect that it is caused by `outputPath` is not read in `readConfigs` function

https://github.com/promptfoo/promptfoo/blob/e6089ec7594282f9be6094a52e8483e044cd7e78/src/config.ts#L357-L390

@mldangelo I went ahead with a PR https://github.com/promptfoo/promptfoo/pull/1734

Shipped in 0.89.4!
Repository: promptfoo/promptfoo
Issue: Different prompts per provider doesn't work
Root cause: Validation Gap

I tried to load different prompts for different providers following this instruction https://www.promptfoo.dev/docs/configuration/parameters/#different-prompts-per-model

```yaml
prompts:
  - id: file://prompts/gpt_chat_prompt.json
    label: gpt_chat_prompt
  - id: file://prompts/llama_completion_prompt.txt
    label: llama_completion_prompt

providers:
  - id: openai:gpt-4o-mini
    prompts:
      - gpt_chat_prompt
  - id: openai:gpt-4o
    prompts:
      - gpt_chat_prompt
  - id: replicate:meta/meta-llama-3.1-405b-instruct
    label: llama-3.1-405b-instruct
    prompts:
      - llama_completion_prompt
```

I got an error by `invariant`
```shell
/Users/albertlieadrian/Documents/Random/ai-llm-playground/evals/node_modules/tiny-invariant/dist/tiny-invariant.cjs.js:14
    throw new Error(value);
          ^

Error: Invariant failed: Invalid prompt
```

I traced it back and I figured it was caused by this checking https://github.com/promptfoo/promptfoo/blob/main/src/config.ts#L339-L342

The prompt's data structure, in this case, would have no `raw`, only `id` and `label`. Hence this configuration will never be able to be executed by promptfoo. 

This is a super hacky way to make it work -- by adding `raw` field in the config
```yaml
prompts:
  - id: file://prompts/gpt_chat_prompt.json
    label: gpt_chat_prompt
     raw: 'propmt gpt'
  - id: file://prompts/llama_completion_prompt.txt
    label: llama_completion_prompt
    raw: 'prompt llama'

providers:
  - id: openai:gpt-4o-mini
    prompts:
      - gpt_chat_prompt
  - id: openai:gpt-4o
    prompts:
      - gpt_chat_prompt
  - id: replicate:meta/meta-llama-3.1-405b-instruct
    label: llama-3.1-405b-instruct
    prompts:
      - llama_completion_prompt
```

But seems like it's not intuitive for users and this might be a blocker for them to run even basic evals if they are going to involve different prompts per provider @mldangelo @typpo 

I also ran in to this, but only when attempting to pass a different configuration file from command line like: `npx promptfoo@latest eval --config eval_all_config.yaml`. If you include the `raw` field in the prompts, the eval runs, but with the content of the `raw` field as the prompt, not what is specified in the file

> I also ran in to this, but only when attempting to pass a different configuration file from command line like: `npx promptfoo@latest eval --config eval_all_config.yaml`. If you include the `raw` field in the prompts, the eval runs, but with the content of the `raw` field as the prompt, not what is specified in the file

@bfdykstra exact same issue for me

Fix in #1757.  A temp workaround is to run `promptfoo eval` in the same directory as the config file, without the `-c` arg

 Hi @albertlieyingadrian, thank you for reporting this issue. This should be fixed in the latest release: [0.90.3](https://github.com/promptfoo/promptfoo/releases/tag/0.90.3) Please let us know if you encounter any further issues.
Repository: promptfoo/promptfoo
Issue: Add more deterministic/math-based assertions
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Some of the common math-based evaluation metrics for NLP/LLM includes ROUGE (already supported), BLEU, METEOR, GLEU and some others.

See https://github.com/Aldenhovel/bleu-rouge-meteor-cider-spice-eval4imagecaption and https://huggingface.co/spaces/evaluate-metric/google_bleu for details and examples. 

**Describe the solution you'd like**
I'd like these common evaluation metrics to be available as assertions in promptfoo.

**Describe alternatives you've considered**
Use a custom assertion to implement them. I believe it would be beneficial to all promptfoo users to have such assertions built-in.



@mldangelo Hi, I was looking at tackling this issue. Here's how I'm planning to go about this: 

1. define meteor, gleu in the enum here: https://github.com/promptfoo/promptfoo/blob/main/src/types/index.ts#L387
2. register a corresponding handler here: https://github.com/promptfoo/promptfoo/blob/main/src/assertions/index.ts#L231
3. define the handlers in their individual files similar to `rouge.ts` 

does this sound good? Let me know if you see any concerns 

Awesome @adityabharadwaj198! 

I just opened a PR with guidance on adding a new assertion: https://github.com/promptfoo/promptfoo/pull/3610 Please feel free to leave comments on the PR if you think parts of it can be improved. 

You can reference https://github.com/promptfoo/promptfoo/pull/3605, https://github.com/promptfoo/promptfoo/pull/2469, and https://github.com/promptfoo/promptfoo/pull/2081 as recent assertion PRs. 

Good luck! And send me an email when you're done to michael @ promptfoo.dev and I'll send you some swag. 

thanks @mldangelo !

@mldangelo I opened a PR for adding meteor score: https://github.com/promptfoo/promptfoo/pull/3776. Would love to hear your thoughts on it! 

Hey @sinedied, circling back on this one. Thanks for the suggestion back in September 2024!

We've since added all the metrics you mentioned:

- **BLEU** (`bleu`) #2081
- **GLEU** (`gleu`) #3674
- **METEOR** (`meteor`) #3776
- **ROUGE** (`rouge-n`) already existed at the time

These join our other deterministic assertions like `levenshtein`, `similar` (cosine/dot/euclidean), and `perplexity`.

Example usage:

```yaml
tests:
  - vars:
      input: "The quick brown fox jumps over the lazy dog"
    assert:
      - type: bleu
        value: "A fast brown fox leaps over a sleepy dog"
        threshold: 0.3
      - type: meteor
        value: "The quick brown fox jumps over the lazy dog"
        threshold: 0.8
```

If you ever need something more specialized (domain-specific metrics), the `python` and `javascript` assertion types let you plug in arbitrary scoring logic:

```yaml
assert:
  - type: python
    value: file://my_custom_metric.py
```

Closing this one out since the original request is covered. If there's a specific metric you'd still like to see built-in, open a new issue and we can discuss, always interested in what evaluation approaches people are using in practice.