Repository: promptfoo/promptfoo
Issue: Python and Javascript asserts from file
Root cause: Documentation Deficiency

Currently Python asserts must be one liners.
Javascript supports multi-line asserts.
Nonetheless, for complex evals, providing a path to a `.js` or `.py` file is sensible.
This allows reusing the code for an eval without cluttering `promptfooconfig.yaml`.

The eval that I'm aiming to use is written in Python. Due to better support for Javascript I've converted the assertion to Javascript. However I'm running into issues with the asynchronous nature of Javascript. A call to an API which performs part of the evaluation with `fetch` causes my program to return a promise to a boolean instead of a plain boolean. Executing this program with plain `node` works fine. Within promptfoo, I receive the following error `[FAIL] Custom function threw error: Custom function must return a boolean or number`. While I think that the proper solution would be an implementation of the above features, I'd also appreciate a hint on how to hack around my problem.

Hey @Wehzie,

Just pushed a change 7308df2484f49f0f176a5abfcef971b281e8603b that should resolve Promises returned by an external JS file.  Can you give it a try?

btw, multiline Python asserts are also supported as of a recent update.  You should be able to pass a multiline string.  It expected the assertion result to be `print`ed (e.g. `print(True)`, `print(0.5)`, etc), which is a bit weird, but it should work nonetheless.

Thank you @typpo.
All works. Issue is correctly closed.

Lessons learned:
Paths to scripts, either `.py` or `.js` are provided with
```
    assert:
      - type: javascript
        value: file://relative/path/to/script.js
      - type: python
        value: file///absolute/path/to/script.py
```

For Python scripts, since `print` triggers the return, other prints should be avoided.
For debugging purposes, a simple `if DEBUG: print` does the job.
Multiline Python works like JavaScript, for example

```
    assert:
      - type: python
        value: |
          some_var = "hello"
          print(some_var == "hello")
```

Thank you for documenting this in https://github.com/promptfoo/promptfoo-docs/commit/9b8c04fa85b9cf32284883518b3e1b066ad43e48

For JavaScript files, the to-be-executed function must be exported. For example, `main`, must be exported with `module.exports = main;`.
JavaScript `async` works as expected. For an example of `assert.js`, see

```
const VALIDATION_ENDPOINT = "https://example.com/api/validate";

async function evaluate(modelResponse) {
    try {
        const response = await fetch(VALIDATION_ENDPOINT, {
          method: 'POST',
          headers: {
            'Content-Type': 'text/plain'
          },
          body: modelResponse
        });
  
        const data = await response.json();
        return data;
    } catch (error) {
        throw error;
    }
}

async function main (output, context) {
    const success = await evaluate(output);
    console.log(`success: ${testResult}`);
    return success;
  }
  
module.exports = main;
```

When JavaScript or Python asserts are defined within `promptfooconfig.yaml` the variable `output` is injected and can be accessed without declaration.
This variable holds the model response.
When JavaScript or Python asserts are loaded with `file://`, the output is accessed with positional arguments.
The call to Python is of the form `python path/to/assert.py model_output context`.
The context variable holds the prompt and the variables of the test case.
The call to JavaScript is of the form `main(output, context)`.
For an example of a `assert.py`, see

```
import sys

def main():
    if len(sys.argv) >= 3:
        output = sys.argv[1]
        context = sys.argv[2]
    else:
        raise ValueError("Model output and context are expected from promptfoo.")
    processed = preprocess_output(output)
    success = test_output(processed)
    return success
  
print(main())
```

I missed most of this from the documentation somehow. Anyway, thanks for implementing all this!

Thanks for the breakdown, @Wehzie.  I can probably make this clearer in the documentation, and/or break it out onto its own page.

This change (https://github.com/promptfoo/promptfoo-docs/commit/69e29d2e8371a60905f13ca217d4ab844888d102) improves assertion documentation and adds much of the detail you recorded above.