Repository: promptfoo/promptfoo
Issue: Implementing custom test assertions which reference both input and output
Root cause: Documentation Deficiency

I'm enjoying getting to know Prompt Foo - thank you for the effort and for sharing the knowledge!

I am not particularly familiar with JS or TypeScript, so I've been trying to implement a set of evaluation metrics we've been using for a specific use case e.g.:

- Word ratio: Calculate the ratio of the number of words in the output to the number of words in the input.
- Word length ratio: Calculate the ratio of the mean length of words in the output to the mean length of words in the output.

Both of these metrics reference the input and output for each LLM call.

Ideally I'd implement these by importing a module via the Python test assertion functionality, but looking at assertions.js I can see that would not work without some modifications (I don't think I can import a module within a call to Python's`print()` function).

So the options available to me are to:

- build some Javascript equivalents and include these over multiple lines in promptfooconfig.yaml. This feels a bit cumbersome
- modify assertions.js to import the function I need. Sounds fine, but will break following an update!
- implement a new test assertion (or two) to suit my needs.

I just wanted to ask if I might be missing a trick here before going down one of these paths? While the Python function is great for running say an anonymous function, or a tiny fragment of a Python script, it would be much more useful if an imported function could be called referencing `context.vars` and `output` as parameters.

Hi @richardguinness,

It sounds like it might help if the `javascript` and `python` assertion types could import full files, similar to the [prompt functions](https://promptfoo.dev/docs/configuration/parameters#prompt-functions) capability.

```yaml
- tests:
  - assert:
      - type: javascript
        value: myassert.js
```

In `myassert.js`:
```js
module.exports = (output, context) => {
  console.log(`Check ${output} and ${JSON.stringify(context.vars)}...`);
};
```

Does this sound like something you'd use?

Yes indeed this would be brilliant, so long as context and output were made available to the called Python script (by way of command line arguments?).

(Accidentally closed!)

This change has been landed on `main` and will be published in the next release.  It is documented here: https://promptfoo.dev/docs/configuration/expected-outputs#load-assertions-from-external-file

Bravo - thank you!

Just to confirm the thinking here:

1. Use Python `.py` (or `.js`) files for both `prompts` and a given assertion's `value` fields
2. Files use a `context['vars']['topic']` (Python syntax) to access variables

And that's how one can distribute `vars` to both input and expected output?

Yep!

This is really great to add a custom metric, e.g., F1 score on a classification task.
However, I'm not sure how to 
1. add multiple metrics in a python call, e.g., precision and recall. Perhaps a [Grading Result](https://www.promptfoo.dev/docs/configuration/reference#gradingresult) ComponentResult could help but not sure how to label different metrics when returning this dictionary/json object.
2. use a custom test file (`-t tests.csv`) along with a custom python metric file. It seems the -t overrides the python metric in conf file altogether. I have temporarily fixed by repeating all csv rows inside the config itself (not ideal).

Perhaps this already exists? @typpo