Repository: promptfoo/promptfoo
Issue: feat: transformations / path
Root cause: Unimplemented Feature Gap

As mentioned earlier `contains-json` is a neat output test. It'd be also great to be able to transform the input, like already somewhat possible with `prefix` and `suffix`.

I propose expanding on this and introducing:
- a `"transforms"` property for both global and specific `vars`, which would allow transforming the input using arbitrary js.
- a path() keyword which points towards a file

### Use-Case:
As a dev, I have a JSON example for my prompt that I keep in separate .json file for maintainability (readability / formatting, ...), but I'd like the generated prompt to contain the stringified version to omit unnecessary tokens and set the expected outcome.

It'd be super convenient to being able to simply do:

```yaml
  - vars:
      outputExample: path("example.json")
      transforms: path("stringify_function.js")
```

`stringify_function.js`
```ts
function( input:string ) {
    const parsedJSON = JSON.parse( input )
    return JSON.stringify(parsedJSON)
}
```

_Note: Made several edits to this_

This would also be useful for `global variables` that can be used within the prompt itself. 
```yaml
prompts:
   vars:
     - prompt_example_var: ...
```
or
```yaml
globalProperties:
   vars:
     - prompt_example_var: ...
```

So we don't need to define them for each `- vars` item.

For clarity, a full example:

---

```markdown
Think about the text below and output an object with these properties:
{
positive: number,
negative: number,
neutral: number,
tags: string[]
}

Example:

--
Input:
{{ input_example_review_1 }}

Output:
{{ output_example_json_1 }}

--

Input:
{{ user_query_input }}

Output:
```

Here, I want to only test `user_query_input`, but use vars for the example to:
- keep the main prompt clean
- add transforms to the output_example_json_1 (stringify)

Also, this setup allows to occasionally add more or less examples if there are different prompts without pasting the full examples into each prompt and keeping them in sync.

---

As an alternative, being able to add custom nunjucks functions would also help.

I have started work on a PR for global objects, and was thinking that defaultTests could also be nicely folded into this .

```yaml
globals:
  - vars
  - tests # prev defaultTest
  - customNunjucksFunctions
```

Although `vars` and `customNunjucksFunctions` are both piped the same way into `nunjucks.renderString(prompt, vars)`, it's cleaner to keep them separate so both can have their own signatures and properly TS checked.

Hey, sorry for the delay here, been traveling and haven't gotten the chance to dig deeper :)

I'm wondering if the global vars case is handled by `defaultTest` already.  The `vars` in the default test are by default merged into every test case.

I think I'll first implement the `customNunjucksFunctions` which I definitely need and make it a PR.
Safe travels (nomad? :P ) !
(Afterwards, maybe I can handle the other parts, at least for my use-case, with some cascading templating.)

Hey @MentalGear, going to close this issue as it is a bit stale, feel free to re-open if you're still interested in implementing parts of it!