Repository: promptfoo/promptfoo
Issue: Support for dynamic external tool call Definitions - i.e. Python, TS, JS
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Currently I can load tool definitions from external file, but only static. I want to be able to provide in a script, like a lot of the other surfaces in promptfoo.
https://github.com/promptfoo/promptfoo/blob/main/examples/tool-use/external_tools.yaml

Tools calls for Anthropic sdk happen [here](https://github.com/promptfoo/promptfoo/blob/5cf28619843fab2c757a8974ed91d485691c336e/src/providers/anthropic.ts#L262) and are routed [here](https://github.com/promptfoo/promptfoo/blob/5cf28619843fab2c757a8974ed91d485691c336e/src/util/index.ts#L1065). Seems similar for other models. This currently doesn't support evaluating python, TS, or JS

**Describe the solution you'd like**
The same behavior with prompts and test files, where this can be provided dynamically by a script.

**Describe alternatives you've considered**
Use an external script to generate these tool defs ahead of running promptfoo. Not ideal.

Hey @dhodun, this is a great idea! We’d love to support this, and if you’re interested in contributing, I would be more than happy to help guide you through it. Also, if you have time to work on it, I’d love to send you some Promptfoo swag as a thank-you! If not, no worries. I’ll add this to my backlog and see when we can get to it. Really appreciate you taking the time to suggest this!

Thanks @mldangelo ! Might take a stab at it.

In general, should I follow the test case generation pattern for simple python read? https://github.com/promptfoo/promptfoo/blob/e5cf9dcd0cbc5f0d315ac9bc446d4252049512d6/src/util/testCaseReader.ts#L106-L117 which uses https://github.com/promptfoo/promptfoo/blob/26e10e9547139d51c41aadeb2642e58b59bfbeec/src/python/pythonUtils.ts#L87

This would probably involve creating some equivalent of https://github.com/promptfoo/promptfoo/blob/e5cf9dcd0cbc5f0d315ac9bc446d4252049512d6/src/util/testCaseReader.ts#L70-L73 since `maybeLoadFromExternalFile` is intended to be basic JSON / YAML loader? Or is there an opportunity here to make a more generic loader that can call scripts anywhere? I'll admit, this was my first impression of the tool that a lot of the config was super extensible this way, just now realizing that external scripts have only been implemented in the most obvious places (`testsCases`, `Providers`, etc).

Rather than the python parsing in Prompts, which seems a little more bespoke? https://github.com/promptfoo/promptfoo/blob/26e10e9547139d51c41aadeb2642e58b59bfbeec/src/prompts/processors/python.ts#L82-L101

I also see, at the moment for testCases it only seems wired up to call various functions and not pass along inputs or context. Maybe I wasn't following best practices in my config, but I was mostly looking to iterate on slight differences in tool defs. With prompts, I found myself creating different getter functions to output slight prompt variations, something that would have been much better handled with a parameterized getter. (Or I could just dynamically create different getters). Maybe inline templating or something like that would be helpful, but as new user I was more comfortable just using the Python examples.

https://github.com/promptfoo/promptfoo/blob/361e8b6e5620fcb838b6f5008a4af9346e3c4784/src/util/transform.ts#L66 I see does pass along context, etc.

Actually, I see dynamic test cases was just added. Seems like this could be expanded upon more generically?

https://github.com/promptfoo/promptfoo/pull/2993/files

The pattern introduced in #2993 is a good one to follow and yes we could definitely abstract readStandaloneTestsFile to also support passing context. This is a really great idea. If you have time to run with it I’d love to see what you come up with. If not I will try to get to it next week. 

I'd love to take a shot.

Confirm my understanding, but it seems that the tool loading is pretty basic, any validation is done in the provider, as I assume different providers have different formats.

For that reason, it might be reasonable to either extend `maybeLoadFromExternalFile` to accept `.py` and `.js` or wrap it with something than can also execute scripts without additional validation?

`maybeLoadFromExternalFile` is used fairly frequently. While it might be nice to allow that to take script input absolutely everywhere, I can see that having unintended consequences and being too broad a change. What do you think?

@mldangelo is there any progress? very glad to use this feature!