Repository: promptfoo/promptfoo
Issue: Suggestion for a different approach to this project
Root cause: Documentation Deficiency

Hi! Thanks for making this. I tried to use this for my project, and promptfoo's approach didn't fit super well with what I was trying to do. I thought I'd leave some observations here in case it's useful to you, but obviously there's no pressure – maybe promptfoo just isn't the right tool for my usecase.

My usecase: I have AI-driven programs, and I want to evaluate how reliable they are, via repeated trials. I like how promptfoo offers a runner engine, does some analysis, and renders results in various formats.

However, there are also some areas where I find promptfoo to be a bit overbearing. What I really want is to give promptfoo a series of `(args: Record<any, any>) => Promise<string>` functions, and then use the promptfoo engine to do the analysis. Something like:

```ts
const trials = [
  { name: 'simple', inputParams: {a: 1} },
  { name: 'moderate', inputParams: {a: 1, b: 2} },
  { name: 'complex', inputParams: {a: 1, b: 2, c: 3} },
];

const candidates = [
    (inputParams) => myFirstAIProgram(inputParams),
    (inputParams) => mySecondAIProgram(inputParams),
    (inputParams) => myThirdAIProgram(inputParams),
];

const outputs = evaluate(trials, candidates);

await writeJsonFile(outputs);

function validate(output: string, inputParams: Record<any, any>) { /* ... */ }

const analysis = analyze(outputs, validate);
```

Key aspects here:
1. Instead of promptfoo calling the model directly, my AI program calls it. The prompt my AI program feeds to the model is complex and determined by runtime factors; trying to have promptfoo call the model directly doesn't work for that. (I know I can get around this with an `ApiProvider`, but it's pretty clunky to have to specify it in a separate file – I just want to pass a function.)
2. Fully custom validation instead of the limited and footgun-prone 'pass JS as a string' approach.
3. Running the analysis in a separate phase from the generation. Generation is very slow; analysis can be very fast. I may want to change my analysis without needing to rerun generation.
4. Also, preferably the Node API would let you output HTML, CSV, etc – I wasn't sure how to do that (or if it was possible) today.

Anyway, like I said – this is your project, and maybe what you're trying to do isn't a good fit for what I'm trying to do, which is totally fine. But in case the feedback was useful to you, I thought I'd leave this note here.

Thanks for this feedback, Nick.  In general, the node package needs some love - I've been pretty focused on the CLI, and right now the package is just a wrapper around the CLI.  Running evals in a script definitely has different requirements/ergonomics.

1. Agree it would be great to support `ApiProviders` that are functions, so you can hook into your full app logic.  Can add this one soon.
2. Also agreed
3. Makes sense, probably not too much of a lift to expose this separately.  Worth mentioning if you are leaning hard into test running, it may be easier to use an established test framework like Jest and use a custom matcher for semantic similarity or LLM grading ([example](https://promptfoo.dev/docs/integrations/jest#creating-custom-jest-matchers)).
4. The NodeJS API is getting support for `outputPath` shortly - https://github.com/promptfoo/promptfoo/issues/91

That all makes sense!

For (3) – I'm actually not in a Jest mindset at all. I'm very much exploring what different prompts do / just want an accuracy score; I don't have anything robust enough that I'd link Jest test pass/fails to it.

Also, in case it's helpful context, this is the framework that my AI-driven programs are built with: https://github.com/fixie-ai/ai-jsx.

#93 addresses some of this feedback, adding support for pure Javascript functions as Providers and Assertions.  

Since this issue was opened, promptfoo has much improved support for use as a [node library](https://promptfoo.dev/docs/usage/node-package) or with [custom API providers](https://promptfoo.dev/docs/providers/custom-api) or [script providers](https://promptfoo.dev/docs/providers/custom-script).  


Providers, prompts, and assertions can be passed as functions like in this example:
```ts
import promptfoo from 'promptfoo';

const prompts = [
  // Prompts can be raw text...
  'Write a haiku about: {{body}}',

  // Or message arrays...
  [{
      role: 'system',
      content: 'You are speaking to a time traveler. Begin your conversation.',
    },
    {
      role: 'user',
      content: '{{body}}',
    },
  ],

  // Or functions
  (vars) => {
    const genres = ['fantasy', 'sci-fi', 'mystery', 'horror'];
    const characters = {
      'fantasy': 'wizard',
      'sci-fi': 'alien',
      'mystery': 'detective',
      'horror': 'ghost'
    };
    const genre = genres[Math.floor(Math.random() * genres.length)];
    const character = characters[genre];
    return [{
        role: 'system',
        content: `You have encountered a ${character} in a ${genre} setting. Engage with the character using the knowledge typical for that genre.`,
      },
      {
        role: 'user',
        content: '{{body}}',
      },
    ];
  },
];

const providers = [
  // Call the OpenAI API GPT 3.5
  'openai:gpt-3.5-turbo',

  // Or use a custom function. Call an LLM API in this function, or any other logic.
  (prompt, context) => {
    console.log(`Prompt: ${prompt}, vars: ${JSON.stringify(context.vars)}`);
    return {
      output: '<LLM output>',
    };
  },
];

const tests = [{
    vars: {
      body: 'Hello world',
    },
  },
  {
    vars: {
      body: "I'm hungry",
    },
    assert: [{
      type: 'javascript',
      value: (output) => {
        // Logic for checking LLM output goes here...
        const pass = output.includes("foo bar");
        return {
          pass,
          score: pass ? 1.0 : 0.0,
          reason: pass ? 'Output contained substring' : 'Output did not contain substring',
        };
      },
    }, ],
  },
];

(async () => {
  const results = await promptfoo.evaluate({ prompts, providers, tests });
  console.log(JSON.stringify(results, null, 2));
})();
```

Hope this is useful and address your asks.  I assume promptfoo wasn't a great fit at the time, so I'm curious what you wound up using for evals.