Repository: promptfoo/promptfoo
Issue: Expose `runAssertion` for advanced integration
Root cause: Unimplemented Feature Gap

I would like to run promptfoo assertions on some previously generated output with the node.js integration.  I know that I can create a test suite with the `echo` provider similar to the `eval` command.  

```
    const testSuite = {
      prompts: [outputString],
      providers: ["echo"],
      tests: [{ assert: [{type: "contains", value: "foo"}] }],
    };

    const result = await promptfoo.evaluate(testSuite, {});
```

Or I could do this for a batch of results using `prompts: ["{{output}}"]` and some `vars`.  

However, there are some limitations with the above.  As far as I can tell, can't pass `cost`, `logProbs`, `latencyMs`, or provide token usage.  This limits some of the assertions I can run.  There are also the `matchesSimilarity` assertions exposed for use in Jest but those are also limiting.

The `runAssertion` or `runAssertions` methods would be a nice interface to expose for more complex integrations.  Or, possibly the `evaluate` method could support more options to better support previously generated responses.  Is `runAssertion` a method you would be willing to export?  I understand it is an internal method and you would have to be much more careful about making changes.  But, it would enable Promptfoo to be more useful for integration into other tooling like sampling production outputs or running assertions within a prompt exploration UI.

I just realized I could create a custom provider that can provide all of the outputs except for `latencyMs`.  I could use that custom provider rather than the "echo".  In theory I could delay the custom provider output the correct time to replicate the latency as well but that would not be ideal.  

I would still prefer to use `runAssertion` directly but I might be able to get `evaluate` to support everything except latency.

For anyone else curious about the code, it looks something like this to run one or more assertions against a single previously generated response.

```javascript

  const messages = request.messages;
  const output = result.choices[0].message.content; 
  const tokenUsage = {
    total: result.usage.total_tokens,
    prompt: result.usage.prompt_tokens,
    completion: result.usage.completion_tokens,
  };
  // TODO: Calculate cost
  const cost = 0;
  const assert = [{type: "contains", value: "foo"}];
  
  const testSuite = {
    prompts: [JSON.stringify(messages, null, 2)],
    providers: [
      async () => {
        return { output, tokenUsage, cost };
      },
    ],
    tests: [{ assert }],
  };

  const evalResults = await promptfoo.evaluate(testSuite, {});
  ```

I think running against multiple would just require putting `output`, `tokenUsage`, `cost` into vars and then pulling it off the context in the provider function.  And including each prompt in prompts.

Hey @mihalik, just opened a small PR to do this :) Thank you for the input! 

Thank you!  This will make things much simpler and lets me use latency tests again!

A fix for this was deployed in [0.68.0](https://github.com/promptfoo/promptfoo/releases/tag/0.68.0) Please let us know if you run into any other issues and thanks again for suggesting it!