Repository: promptfoo/promptfoo
Issue: Question : Can you run one suite of assertions then transform the llm response and apply a different set of outputs without calling the external LLM twice?
Root cause: Documentation Deficiency



So I have an LLM agent that returns a JSON format. Within the JSON there is an element that contains an SQL statement.

Example:
{
...
"sqlQuery":  "SELECT * ..."
...
}

I have build a promptfoo system that evaluates de SQL statements and also managed to extract with transform.py the SQL statement from it. Basically I am running a python script : generate_assertions.py and I get the assertions needed for sql in runtime_assertions

Now I am creating a seperate JSON validator for the JSON part of the answer.

My question is : How to have the 2 sets of evaluations run on the same output from the llm without running the system twice.

I was thinking of the following solution but then I feared that for the same set of questions (that are inside of the yaml assertions) I would get the same answer processed twice thus increasing the cost unnecessarily. 

Basically I have a set of 10 questions and with those 10 I create the **runtime_assertions_json.yaml** and **runtime_assertions_sql.yaml** is there a way to make sure the calls are unique and not double for each set of asserts ? 
```
providers:
  - id: "python:Interface.py"
    label: "Development JSON"
    config:
      model: "gpt-4o"
      tokens: 2000
      temperature: 0
defaultTest:
  options:
    transform: file://transform_json.py

tests: runtime_assertions_json.yaml

providers:
  - id: "python:Interface.py"
    label: "Development SQL"
    config:
      model: "gpt-4o"
      tokens: 2000
      temperature: 0
defaultTest:
  options:
    transform: file://transform_sql.py

tests: runtime_assertions_sql.yaml

```

Hi @SysOverdrive! Thanks for another great question! You have a few options to run different assertions on the same LLM output without duplicate calls:

1. Caching: Promptfoo caches results, so running twice won't incur extra costs. You can run one set of tests and then the other. If your cache is being invalidated, let me know and we can look into that.

2. Move transforms to individual tests instead of the default level. You can put a transform on each test. This might be the easiest if you want to do everything at once (I would try this first). Note that if you have a default transform and then transforms on individual tests, only the individual ones are run (we don't run both).

3. Update your transform to return both JSON and SQL outputs, and then update your assertions to grab the specific field you need.

I also want to mention that we have good built-in support for both JSON (https://www.promptfoo.dev/docs/guides/evaluate-json/#ensuring-that-outputs-are-valid-json) and SQL validation (https://www.promptfoo.dev/docs/configuration/expected-outputs/deterministic/#is-sql). You should try our built-ins if you haven't already.

Please feel free to share more (your transforms, a complete working example, etc.) if you would like more specific advice for your use case. Let me know if you need any clarification! Hope this helps!

Thanks for the info.