Repository: promptfoo/promptfoo
Issue: Question: how to aggregate multiple `prompts.txt`
Root cause: Unimplemented Feature Gap

I have a few fundamentally different conversations I want to test, let's say 3 for simplicity.  So thus I have three different `prompts.txt` files.  Each `prompts.txt` file has its own unique set of variables and tests.

How can one either:
- Have one `promptfooconfig` that is able to have multiple (`prompts.txt`, set of `vars`, set of `assert`s)
- Aggregate the results of 2+ `promptfooconfig` from different subdirectories?

So I see for the [`promptfoo-action`](https://promptfoo.dev/docs/integrations/github-action#using-the-github-action), it's looking in `prompts/`.  What does the directory structure within `prompts/` look like?  Are there like subfolders with many pairs of `prompts.txt`/`promptfooconfig.yaml`s?

```txt
prompts
├── can-fly
│   ├── promptfooconfig.yaml
│   └── prompts.txt
└── can-swim
    ├── promptfooconfig.yaml
    └── prompts.txt
```

---

Update, just reading in the [example Python notebook](https://promptfoo.dev/docs/integrations/python), I see there's a `mathprompt.txt`/`mathconfig.yaml`.  So alternately, is it encouraged to deviate from `prompts`/`promptfooconfig` namings?

```txt
prompts
├── can-fly-config.yaml
├── can-fly-prompts.txt
├── can-swim-config.yaml
└── can-swim-prompts.txt
```

How can one combine the results from a `can-fly-config.yaml` with another `can-swim-config.yaml`?

Is the desired result to have all the vars and tests combined into a single view?  In this case, would it only make sense if there was some overlap between the vars/prompts/models being tested?

I think it's combining the results of 2+ configs (with separate vars/prompts/tests) into:

- Main ask: aggregated statistics
- Nice to have: one `promptfoo view`

So lets say `can-fly` will have 10 passes and 2 failures, and `can-swim` will have 6 passes and 4 failures.  How can those be combined into a summary statistic (and possibly one `promptfoo view`), with just one `promptfoo` invocation?

In other words, it's like having `promptfoo` run on an entire directory of configs, and report an aggregated result

Got it.  I think https://github.com/promptfoo/promptfoo/pull/285 should solve this use case, both aggregated stats and aggregated view.  The web view might get a bit unwieldy if the evals don't have prompt/provider/var overlap.

Oh nice I like that, sweet! Can you add the ability to crawl directories?  Something like `promptfoo eval -d prompts`

Added :)  Will land the PR shortly

On Sun, Nov 19, 2023 at 2:57 PM James Braza ***@***.***>
wrote:

> Oh nice I like that, sweet! Can you add the ability to crawl directories?
> Something like promptfoo eval -d prompts
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/170#issuecomment-1818005252>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJVWZAWSNRV5PQRTHS3YFKFDVAVCNFSM6AAAAAA5AUIAVGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJYGAYDKMRVGI>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


Looks like the PR landed, closing this out, thank you again!