Repository: promptfoo/promptfoo
Issue: scenarios tied to prompts and not to vars
Root cause: Environment Incompatibility

Is there a way to (other than iteratively running) so that there are grouped sets of prompts that use the same vars?

I think something like scenarios but linked to sets of prompts.

E.g., Group A - prompt1, prompt2, prompt 3
Group B - prompt4, prompt5, prompt6

And run / have separate GroupA and Group B in one run - the var is the same across Group A and Group B.

Thanks. New to promptfoo but appreciating its power.

@aisonym - Thank you for your question and for using promptfoo! 

To address your scenario of grouping sets of prompts that use the same vars, you can utilize the prompt labels feature. Please check out our examples here: [Prompt Labels Example](https://github.com/promptfoo/promptfoo/tree/main/examples/prompt-labels). 

With this approach, you can assign a label to each prompt and then add a `prompts:` attribute to each provider. This should allow you to run separate groups like Group A (prompt1, prompt2, prompt3) and Group B (prompt4, prompt5, prompt6) within a single run while maintaining the same vars across both groups.

If this solution doesn't fully address your needs or you have further questions, please let me know and re-open the issue. I'm happy to help!

Thanks for your response. I did check out labels.

I find when running prompts via a file it doesn't replicate the experience of running prompts individually (put a separate note on GitHub). 

E.g., 

If the .txt is:
prompt1
prompt2

I get a single response (with the label) that is like asking the model - hey answer prompt1, prompt2 and I get much shorter answers. Recognize I may be missing something - but labels don't work on this.



/reopen

@aisonym Are your prompts in your txt file delimited by 
```
---
```
or a custom delimiter (`process.env.PROMPTFOO_PROMPT_SEPARATOR`)? If not, they could be loaded all at once.  Can you please provide an example of your configuration?

Thanks for the response.

Yes. Delimited by ---.

If I run without a label the prompt text file runs the questions separately as expected (filename: Example.txt).
```
Why is the 4th of July special to the {{country}}?
---
When did the {{country}} declare independence?
```

But if I run with a label: 
```
prompts:
  - id: file://Prompts/Example.txt
    label: Happy 4th
```

The multiple questions get asked as a single question (I'm also doing an answer-relevance test):

[anthropic:messages:claude-3-5-sonnet-20240620] Happy 4th
--
[PASS] (0.79)The 4th of July is special to the United States of America because it is Independence Day, the country's national holiday commemorating the adoption of the Declaration of Independence.To answer your second question:The United States of America declared independence on July 4, 1776. On this day, the Continental Congress, representing the 13 American colonies, formally adopted the Declaration of Independence. This document proclaimed the colonies' separation from Great Britain and their establishment as free and independent states.While the vote for independence actually took place on July 2, 1776, the Declaration was officially adopted two days later, on July 4. This is why Americans celebrate Independence Day on the 4th of July each year.It's worth noting that although independence was declared on this date, the American Revolutionary War continued until 1783, when Great Britain formally recognized the United States as an independent nation through the Treaty of Paris.Pass Reason: All assertions passed

What I'm trying to achieve is to have e.g., a group of ten questions that are grouped into two groups of 5 such that the group  label is included in the output for easier analysis.










@aisonym Unfortunately I am having trouble reproducing your issue. I ran this exact config

#### promptfooconfig.yaml
```yaml
prompts:
  - id: file://Prompts/Example.txt
    label: Happy 4th
providers:
  - anthropic:messages:claude-3-5-sonnet-20240620
tests:
  - vars:
      country: United States
    assert:
      - type: answer-relevance
```

with 
#### Prompts/Example.txt
```text
Why is the 4th of July special to the {{country}}?
---
When did the {{country}} declare independence?
```
And got
<img width="836" alt="Screenshot 2024-07-04 at 5 12 27 PM" src="https://github.com/promptfoo/promptfoo/assets/7235481/c6bb45e3-0646-422c-82a5-14926712c7d3">

Can you please provide a full config and I'll try to fix it? 


Thanks for looking. The only difference is that when I'm running (didn't show this in original) I'm doing two providers.

```
# Learn more about building a configuration: https://promptfoo.dev/docs/configuration/guide
description: "testing labels"

prompts:
  - id: file://Prompts/Example.txt
    label: Happy 4th


providers:
  - id: "openai:gpt-3.5-turbo"
  - id: "anthropic:messages:claude-3-5-sonnet-20240620"


defaultTest:
  assert:
    - type: answer-relevance

tests:
  - vars:
      country: United States of America
```

Would you mind running `promptfoo share` immediately after running the eval, and then give us the shared URL?  That will help debug.



@aisonym with your configuration I get:

<img width="1184" alt="Screenshot 2024-07-04 at 11 37 01 PM" src="https://github.com/promptfoo/promptfoo/assets/7235481/95c4e4b8-0cf2-4414-ac3d-1859ca82295b">

Is this identical to what you're getting? I am not sure if I understand your issue. Would you mind running `npx promptfoo@latest share` so we can review it? Thanks!



Thankful for the support - don't mind sharing.

https://app.promptfoo.dev/eval/f:b3aa2971-c36d-49eb-89a4-bca083625260

<img width="1030" alt="Screenshot 2024-07-05 at 5 03 19 AM" src="https://github.com/promptfoo/promptfoo/assets/173861774/f0951016-b22f-488a-98c2-d60d2db56da6">


Noting that I'm still running 0.66 (shows up in the screen shot) but what I get is different than yours.






FWIW - updating to 0.68 solved the issue and I now get a screen similar to yours.

Sorry for the runaround.

Glad it's working now!

On Fri, Jul 5, 2024, 5:18 AM aisonym ***@***.***> wrote:

> Closed #1076 <https://github.com/promptfoo/promptfoo/issues/1076> as
> completed.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/1076#event-13405477723>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJR2WLNZ6M24FITTH53ZK2FJVAVCNFSM6AAAAABKKD6TG6VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTGQYDKNBXG43TEMY>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>