Repository: promptfoo/promptfoo
Issue: Providing different prompts per model always takes arguments of last provider
Root cause: Algorithmic Error

Since version `0.69.1` (up to `0.70.0`), providing different prompts per model is broken, if the same provider id is used. It will always take the prompt arguments that are configured for the last provider.

Here is a reproducing example:

promptfooconfig.yaml:
```
prompts:
    - prompt1
    - prompt2

providers:
    - id: 'python:provider.py'
      label: 'Provider 1'
      prompts:
          - prompt1
    - id: 'python:provider.py'
      label: 'Provider 2'
      prompts:
          - prompt2

tests:
    - description: 'Assert equals'
      assert:
          - type: equals
            value: 'output'
```

provider.py:
```
def call_api(prompt: str, options: dict, context: dict):
    print(f"Prompt: {prompt}")
    return { "output": "output" }
```

When you run `LOG_LEVEL=debug promptfoo evaluate`, you can see that it will print out `Prompt: prompt2` two times, instead of `Prompt: prompt1` for the first provider, and `Prompt: prompt2` for the second. Note that this is a regression from `0.69.0`, where this case worked without problems. Note also that when there is no `label` provided for the providers, then it also didn't work in older versions (e.g. `0.67.0`).

The issue is maybe caused by https://github.com/promptfoo/promptfoo/pull/1106, removing the code commented with `Hack(ian): Override id with label`.

My use case is: I have always the same custom provider, but depending on the prompt, I will do different things. My current workaround is to create separate providers for each prompt, but this is an overhead I want to avoid in the future.

Thanks for flagging this. I think you're probably right that the id/label
override was necessary in that code. We'll get a fix out quickly.

On Wed, Jul 10, 2024, 2:43 AM yakuhzi ***@***.***> wrote:

> Since version 0.69.1 (up to 0.70.0), providing different prompts per
> model is broken, if the same provider id is used. It will always take the
> prompt arguments that are configured for the last provider.
>
> Here is a reproducing example:
>
> promptfooconfig.yaml:
>
> prompts:
>     - prompt1
>     - prompt2
>
> providers:
>     - id: 'python:provider.py'
>       label: 'Provider 1'
>       prompts:
>           - prompt1
>     - id: 'python:provider.py'
>       label: 'Provider 2'
>       prompts:
>           - prompt2
>
> tests:
>     - description: 'Assert equals'
>       assert:
>           - type: equals
>             value: 'output'
>
> provider.py:
>
> def call_api(prompt: str, options: dict, context: dict):
>     print(f"Prompt: {prompt}")
>     return { "output": "output" }
>
> When you run LOG_LEVEL=debug promptfoo evaluate, you can see that it will
> print out Prompt: prompt2 two times, instead of Prompt: prompt1 for the
> first provider, and Prompt: prompt2 for the second. Note that this is a
> regression from 0.69.0, where this case worked without problems. Note
> also that when there is no label provided for the providers, then it also
> didn't work in older versions (e.g. 0.67.0).
>
> The issue is maybe caused by #1106
> <https://github.com/promptfoo/promptfoo/pull/1106>, removing the code
> commented with Hack(ian): Override id with label.
>
> My use case is: I have always the same custom provider, but depending on
> the prompt, I will do different things. My current workaround is to create
> separate providers for each prompt, but this is an overhead I want to avoid
> in the future.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/1127>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJWIX24HMYU4D4KHT63ZLT63NAVCNFSM6AAAAABKUQ2CCCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQYDAMRVHE4DCMI>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>