Repository: promptfoo/promptfoo
Issue: Slow evaluation with inner provider loop
Root cause: Algorithmic Error

Evaluation on multiple local Ollama models is very slow as currently in the code the outer loop is over prompts and inner loop over providers. I believe this leads Ollama to load up a local model (transfers gigabytes into memory) to just generate for a single prompt, and then discard the model and proceed to load up a next one. I'd assume it would be faster to loop over providers on outer loop and execute all (inner loop) prompts on the loaded model. 

That is, in code here https://github.com/promptfoo/promptfoo/blob/main/src/evaluator.ts#L652

          for (const prompt of testSuite.prompts) {
            for (const provider of testSuite.providers) {
            
have instead 

          for (const provider of testSuite.providers) {
            for (const prompt of testSuite.prompts) {

Maybe the same trivial change is needed also at https://github.com/promptfoo/promptfoo/blob/main/src/evaluator.ts#L524

Unfortunately I was not able to execute promptfoo from a locally altered repo (could not find instructions for that, only for self hosted dockerized instance), thus I'm afraid I cannot provide a PR.

If the change makes sense I suggest to keep both loops, and provide a command line option to allow user choose which looping to use. And of course update the docs accordingly. 

Also, it would be fabulous to also expand docs how to contribute, develop, test and propose changes for promptfoo.

Great suggestion, I think it makes sense to flip the loop nesting.

The local dev instructions are tucked away at the end of the repo README.
I'll make sure they are added to the website docs as well, along with
contribution guidelines.

On Wed, Jul 10, 2024, 1:06 AM ErrorEradicator ***@***.***>
wrote:

> Evaluation on multiple local Ollama models is very slow as currently in
> the code the outer loop is over prompts and inner loop over providers. I
> believe this leads Ollama to load up a local model (transfers gigabytes
> into memory) to just generate for a single prompt, and then discard the
> model and proceed to load up a next one. I'd assume it would be faster to
> loop over providers on outer loop and execute all (inner loop) prompts on
> the loaded model.
>
> That is, in code here
> https://github.com/promptfoo/promptfoo/blob/main/src/evaluator.ts#L652
>
>       for (const prompt of testSuite.prompts) {
>         for (const provider of testSuite.providers) {
>
> have instead
>
>       for (const provider of testSuite.providers) {
>         for (const prompt of testSuite.prompts) {
>
> Maybe the same trivial change is needed also at
> https://github.com/promptfoo/promptfoo/blob/main/src/evaluator.ts#L524
>
> Unfortunately I was not able to execute promptfoo from a locally altered
> repo (could not find instructions for that, only for self hosted dockerized
> instance), thus I'm afraid I cannot provide a PR.
>
> If the change makes sense I suggest to keep both loops, and provide a
> command line option to allow user choose which looping to use. And of
> course update the docs accordingly.
>
> Also, it would be fabulous to also expand docs how to contribute, develop,
> test and propose changes for promptfoo.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/promptfoo/promptfoo/issues/1125>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACLYJQPUNX3EPPMCIOPARTZLTTRPAVCNFSM6AAAAABKULIR6SVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQYDAMBVGEZTCMA>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>


Landed in #1132

Hi @ErrorEradicator, the fix has been merged and deployed in our latest release: [0.70.1](https://github.com/promptfoo/promptfoo/releases/tag/0.70.1).

We also plan to expand our documentation to include local development and contribution guidelines soon.

Thanks again for your valuable feedback!

In my first test I can confirm I got 25% speed increase in wall clock time.