Repository: promptfoo/promptfoo
Issue: Poor python performance: Custom python provider initializes on every test case
Root cause: Architectural Constraint

**Describe the bug**
The docs for the custom python provider specifically say that to improve performance, put common setup outside of the `call_api()` functions:

https://www.promptfoo.dev/docs/providers/python/#performance-optimization

However, a trivial test shows that this doesn't work. This makes tests run a lot slower when initializing our own model.

**To Reproduce**

** Remember to use `--no-cache`**

`promptfoo eval --no-cache`

```yaml
description: "Simple repro"

prompts:
  - Hello world, how are you today?
  - Another prompt
  - Yet another prompt
  - One more prompt
  - The last prompt
  - Not really a prompt
  - Okay
  - Just a few more
  - One more
  - Wobbles
  - Blobs
  - Jibberish
  - Garbage
  - Random
  - Garble
  - Jibber

providers:
  - file://custom_python_provider.py
```

### custom_python_provider.py
```py
import logging

logger = logging.getLogger(__name__)
logger.warning("Initializing script")

async def call_api(prompt, options, context):
    # Needed to make warning messages show
    await asyncio.sleep(1)
    return {"output": "Hello world"}
```

**Expected behavior**
The python script should be reused and we shouldn't see "Initializing script" repeatedly.

**Screenshots**

<img width="1117" height="449" alt="Image" src="https://github.com/user-attachments/assets/e1796ac2-d0b5-443d-a275-bba763276b80" />


**System information:**

If possible, please output the results of `promptfoo debug` and paste the output here.

```json
{
  "version": "0.118.3",
  "platform": {
    "os": "darwin",
    "release": "24.6.0",
    "arch": "arm64",
    "nodeVersion": "v22.11.0"
  },
  "env": {},
  "configInfo": {
    "defaultConfigPath": "/Users/dougli/code/lightfern/benchmarks/src/benchmarks/experiment_configs/text_after_cursor/promptfooconfig.yaml",
    "configExists": true,
    "configContent": {
      "config": {
        "tags": {},
        "description": "Ensure model doesn't reproduce existing text after the cursor",
        "prompts": [
          "Hello world, how are you today?",
          "Another prompt",
          "Yet another prompt",
          "One more prompt",
          "The last prompt",
          "Not really a prompt",
          "Okay",
          "Just a few more",
          "One more",
          "Wobbles",
          "Blobs",
          "Jibberish",
          "Garbage",
          "Random",
          "Garble",
          "Jibber"
        ],
        "providers": [
          "file://promptfoo_completions.py:call_api"
        ],
        "tests": [],
        "scenarios": [],
        "env": {},
        "sharing": false,
        "outputPath": [],
        "extensions": [],
        "metadata": {}
      },
      "testSuite": {
        "description": "Ensure model doesn't reproduce existing text after the cursor",
        "tags": {},
        "prompts": [
          {
            "raw": "Hello world, how are you today?",
            "label": "Hello world, how are you today?"
          },
          {
            "raw": "Another prompt",
            "label": "Another prompt"
          },
          {
            "raw": "Yet another prompt",
            "label": "Yet another prompt"
          },
          {
            "raw": "One more prompt",
            "label": "One more prompt"
          },
          {
            "raw": "The last prompt",
            "label": "The last prompt"
          },
          {
            "raw": "Not really a prompt",
            "label": "Not really a prompt"
          },
          {
            "raw": "Okay",
            "label": "Okay"
          },
          {
            "raw": "Just a few more",
            "label": "Just a few more"
          },
          {
            "raw": "One more",
            "label": "One more"
          },
          {
            "raw": "Wobbles",
            "label": "Wobbles"
          },
          {
            "raw": "Blobs",
            "label": "Blobs"
          },
          {
            "raw": "Jibberish",
            "label": "Jibberish"
          },
          {
            "raw": "Garbage",
            "label": "Garbage"
          },
          {
            "raw": "Random",
            "label": "Random"
          },
          {
            "raw": "Garble",
            "label": "Garble"
          },
          {
            "raw": "Jibber",
            "label": "Jibber"
          }
        ],
        "providers": [
          {
            "options": {
              "config": {
                "basePath": "/Users/dougli/code/lightfern/benchmarks/src/benchmarks/experiment_configs/text_after_cursor"
              },
              "env": {}
            },
            "isInitialized": false,
            "initializationPromise": null,
            "scriptPath": "promptfoo_completions.py",
            "functionName": "call_api",
            "label": "",
            "config": {
              "basePath": "/Users/dougli/code/lightfern/benchmarks/src/benchmarks/experiment_configs/text_after_cursor"
            }
          }
        ],
        "providerPromptMap": {},
        "tests": [],
        "scenarios": [],
        "defaultTest": {
          "metadata": {},
          "options": {}
        },
        "nunjucksFilters": {},
        "extensions": []
      },
      "basePath": "/Users/dougli/code/lightfern/benchmarks/src/benchmarks/experiment_configs/text_after_cursor",
      "commandLineOptions": {}
    }
  }
}
```

- Promptfoo version: 0.118.3

**Additional context**
None