Repository: promptfoo/promptfoo
Issue: Custom provider ignored during red teaming generation
Root cause: Documentation Deficiency

**Describe the bug**
Remote generation via the `api.promptfoo.dev` backend cannot be switched for adversarial prompts generation.

**To Reproduce**
Even if the custom adversarial provider is configured:
```
redteam:
  purpose: "Helpful assistant"
  provider: vertex:gemini-1.5-pro-001
```
promptfoo still attempts to connect to its backend:
```
% PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=false  npx promptfoo@latest redteam generate
...
Generating | █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ | 1% | 2/124 | Extracting entitiesError using remote generation for task 'entities': Error: Request failed after 4 retries: request to https://api.promptfoo.dev/v13
Error using remote generation, falling back to local extraction: Error: Request failed after 4 retries: request to https://api.promptfoo.dev/v1/generate failed, reason: connect ECONNREFUSED 127.0.0.1:443
```
I intentionally redirected `api.promptfoo.dev` to my localhost to see if the connection is really switched off.

Now, when I run 
```
% PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true  npx promptfoo@latest redteam generate
```
I get the following error:
```
Error: Invariant failed: fetchRemoteTestCases should never be called when remote generation is disabled
    at invariant (/Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/tiny-invariant/dist/tiny-invariant.cjs.js:14:11)
    at fetchRemoteTestCases (/Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/redteam/plugins/index.js:30:34)
    at action (/Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/redteam/plugins/index.js:109:20)
    at /Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/promptfoo/dist/src/redteam/index.js:235:39
    at /Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/async/dist/async.js:151:38
    at /Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/async/dist/async.js:2589:44
    at replenish (/Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/async/dist/async.js:447:21)
    at /Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/async/dist/async.js:452:13
    at Object.eachLimit (/Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/async/dist/async.js:2719:36)
    at /Users/vaclav.belak/.npm/_npx/81bbc6515d992ace/node_modules/async/dist/async.js:220:25
```

**Expected behavior**
According to this: https://www.promptfoo.dev/docs/red-team/configuration/#remote-generation I would expect that when the remote generation is switched off, only the custom/overriding provider would be used.

**System information (please complete the following information):**
Mac OS X Sonoma 14.6.1
- Promptfoo version:
0.93.3
**Additional context**
This is a very important feature for my use-case since due to the data confidentiality, I cannot send any samples to an non-approved backend.


Thanks for flagging. It's good that the invariant caught it before making a request.

I think what is happening is that you are using a plugin that requires remote generation.  This is the case for a handful of minor plugins:
```
  'ascii-smuggling',
  'bfla',
  'bola',
  'competitors',
  'hijacking',
  'religion',
  'ssrf',
```
We've moved some of these plugins to the backend for reliability because we are increasingly seeing refusals from the most recent versions of e.g. GPT-4o and Claude (not sure about Gemini).

Thanks a lot Ian for your answer. It is true that this happens only to specific plugins. I wonder would it be possible to override the grader model even for these plugins/tests? There are a couple of reasons why this is a legitimate use-case:

1. The user may use e.g. Gemini remote model with content-filtering switched off which makes it less sensitive than GPT-4o or Claude.
2. The user may host a local model, like Llama 3 or whatever
3. The user might not be allowed to send any data outside their organisation.

Thanks for reporting this and for the follow-up from @ijansky.

The `redteam.provider` config controls the provider used for **local generation and grading**, but certain plugins require promptfoo's hosted inference due to model refusals from standard LLMs (GPT-5, Claude 4.5, Gemini refuse to generate certain adversarial content).

**To run entirely locally:**

```bash
export PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true
```

This disables remote-only plugins (marked with 🌐 in the [plugin documentation](https://www.promptfoo.dev/docs/red-team/plugins)) and uses your configured provider for everything else. You'll need `OPENAI_API_KEY` or a custom provider configured.

**Plugins that work locally** include: `contracts`, `debug-access`, `excessive-agency`, `hallucination`, `imitation`, `overreliance`, `pii`, `policy`, `prompt-extraction`, `rbac`, `shell-injection`, `sql-injection`, and others.

**Remote-only plugins** (require hosted inference): `harmful:*`, `bias:*`, domain-specific plugins (medical, financial, etc.), and security plugins like `ssrf`, `bola`, `bfla`.

For complete air-gapped operation with all plugins, see [Promptfoo Enterprise](https://www.promptfoo.dev/docs/enterprise).

**Documentation:**
- [Data Handling and Privacy](https://www.promptfoo.dev/docs/red-team/troubleshooting/data-handling) - explains remote-only plugins and local configuration
- [Remote Generation Configuration](https://www.promptfoo.dev/docs/red-team/configuration#remote-generation)

The invariant error you experienced in v0.93.3 has been fixed in later versions - the code now properly checks if remote generation is disabled before attempting remote calls.

Closing as this is working as designed. The distinction between local and remote-only plugins allows us to provide high-quality adversarial testing while giving users control over data residency for plugins that support local generation. Please feel free to reach out or comment if you have any more questions.