Repository: promptfoo/promptfoo
Issue: Documentation claims "100% local" but promptfoo sends data to remote API by default!
Root cause: Documentation Deficiency

## Description

The README.md states that promptfoo is "100% local" and "your prompts never leave your machine", but this is not accurate. By default, certain features send data to `https://api.promptfoo.app/api/v1/task`.

**Current claim in README.md (line 58):**
> 🔒 **Private**: Runs 100% locally - your prompts never leave your machine

## What actually happens

### Remote API usage

When using certain features (adversarial test generation, simulated user interactions), promptfoo sends the following data to `api.promptfoo.app` by default:

**Data sent to remote API:**
- Instructions/prompts
- Conversation history (messages)
- User email (from `getUserEmail()`)
- Promptfoo version
- System purpose description

**Relevant code:**
- [`src/providers/promptfoo.ts`](https://github.com/promptfoo/promptfoo/blob/main/src/providers/promptfoo.ts#L209-L254) - `PromptfooSimulatedUserProvider`
- [`src/providers/promptfoo.ts`](https://github.com/promptfoo/promptfoo/blob/main/src/providers/promptfoo.ts#L28-L104) - `PromptfooHarmfulCompletionProvider`
- [`src/redteam/remoteGeneration.ts`](https://github.com/promptfoo/promptfoo/blob/main/src/redteam/remoteGeneration.ts#L6-L19) - Remote generation URL logic

### When remote generation is used

According to [`remoteGeneration.ts:68-81`](https://github.com/promptfoo/promptfoo/blob/main/src/redteam/remoteGeneration.ts#L68-L81):
- If logged into promptfoo cloud
- OR if `OPENAI_API_KEY` is not set
- OR if `--remote` flag is used

### How to disable

Users can disable remote generation with:
```bash
export PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true
```

However, **this flag is not respected by `SimulatedUser`** - see [`src/providers/simulatedUser.ts:116`](https://github.com/promptfoo/promptfoo/blob/main/src/providers/simulatedUser.ts#L116) which always instantiates `PromptfooSimulatedUserProvider` regardless of the environment variable.

## Issues

1. **Marketing claim is misleading**: The README prominently claims 100% local operation, but sends data to external servers by default
2. **Incomplete documentation**: The FAQ mentions this indirectly but doesn't clearly state which features use remote generation
3. **Bug in SimulatedUser**: The `PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION` flag doesn't work for `SimulatedUser` provider
4. **Unused config option**: `SimulatedUser` accepts a `userProvider` config option but never uses it (see TODO comment at line 32)
5. **No visibility in debug logs**: Even with `LOG_LEVEL=debug`, the remote API calls are not clearly logged, making it difficult for users to know their data is being sent externally

**System information:**

version: "0.118.11"
node v24.6.0, MacOS 14.5


Hi @kglowacki, thanks for submitting this issue. To clarify, Promptfoo is 100% local for LLM evaluations, and prompts always remain local. 

Promptfoo does require remote generation for a subset of red team plugins, listed [here](https://www.promptfoo.dev/docs/red-team/plugins/). Any plugin with the globe icon requires remote inference.  It is possible to self-host the inference endpoint as part of our enterprise offering. The "system purpose description" is used by the remote server to generate attacks.  The target's prompt and other target configuration details are never sent.

As you mentioned, you can disable remote generation using `export PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION=true`. This would prevent you from generating attacks for a subset of plugins that rely on remote inference.

We are looking into the issue you flagged for `PromptfooSimulatedUserProvider` and will get back to you. 

wow, that was quick. great job!