Repository: promptfoo/promptfoo
Issue: Default settings to disable data transmission to promptfoo.dev backend
Root cause: Architectural Constraint

**Description:**

I am writing to express my concern and request a change regarding the default settings of promptfoo. Currently, the default configuration sends information to the backend (promptfoo.dev), which raises significant privacy concerns. I believe it would be more transparent and user-friendly if the default settings did not send any information to the backend unless explicitly enabled by the user.

**Current Settings to Disable Data Transmission:**

To prevent data from being sent to the backend, I currently have to manage the following settings:
- `PROMPTFOO_REMOTE_APP_BASE_URL`
- `PROMPTFOO_DISABLE_TELEMETRY`
- `PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION`

Additionally, with each new version, new flags are introduced that need to be disabled, such as the recent `PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION`. This makes it challenging to keep track of all necessary settings to ensure data privacy.

**Proposed Change:**

1. **Default to No Data Transmission:** Make the default settings such that no data is sent to the backend. Users should have to opt-in to enable data transmission.
2. **Informative Prompts:** If certain features are disabled or not working due to these settings (eg. redteam test generation), provide clear and informative prompts to the user, explaining the limitations and offering the option to enable backend usage if desired.
3. **Consolidated Settings Management:** Simplify the management of these settings, possibly through a single configuration file or a user-friendly interface, to make it easier to ensure all necessary flags are set.

**Benefits:**

- **Enhanced Privacy:** Users can be assured that their data is not being transmitted without their explicit consent and avoid situations like https://github.com/promptfoo/promptfoo/issues/1607 or https://github.com/promptfoo/promptfoo/issues/1969
- **Transparency:** Clear communication about what data is being sent and why, fostering trust with users.
- **Ease of Use:** Simplified settings management will reduce the burden on users to keep track of multiple flags across different versions.

Thank you for considering this request. I believe these changes will greatly improve the user experience and trust in promptfoo.

Hi @AlexRRR,

Thank you for sharing these concerns. Privacy and user control are very important to us, and I appreciate the opportunity to address them.

To give some background:

- **`PROMPTFOO_DISABLE_TELEMETRY`**  
  Telemetry helps us improve functionality. The anonymous telemetry we collect enables us to continually refine Promptfoo for everyone. Promptfoo’s default settings are designed to prioritize ease of use. We spend a lot of time analyzing where users drop off and use this information to enhance features. Honestly, we would like to collect even more because we do not have error reporting but we have decided not to due to user privacy concerns. 

- **`PROMPTFOO_REMOTE_APP_BASE_URL`**  
  This setting is used exclusively for sharing, and there’s a confirmation prompt before anything is sent. If you choose to share an eval, it’s done via an anonymous link that is automatically deleted after two weeks.

- **`PROMPTFOO_DISABLE_REDTEAM_REMOTE_GENERATION`**  
  This allows requests to our backend models to generate effective test cases with harmful intents. Generating useful harmful inputs requires our fine-tuned backend models, as public models aren’t as effective. Disabling this by default would significantly reduce the utility of this feature. I’ll consider whether there’s a better way to alert users about this.

I understand that managing multiple flags across versions can be challenging, especially for users who prioritize privacy. If you’re mainly using core eval features, setting `PROMPTFOO_DISABLE_TELEMETRY` should meet your needs. However, if you’re using our red teaming API and want a higher level of privacy, feel free to book a meeting to discuss our self-hosted enterprise offering: [https://cal.com/team/promptfoo/intro](https://cal.com/team/promptfoo/intro).

Based on your feedback, I’ll also consider consolidating these environment variables into a single control, such as `PROMPTFOO_DISABLE_ALL_REMOTE`, to simplify configuration for advanced users.

Additionally, I’ll review our privacy page and terms to ensure they clearly communicate what data is collected and how it’s processed. If you have any further suggestions for privacy features or settings, I’d be happy to connect. Please feel free to reach out if you’d like to set up a time to talk.

Thanks again for your valuable feedback - it’s incredibly helpful as we refine Promptfoo.

Hi @mldangelo 

Thanks for getting back to me with such a detailed response. I really appreciate it!

some thoughts:

- **Single Flag Solution**: The idea of having a single control like PROMPTFOO_DISABLE_ALL_REMOTE (which should be TRUE by default) sounds great and definitely more future-proof. But it would be awesome if we could still choose which data to send. Like, if the flag is off, maybe we could still opt to send some data but not others.
- **Confirmation Dialogs**: It would be super helpful to have confirmation dialogs (like a y/n prompt) when the environment variables aren’t set. Or maybe a banner, similar to the one shown during version upgrades, to keep things transparent.
- **Improving Adoption**: I think these changes could really help with adoption. Teams looking into Promptfoo, especially for red teaming, will be usually pretty security-focused. Having strong privacy controls could prevent them from looking elsewhere. 

I really like Promptfoo and see a lot of potential in it. I want to see it succeed and am happy to provide more feedback if needed.

Thanks again for listening. Looking forward to seeing the improvements!

Hi @AlexRRR, I wanted to follow up on this. I’ve just pushed an update to our privacy policy that explicitly lists all environment variables controlling data transmission. While your suggestions are valuable, they introduce some friction for the default user. Thanks again for helping us improve our transparency and documentation. We’re committed to making privacy controls simple and straightforward for our users. If you’d like to discuss enterprise use cases (including on-prem), feel free to schedule a call.

Hello, the truth is that I was surprised when I was asked for the email when I clicked on the “share” button, thinking that it could be due to an email delivery.
But what has happened is that a PUBLIC link has been created with information that maybe I don't want to be accessible to everyone.

So I have two concerns:
- **Why** am I being asked for my email? Has it been sent to your server?
- The sharing option could be to export it as a zip file and view the result in a local html. And if not, at least give a direct option to delete the report.

Congratulations for the work done as it seems a useful tool, but unfortunately with this lack of privacy and transparency with this, I will have to withdraw its use in my company, if is not fixed and explained clearly.

Thanks in advance