Repository: confident-ai/deepeval
Issue: pytest plugin 'plugins' has generic name, triggers telemetry by default.
Root cause: Architectural Constraint

Deepeval includes a pytest plugin called ["plugins"](https://github.com/confident-ai/deepeval/blob/80c978386c916b37995d156b8f40721b960844ab/pyproject.toml#L15).

(found in entry_points.txt:)
```
[console_scripts]
deepeval = deepeval.cli.main:app

[pytest11]
plugins = deepeval.plugins.plugin
```

If the current python envrionment contains deepeval, the plugin automatically loads when executing any tests with pytest, regardless of whether these tests use deepeval.

The plugin will by default enable telemetry and update-check, and start making outgoing connections, before any pytest fixture or other python code can modify the variables `DEEPEVAL_TELEMETRY_OPT_OUT`, or `DEEPEVAL_UPDATE_WARNING_OPT_OUT`.

This may cause serious problems when executing any tests, even those that make no use of deepeval, in a sensitive or restricted environment.

Determining the root cause of these network requests was quite challenging

## Workaround

The plugin can be disabled by calling e.g. `pytest -p no:plugins`, or adding this switch to the `pytest.ini` config settings.

However, it is very hard to find out that this plugin is related to deepeval at all. It is shown when executing `pytest --trace-config`, but only by name, without any reference to deepeval.


## **Describe the solution you'd like**

I would like a less generic name, such as `deepeval-pytest-plugin` instead of `plugins`, and some mention in the documentation, that a plugin with this name exists, and how it can be disabled.

The plugin should not start accessing the network, checking for updates, or sending telemetry, unless any deepeval features are actually used in a test.

A way to configure the telemetry behaviour from within the tests, without relying on environment variables from the OS, would be great



Similarly:
- The plugin also prints stuff to stdout, even when deepeval isn't part of the test. `]Running teardown with pytest sessionfinish...`, and prints some more details about tests that have been skipped
- It makes the tests much much slower. Running a test goes from 1.2 seconds to 3.6 seconds — what's it doing? Again deepeval is not involved with this test (the test passes even with it uninstalled...)

```
________________________________________________________
Executed in    3.65 secs    fish           external
   usr time    1.46 secs    0.04 millis    1.46 secs
   sys time    0.35 secs    1.19 millis    0.35 secs

```

```
________________________________________________________
Executed in    1.26 secs      fish           external
   usr time  537.05 millis    0.05 millis  537.00 millis
   sys time  133.42 millis    1.46 millis  131.96 millis

```

I'm not some privacy or free-software activist — I support companies making money from their open-source efforts, and even think some light telemetry is OK. 

But this is pretty bad — I'm going to look for alternatives and would not recommend anyone use deepeval. And unfortunately I'll be warier of adding dependencies like this — the gunk somewhat validates the free-software activists...

I ran into the same issue. `DEEPEVAL_TELEMETRY_OPT_OUT` was being ignored in my tests because `load_dotenv` was called after the plugin loaded.

As a workaround, I set `DEEPEVAL_TELEMETRY_OPT_OUT` at the pyproject.toml level:

```toml
[tool.poe.env]
DEEPEVAL_TELEMETRY_OPT_OUT = "YES"
```

It's not merely a "generic" name, it's an undiscoverable anti-name that has cost me hours of happiness.

Thank you everyone for the very detailed reports and for sticking with this, I will be making improvements and clarifying what you can do today.

A quick summary of what we have changed since this issue was opened:

1. **Plugin name & disabling**

   We accepted a PR from @gavmor that renames the pytest entry point from:

   ```toml
   [tool.poetry.plugins."pytest11"]
   plugins = "deepeval.plugins.plugin"
   ```

   to:

   ```toml
   [tool.poetry.plugins."pytest11"]
   deepeval = "deepeval.plugins.plugin"
   ```

   That means:

   * You can disable it explicitly with:

   ```bash
   pytest -p no:deepeval
   ```

   * Or, if you want to completely opt out of all third-party plugins, you can still use:

   ```bash
   PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest ...
   ```

2. **Dotenv autoload vs `.env.test`**

   DeepEval’s root package autoloads `.env`, `.env.{APP_ENV}`, and `.env.local` on import in order to configure DeepEval’s Pydantic settings. Understandably, that was a surprise for users when they ran pytest and found their dotenv files being loaded unexpectedly.

   We do have a dedicated kill switch for this (I will be improving the docs to highlight it more clearly):

   ```bash
   DEEPEVAL_DISABLE_DOTENV=1 pytest ...
   ```

   When `DEEPEVAL_DISABLE_DOTENV=1` is set **before pytest starts**, DeepEval completely skips loading any `.env` files, so your own dotenv strategy (`.env.test`, pytest-dotenv, etc.) is in full control again. I have verified this behavior with a minimal repro where a `.env` defines `foo=bar` and the test asserts that `os.environ.get("foo") is None`, without the flag it fails, with `DEEPEVAL_DISABLE_DOTENV=1` it passes.

   (Note: deepeval dotenv autoload never overwrites existing process environment variables, it only populates keys that are not already set.)

3. **Telemetry and update-check configuration**

   A couple of clarifications:

   * **Update check** is *opt-in* only. We only call out to PyPI when:

     ```bash
     DEEPEVAL_UPDATE_WARNING_OPT_IN=1
     ```

     is set. If you never set this variable, there is no update check or PyPI traffic.

     You can see that here in `deepeval/__init__.py`:

     ```python
     def update_warning_opt_in():
         return os.getenv("DEEPEVAL_UPDATE_WARNING_OPT_IN") == "1"
     ...

     if update_warning_opt_in():
         check_for_update()
     ```

   * **Telemetry** is controlled via:

     ```bash
     DEEPEVAL_TELEMETRY_OPT_OUT=1
     ```

     (with backwards-compatibility for `DEEPEVAL_TELEMETRY_ENABLED`). 

    Because pytest plugins are imported before `conftest.py`, this flag needs to be set at the process level (shell env, CI config, or tool-managed env like `[tool.poe.env]`), not from inside tests. That is exactly the workaround some of you already found, and I’ll be making this more prominent in the docs.

   Putting it together, a "no dotenv, no telemetry" test run looks like:

   ```bash
   DEEPEVAL_DISABLE_DOTENV=1 \
   DEEPEVAL_TELEMETRY_OPT_OUT=1 \
   pytest ...
   ```

4. **Stdout noise and overhead**

   You are also right that `Running teardown with pytest sessionfinish...` and the extra skip-reason printing are not appropriate when DeepEval isn’t actually being used. Our current recommendation if you don’t want any DeepEval behavior in a given repository is to disable the plugin entirely:

   ```bash
   pytest -p no:deepeval
   # or globally:
   PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 pytest -p pytest_cov -p pytest_asyncio ...
   ```

   We are tracking further work to make the plugin quieter and to reduce import-time side effects so that test runs that do not use DeepEval do not pay any noticeable penalty.

---

I really appreciate everyone who reported, debugged, and even sent a PR (special thanks to @gavmor for the plugin rename). In the next release of DeepEval (3.7.3), if you run with `DEEPEVAL_DISABLE_DOTENV` / `DEEPEVAL_TELEMETRY_OPT_OUT` and `-p no:deepeval` where appropriate and still see unexpected behavior, please mention me and I will investigate.