databricks-labs-pytester 0.7.4
Python Testing for Databricks
Requires Python: >=3.10
Keywords: Databricks, pytest

# Python Testing for Databricks

[![python](https://img.shields.io/badge/python-3.10,%203.11,%203.12-green)](https://github.com/databrickslabs/pytester/actions/workflows/push.yml)
[![lines of code](https://tokei.rs/b1/github/databrickslabs/pytester)]([https://github.com/databrickslabs/pytester](https://github.com/databrickslabs/pytester))


<!-- TOC -->
* [Python Testing for Databricks](#python-testing-for-databricks)
  * [Installation](#installation)
  * [Ecosystem](#ecosystem)
  * [PyTest Fixtures](#pytest-fixtures)
    * [Logging](#logging)
    * [`debug_env_name` fixture](#debug_env_name-fixture)
    * [`debug_env` fixture](#debug_env-fixture)
    * [`env_or_skip` fixture](#env_or_skip-fixture)
    * [`ws` fixture](#ws-fixture)
    * [`make_run_as` fixture](#make_run_as-fixture)
    * [`acc` fixture](#acc-fixture)
    * [`spark` fixture](#spark-fixture)
    * [`sql_backend` fixture](#sql_backend-fixture)
    * [`sql_exec` fixture](#sql_exec-fixture)
    * [`sql_fetch_all` fixture](#sql_fetch_all-fixture)
    * [`make_random` fixture](#make_random-fixture)
    * [`make_instance_pool` fixture](#make_instance_pool-fixture)
    * [`make_instance_pool_permissions` fixture](#make_instance_pool_permissions-fixture)
    * [`make_job` fixture](#make_job-fixture)
    * [`make_job_permissions` fixture](#make_job_permissions-fixture)
    * [`make_cluster` fixture](#make_cluster-fixture)
    * [`make_cluster_permissions` fixture](#make_cluster_permissions-fixture)
    * [`make_cluster_policy` fixture](#make_cluster_policy-fixture)
    * [`make_cluster_policy_permissions` fixture](#make_cluster_policy_permissions-fixture)
    * [`make_pipeline` fixture](#make_pipeline-fixture)
    * [`make_warehouse` fixture](#make_warehouse-fixture)
    * [`make_group` fixture](#make_group-fixture)
    * [`make_acc_group` fixture](#make_acc_group-fixture)
    * [`make_user` fixture](#make_user-fixture)
    * [`make_pipeline_permissions` fixture](#make_pipeline_permissions-fixture)
    * [`make_notebook` fixture](#make_notebook-fixture)
    * [`make_notebook_permissions` fixture](#make_notebook_permissions-fixture)
    * [`make_workspace_file` fixture](#make_workspace_file-fixture)
    * [`make_directory` fixture](#make_directory-fixture)
    * [`make_directory_permissions` fixture](#make_directory_permissions-fixture)
    * [`make_repo` fixture](#make_repo-fixture)
    * [`make_repo_permissions` fixture](#make_repo_permissions-fixture)
    * [`make_workspace_file_permissions` fixture](#make_workspace_file_permissions-fixture)
    * [`make_workspace_file_path_permissions` fixture](#make_workspace_file_path_permissions-fixture)
    * [`make_secret_scope` fixture](#make_secret_scope-fixture)
    * [`make_secret_scope_acl` fixture](#make_secret_scope_acl-fixture)
    * [`make_authorization_permissions` fixture](#make_authorization_permissions-fixture)
    * [`make_udf` fixture](#make_udf-fixture)
    * [`make_catalog` fixture](#make_catalog-fixture)
    * [`make_schema` fixture](#make_schema-fixture)
    * [`make_table` fixture](#make_table-fixture)
    * [`make_storage_credential` fixture](#make_storage_credential-fixture)
    * [`make_volume` fixture](#make_volume-fixture)
    * [`product_info` fixture](#product_info-fixture)
    * [`make_model` fixture](#make_model-fixture)
    * [`make_experiment` fixture](#make_experiment-fixture)
    * [`make_experiment_permissions` fixture](#make_experiment_permissions-fixture)
    * [`make_warehouse_permissions` fixture](#make_warehouse_permissions-fixture)
    * [`make_lakeview_dashboard_permissions` fixture](#make_lakeview_dashboard_permissions-fixture)
    * [`log_workspace_link` fixture](#log_workspace_link-fixture)
    * [`log_account_link` fixture](#log_account_link-fixture)
    * [`make_dashboard_permissions` fixture](#make_dashboard_permissions-fixture)
    * [`make_alert_permissions` fixture](#make_alert_permissions-fixture)
    * [`make_query` fixture](#make_query-fixture)
    * [`make_query_permissions` fixture](#make_query_permissions-fixture)
    * [`make_registered_model_permissions` fixture](#make_registered_model_permissions-fixture)
    * [`make_serving_endpoint` fixture](#make_serving_endpoint-fixture)
    * [`make_serving_endpoint_permissions` fixture](#make_serving_endpoint_permissions-fixture)
    * [`make_feature_table` fixture](#make_feature_table-fixture)
    * [`make_feature_table_permissions` fixture](#make_feature_table_permissions-fixture)
    * [`watchdog_remove_after` fixture](#watchdog_remove_after-fixture)
    * [`watchdog_purge_suffix` fixture](#watchdog_purge_suffix-fixture)
    * [`is_in_debug` fixture](#is_in_debug-fixture)
* [Project Support](#project-support)
<!-- TOC -->

## Installation

Add a `databricks-labs-pytester` dependency to your `pyproject.toml` file (or legacy `requirements.txt` file). You can
also install it directly from the command line:

```shell
pip install databricks-labs-pytester
```

If you use `hatch` as a build system, make sure to add `databricks-labs-pytester` as
a test-time dependency and not as a compile-time dependency, otherwise your wheels will
transitively depend on `pytest`, which is not usually something you need.

```toml
[project]
name = "name-of-your-project"
# ...
dependencies = [
  "databricks-sdk~=0.30",
  # ... dependencies required for your code to execute
]

[tool.hatch.envs.default]
dependencies = [
  # ... dependencies required to test/validate/format your code:
    "black~=24.3.0",
    "coverage[toml]~=7.4.4",
    "mypy~=1.9.0",
    "pylint~=3.2.2",
    "pylint-pytest==2.0.0a0",
    "databricks-labs-pylint~=0.4.0",
    "databricks-labs-pytester~=0.2", # <= this library
    "pytest~=8.3.3",
    "pytest-cov~=4.1.0",
    "pytest-mock~=3.14.0",
    "pytest-timeout~=2.3.1",
    "pytest-xdist~=3.5.0",
    "python-lsp-server>=1.9.0",
    "ruff~=0.3.4",
    "types-PyYAML~=6.0.12",
    "types-requests~=2.31.0",
]
```

[[back to top](#python-testing-for-databricks)]

## Ecosystem

Built on top of [Databricks SDK for Python](https://github.com/databricks/databricks-sdk-py), this library is part of the Databricks Labs Python ecosystem, which includes the following projects:
* [PyLint Plugin for Databricks](https://github.com/databrickslabs/pylint-plugin) for static code analysis and early bug detection.
* [Blueprint](https://github.com/databrickslabs/blueprint) for
  [Python-native pathlib.Path-like interfaces](https://github.com/databrickslabs/blueprint#python-native-pathlibpath-like-interfaces),
  [Managing Python App installations within Databricks Workspaces](https://github.com/databrickslabs/blueprint#application-and-installation-state),
  [Application Migrations](https://github.com/databrickslabs/blueprint#application-state-migrations), and
  [Building Wheels](https://github.com/databrickslabs/blueprint#building-wheels).
* [LSQL](https://github.com/databrickslabs/lsql) for lightweight SQL handling and dashboards-as-code.
* [UCX](https://github.com/databrickslabs/ucx) for automated migrations into Unity Catalog and LSP plugin for static code analysis for UC compatibility.

See [this video](https://www.youtube.com/watch?v=CNypO79IATc) for a quick overview of the Databricks Labs Python ecosystem.

[[back to top](#python-testing-for-databricks)]

## PyTest Fixtures

[PyTest Fixtures](https://docs.pytest.org/en/latest/explanation/fixtures.html) are a powerful way to manage test setup and teardown in Python. This library provides
a set of fixtures to help you write integration tests for Databricks. These fixtures were incubated
within the [Unity Catalog Automated Migrations project](https://github.com/databrickslabs/ucx/blame/df7f1d7647251fb8f0f23c56a548b99092484a7c/src/databricks/labs/ucx/mixins/fixtures.py)
for more than a year and are now available for other projects to simplify integration testing with Databricks.

[[back to top](#python-testing-for-databricks)]

### Logging

This library is built on years of debugging integration tests for Databricks and its ecosystem.

That's why it comes with a built-in logger that traces creation and deletion of dummy entities through links in
the Databricks Workspace UI. If you run the following code:

```python
def test_new_user(make_user, ws):
    new_user = make_user()
    home_dir = ws.workspace.get_status(f"/Users/{new_user.user_name}")
    assert home_dir.object_type == ObjectType.DIRECTORY
```

You will see the following output, where the first line is clickable and will take you to the user's profile in the Databricks Workspace UI:

```text
12:30:53  INFO [d.l.p.fixtures.baseline] Created dummy-xwuq-...@example.com: https://.....azuredatabricks.net/#settings/workspace/identity-and-access/users/735...
12:30:53 DEBUG [d.l.p.fixtures.baseline] added workspace user fixture: User(active=True, display_name='dummy-xwuq-...@example.com', ...)
12:30:58 DEBUG [d.l.p.fixtures.baseline] clearing 1 workspace user fixtures
12:30:58 DEBUG [d.l.p.fixtures.baseline] removing workspace user fixture: User(active=True, display_name='dummy-xwuq-...@example.com', ...)
```

You may need to add the following to your `conftest.py` file to enable this:

```python
import logging

from databricks.labs.blueprint.logger import install_logger

install_logger()

logging.getLogger('databricks.labs.pytester').setLevel(logging.DEBUG)
```

[[back to top](#python-testing-for-databricks)]

<!-- FIXTURES -->
### `debug_env_name` fixture
Specify the name of the debug environment. By default, it is set to `.env`,
which will try to find a [file named `.env`](https://www.dotenv.org/docs/security/env)
in any of the parent directories of the current working directory and load
the environment variables from it via the [`debug_env` fixture](#debug_env-fixture).

Alternatively, if you are concerned of the
[risk of `.env` files getting checked into version control](https://thehackernews.com/2024/08/attackers-exploit-public-env-files-to.html),
we recommend using the `~/.databricks/debug-env.json` file to store different sets of environment variables.
The file cannot be checked into version control by design, because it is stored in the user's home directory.

This file is used for local debugging and integration tests in IDEs like PyCharm, VSCode, and IntelliJ IDEA
while developing Databricks Platform Automation Stack, which includes Databricks SDKs for Python, Go, and Java,
as well as Databricks Terraform Provider and Databricks CLI. This file enables multi-environment and multi-cloud
testing with a single set of integration tests.

The file is typically structured as follows:

```shell
$ cat ~/.databricks/debug-env.json
{
   "ws": {
     "CLOUD_ENV": "azure",
     "DATABRICKS_HOST": "....azuredatabricks.net",
     "DATABRICKS_CLUSTER_ID": "0708-200540-...",
     "DATABRICKS_WAREHOUSE_ID": "33aef...",
        ...
   },
   "acc": {
     "CLOUD_ENV": "aws",
     "DATABRICKS_HOST": "accounts.cloud.databricks.net",
     "DATABRICKS_CLIENT_ID": "....",
     "DATABRICKS_CLIENT_SECRET": "....",
     ...
   }
}
```

And you can load it in your `conftest.py` file as follows:

```python
@pytest.fixture
def debug_env_name():
    return "ws"
```

This will load the `ws` environment from the `~/.databricks/debug-env.json` file.

If any of the environment variables are not found, [`env_or_skip` fixture](#env_or_skip-fixture)
will gracefully skip the execution of tests.

See also [`debug_env`](#debug_env-fixture).


[[back to top](#python-testing-for-databricks)]

### `debug_env` fixture
Loads environment variables specified in [`debug_env_name` fixture](#debug_env_name-fixture) from a file
for local debugging in IDEs, otherwise allowing the tests to run with the default environment variables
specified in the CI/CD pipeline.

See also [`acc`](#acc-fixture), [`env_or_skip`](#env_or_skip-fixture), [`ws`](#ws-fixture), [`debug_env_name`](#debug_env_name-fixture), [`is_in_debug`](#is_in_debug-fixture).


[[back to top](#python-testing-for-databricks)]

### `env_or_skip` fixture
Fixture to get environment variables or skip tests.

It is extremely useful to skip tests if the required environment variables are not set.

In the following example, `test_something` would only run if the environment variable
`SOME_EXTERNAL_SERVICE_TOKEN` is set:

```python
def test_something(env_or_skip):
    token = env_or_skip("SOME_EXTERNAL_SERVICE_TOKEN")
    assert token is not None
```

See also [`acc`](#acc-fixture), [`make_run_as`](#make_run_as-fixture), [`make_udf`](#make_udf-fixture), [`sql_backend`](#sql_backend-fixture), [`debug_env`](#debug_env-fixture), [`is_in_debug`](#is_in_debug-fixture).


[[back to top](#python-testing-for-databricks)]

### `ws` fixture
Create and provide a Databricks WorkspaceClient object.

This fixture initializes a Databricks WorkspaceClient object, which can be used
to interact with the Databricks workspace API. The created instance of WorkspaceClient
is shared across all test functions within the test session.

See [detailed documentation](https://databricks-sdk-py.readthedocs.io/en/latest/authentication.html) for the list
of environment variables that can be used to authenticate the WorkspaceClient.

In your test functions, include this fixture as an argument to use the WorkspaceClient:

```python
def test_workspace_operations(ws):
    clusters = ws.clusters.list_clusters()
    assert len(clusters) >= 0
```

See also [`log_workspace_link`](#log_workspace_link-fixture), [`make_alert_permissions`](#make_alert_permissions-fixture), [`make_authorization_permissions`](#make_authorization_permissions-fixture), [`make_catalog`](#make_catalog-fixture), [`make_cluster`](#make_cluster-fixture), [`make_cluster_permissions`](#make_cluster_permissions-fixture), [`make_cluster_policy`](#make_cluster_policy-fixture), [`make_cluster_policy_permissions`](#make_cluster_policy_permissions-fixture), [`make_dashboard_permissions`](#make_dashboard_permissions-fixture), [`make_directory`](#make_directory-fixture), [`make_directory_permissions`](#make_directory_permissions-fixture), [`make_experiment`](#make_experiment-fixture), [`make_experiment_permissions`](#make_experiment_permissions-fixture), [`make_feature_table`](#make_feature_table-fixture), [`make_feature_table_permissions`](#make_feature_table_permissions-fixture), [`make_group`](#make_group-fixture), [`make_instance_pool`](#make_instance_pool-fixture), [`make_instance_pool_permissions`](#make_instance_pool_permissions-fixture), [`make_job`](#make_job-fixture), [`make_job_permissions`](#make_job_permissions-fixture), [`make_lakeview_dashboard_permissions`](#make_lakeview_dashboard_permissions-fixture), [`make_model`](#make_model-fixture), [`make_notebook`](#make_notebook-fixture), [`make_notebook_permissions`](#make_notebook_permissions-fixture), [`make_pipeline`](#make_pipeline-fixture), [`make_pipeline_permissions`](#make_pipeline_permissions-fixture), [`make_query`](#make_query-fixture), [`make_query_permissions`](#make_query_permissions-fixture), [`make_registered_model_permissions`](#make_registered_model_permissions-fixture), [`make_repo`](#make_repo-fixture), [`make_repo_permissions`](#make_repo_permissions-fixture), [`make_run_as`](#make_run_as-fixture), [`make_secret_scope`](#make_secret_scope-fixture), [`make_secret_scope_acl`](#make_secret_scope_acl-fixture), [`make_serving_endpoint`](#make_serving_endpoint-fixture), [`make_serving_endpoint_permissions`](#make_serving_endpoint_permissions-fixture), [`make_storage_credential`](#make_storage_credential-fixture), [`make_udf`](#make_udf-fixture), [`make_user`](#make_user-fixture), [`make_volume`](#make_volume-fixture), [`make_warehouse`](#make_warehouse-fixture), [`make_warehouse_permissions`](#make_warehouse_permissions-fixture), [`make_workspace_file`](#make_workspace_file-fixture), [`make_workspace_file_path_permissions`](#make_workspace_file_path_permissions-fixture), [`make_workspace_file_permissions`](#make_workspace_file_permissions-fixture), [`spark`](#spark-fixture), [`sql_backend`](#sql_backend-fixture), [`debug_env`](#debug_env-fixture), [`product_info`](#product_info-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_run_as` fixture
This fixture provides a function to create an account service principal via [`acc` fixture](#acc-fixture) and
assign it to a workspace. The service principal is removed after the test is complete. The service principal is
created with a random display name and assigned to the workspace with the default permissions.

Use the `account_groups` argument to assign the service principal to account groups, which have the required
permissions to perform a specific action.

Example:

```python
def test_run_as_lower_privilege_user(make_run_as, ws):
    run_as = make_run_as(account_groups=['account.group.name'])
    through_query = next(run_as.sql_fetch_all("SELECT CURRENT_USER() AS my_name"))
    me = ws.current_user.me()
    assert me.user_name != through_query.my_name
```

Returned object has the following properties:
* `ws`: Workspace client that is authenticated as the ephemeral service principal.
* `sql_backend`: SQL backend that is authenticated as the ephemeral service principal.
* `sql_exec`: Function to execute a SQL statement on behalf of the ephemeral service principal.
* `sql_fetch_all`: Function to fetch all rows from a SQL statement on behalf of the ephemeral service principal.
* `display_name`: Display name of the ephemeral service principal.
* `application_id`: Application ID of the ephemeral service principal.
* if you want to have other fixtures available in the context of the ephemeral service principal, you can override
  the [`ws` fixture](#ws-fixture) on the file level, which would make all workspace fixtures provided by this
  plugin to run as lower privilege ephemeral service principal. You cannot combine it with the account-admin-level
  principal you're using to create the ephemeral principal.

Example:

```python
from pytest import fixture

@fixture
def ws(make_run_as):
    run_as = make_run_as(account_groups=['account.group.used.for.all.tests.in.this.file'])
    return run_as.ws

def test_creating_notebook_on_behalf_of_ephemeral_principal(make_notebook):
    notebook = make_notebook()
    assert notebook.exists()
```

This fixture currently doesn't work with Databricks Metadata Service authentication on Azure Databricks.

See also [`acc`](#acc-fixture), [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`env_or_skip`](#env_or_skip-fixture), [`log_account_link`](#log_account_link-fixture), [`is_in_debug`](#is_in_debug-fixture).


[[back to top](#python-testing-for-databricks)]

### `acc` fixture
Create and provide a Databricks AccountClient object.

This fixture initializes a Databricks AccountClient object, which can be used
to interact with the Databricks account API. The created instance of AccountClient
is shared across all test functions within the test session.

Requires `DATABRICKS_ACCOUNT_ID` environment variable to be set. If `DATABRICKS_HOST`
points to a workspace host, the fixture would automatically determine the account host
from it.

See [detailed documentation](https://databricks-sdk-py.readthedocs.io/en/latest/authentication.html) for the list
of environment variables that can be used to authenticate the AccountClient.

In your test functions, include this fixture as an argument to use the AccountClient:

```python
def test_listing_workspaces(acc):
    workspaces = acc.workspaces.list()
    assert len(workspaces) >= 1
```

See also [`log_account_link`](#log_account_link-fixture), [`make_acc_group`](#make_acc_group-fixture), [`make_run_as`](#make_run_as-fixture), [`debug_env`](#debug_env-fixture), [`product_info`](#product_info-fixture), [`env_or_skip`](#env_or_skip-fixture).


[[back to top](#python-testing-for-databricks)]

### `spark` fixture
Get Databricks Connect Spark session. Requires `databricks-connect` package to be installed.

To enable serverless set the local environment variable `DATABRICKS_SERVERLESS_COMPUTE_ID` to `"auto"`.
If this environment variable is set, Databricks Connect ignores the cluster_id.
If `DATABRICKS_SERVERLESS_COMPUTE_ID` is set to a specific serverless cluster ID, that cluster will be used instead.
However, this is not recommended, as serverless clusters are ephemeral by design.
See more details [here](https://docs.databricks.com/en/dev-tools/databricks-connect/cluster-config.html#configure-a-connection-to-serverless-compute).

Usage:
```python
def test_databricks_connect(spark):
    rows = spark.sql("SELECT 1").collect()
    assert rows[0][0] == 1
```

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `sql_backend` fixture
Create and provide a SQL backend for executing statements.

Requires the environment variable `DATABRICKS_WAREHOUSE_ID` to be set.

See also [`make_schema`](#make_schema-fixture), [`make_table`](#make_table-fixture), [`make_udf`](#make_udf-fixture), [`sql_exec`](#sql_exec-fixture), [`sql_fetch_all`](#sql_fetch_all-fixture), [`ws`](#ws-fixture), [`env_or_skip`](#env_or_skip-fixture).


[[back to top](#python-testing-for-databricks)]

### `sql_exec` fixture
Execute SQL statement and don't return any results.

See also [`sql_backend`](#sql_backend-fixture).


[[back to top](#python-testing-for-databricks)]

### `sql_fetch_all` fixture
Fetch all rows from a SQL statement.

See also [`sql_backend`](#sql_backend-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_random` fixture
Fixture to generate random strings.

This fixture provides a function to generate random strings of a specified length.
The generated strings are created using a character set consisting of uppercase letters,
lowercase letters, and digits.

To generate a random string with default length of 16 characters:

```python
random_string = make_random()
assert len(random_string) == 16
```

To generate a random string with a specified length:

```python
random_string = make_random(k=8)
assert len(random_string) == 8
```

See also [`make_acc_group`](#make_acc_group-fixture), [`make_catalog`](#make_catalog-fixture), [`make_cluster`](#make_cluster-fixture), [`make_cluster_policy`](#make_cluster_policy-fixture), [`make_directory`](#make_directory-fixture), [`make_experiment`](#make_experiment-fixture), [`make_feature_table`](#make_feature_table-fixture), [`make_group`](#make_group-fixture), [`make_instance_pool`](#make_instance_pool-fixture), [`make_job`](#make_job-fixture), [`make_model`](#make_model-fixture), [`make_notebook`](#make_notebook-fixture), [`make_pipeline`](#make_pipeline-fixture), [`make_query`](#make_query-fixture), [`make_repo`](#make_repo-fixture), [`make_run_as`](#make_run_as-fixture), [`make_schema`](#make_schema-fixture), [`make_secret_scope`](#make_secret_scope-fixture), [`make_serving_endpoint`](#make_serving_endpoint-fixture), [`make_table`](#make_table-fixture), [`make_udf`](#make_udf-fixture), [`make_user`](#make_user-fixture), [`make_volume`](#make_volume-fixture), [`make_warehouse`](#make_warehouse-fixture), [`make_workspace_file`](#make_workspace_file-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_instance_pool` fixture
Create a Databricks instance pool and clean it up after the test. Returns a function to create instance pools.
Use `instance_pool_id` attribute from the returned object to get an ID of the pool.

Keyword Arguments:
* `instance_pool_name` (str, optional): The name of the instance pool. If not provided, a random name will be generated.
* `node_type_id` (str, optional): The node type ID of the instance pool. If not provided, a node type with local disk and 16GB memory will be used.
* other arguments are passed to `WorkspaceClient.instance_pools.create` method.

Usage:
```python
def test_instance_pool(make_instance_pool):
    logger.info(f"created {make_instance_pool()}")
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_remove_after`](#watchdog_remove_after-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_instance_pool_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_job` fixture
Create a Databricks job and clean it up after the test. Returns a function to create jobs, that returns
a [`Job`](https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/jobs.html#databricks.sdk.service.jobs.Job) instance.

Keyword Arguments:
* `name` (str, optional): The name of the job. If not provided, a random name will be generated.
* `path` (str, optional): The path to the notebook or file used in the job. If not provided, a random notebook or file will be created
* [DEPRECATED: Use `path` instead] `notebook_path` (str, optional): The path to the notebook. If not provided, a random notebook will be created.
* `content` (str | bytes, optional): The content of the notebook or file used in the job. If not provided, default content of `make_notebook` will be used.
* `task_type` (type[NotebookTask] | type[SparkPythonTask], optional): The type of task. If not provides, `type[NotebookTask]` will be used.
* `instance_pool_id` (str, optional): The instance pool id to add to the job cluster. If not provided, no instance pool will be used.
* `spark_conf` (dict, optional): The Spark configuration of the job. If not provided, Spark configuration is not explicitly set.
* `libraries` (list, optional): The list of libraries to install on the job.
* `tags` (list[str], optional): A list of job tags. If not provided, no additional tags will be set on the job.
* `tasks` (list[Task], optional): A list of job tags. If not provided, a single task with a notebook task will be
   created, along with a disposable notebook. Latest Spark version and a single worker clusters will be used to run
   this ephemeral job.

Usage:
```python
def test_job(make_job):
    logger.info(f"created {make_job()}")
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`make_notebook`](#make_notebook-fixture), [`make_workspace_file`](#make_workspace_file-fixture), [`watchdog_remove_after`](#watchdog_remove_after-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_job_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_cluster` fixture
Create a Databricks cluster, waits for it to start, and clean it up after the test.
Returns a function to create clusters. You can get `cluster_id` attribute from the returned object.

Keyword Arguments:
* `single_node` (bool, optional): Whether to create a single-node cluster. Defaults to False.
* `cluster_name` (str, optional): The name of the cluster. If not provided, a random name will be generated.
* `spark_version` (str, optional): The Spark version of the cluster. If not provided, the latest version will be used.
* `autotermination_minutes` (int, optional): The number of minutes before the cluster is automatically terminated. Defaults to 10.

Usage:
```python
def test_cluster(make_cluster):
    logger.info(f"created {make_cluster(single_node=True)}")
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_remove_after`](#watchdog_remove_after-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_cluster_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_cluster_policy` fixture
Create a Databricks cluster policy and clean it up after the test. Returns a function to create cluster policies,
which returns [`CreatePolicyResponse`](https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/compute.html#databricks.sdk.service.compute.CreatePolicyResponse) instance.

Keyword Arguments:
* `name` (str, optional): The name of the cluster policy. If not provided, a random name will be generated.

Usage:
```python
def test_cluster_policy(make_cluster_policy):
    logger.info(f"created {make_cluster_policy()}")
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_purge_suffix`](#watchdog_purge_suffix-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_cluster_policy_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_pipeline` fixture
Create Delta Live Table Pipeline and clean it up after the test. Returns a function to create pipelines.
Results in a [`CreatePipelineResponse`](https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/pipelines.html#databricks.sdk.service.pipelines.CreatePipelineResponse) instance.

Keyword Arguments:
* `name` (str, optional): The name of the pipeline. If not provided, a random name will be generated.
* `libraries` (list, optional): The list of libraries to install on the pipeline. If not provided, a random disposable notebook will be created.
* `clusters` (list, optional): The list of clusters to use for the pipeline. If not provided, a single node cluster will be created with 16GB memory and local disk.

Usage:
```python
def test_pipeline(make_pipeline, make_pipeline_permissions, make_group):
    group = make_group()
    pipeline = make_pipeline()
    make_pipeline_permissions(
        object_id=pipeline.pipeline_id,
        permission_level=PermissionLevel.CAN_MANAGE,
        group_name=group.display_name,
    )
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`make_notebook`](#make_notebook-fixture), [`watchdog_remove_after`](#watchdog_remove_after-fixture), [`watchdog_purge_suffix`](#watchdog_purge_suffix-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_warehouse` fixture
Create a Databricks warehouse and clean it up after the test. Returns a function to create warehouses.

Keyword Arguments:
* `warehouse_name` (str, optional): The name of the warehouse. If not provided, a random name will be generated.
* `warehouse_type` (CreateWarehouseRequestWarehouseType, optional): The type of the warehouse. Defaults to `PRO`.
* `cluster_size` (str, optional): The size of the cluster. Defaults to `2X-Small`.

Usage:
```python
def test_warehouse_has_remove_after_tag(ws, make_warehouse):
    new_warehouse = make_warehouse()
    created_warehouse = ws.warehouses.get(new_warehouse.response.id)
    warehouse_tags = created_warehouse.tags.as_dict()
    assert warehouse_tags["custom_tags"][0]["key"] == "RemoveAfter"
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_remove_after`](#watchdog_remove_after-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_group` fixture
This fixture provides a function to manage Databricks workspace groups. Groups can be created with specified
members and roles, and they will be deleted after the test is complete. Deals with eventual consistency issues by
retrying the creation process for 30 seconds and then waiting for up to 3 minutes for the group to be provisioned.
Returns an instance of [`Group`](https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/iam.html#databricks.sdk.service.iam.Group).

Keyword arguments:
* `members` (list of strings): A list of user IDs to add to the group.
* `roles` (list of strings): A list of roles to assign to the group.
* `display_name` (str): The display name of the group.
* `entitlements` (list of strings): A list of entitlements to assign to the group.

The following example creates a group with a single member and independently verifies that the group was created:

```python
def test_new_group(make_group, make_user, ws):
    user = make_user()
    group = make_group(members=[user.id])
    loaded = ws.groups.get(group.id)
    assert group.display_name == loaded.display_name
    assert group.members == loaded.members
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_purge_suffix`](#watchdog_purge_suffix-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_acc_group` fixture
This fixture provides a function to manage Databricks account groups. Groups can be created with
specified members and roles, and they will be deleted after the test is complete.

Has the same arguments and behavior as [`make_group` fixture](#make_group-fixture) but uses the account
client instead of the workspace client.

Example usage:
```python
def test_new_account_group(make_acc_group, acc):
    group = make_acc_group()
    loaded = acc.groups.get(group.id)
    assert group.display_name == loaded.display_name
```

See also [`acc`](#acc-fixture), [`make_random`](#make_random-fixture), [`watchdog_purge_suffix`](#watchdog_purge_suffix-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_user` fixture
This fixture returns a function that creates a Databricks workspace user
and removes it after the test is complete. In case of random naming conflicts,
the fixture will retry the creation process for 30 seconds. Returns an instance
of [`User`](https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/iam.html#databricks.sdk.service.iam.User). Usage:

```python
def test_new_user(make_user, ws):
    new_user = make_user()
    home_dir = ws.workspace.get_status(f"/Users/{new_user.user_name}")
    assert home_dir.object_type == ObjectType.DIRECTORY
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_purge_suffix`](#watchdog_purge_suffix-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_pipeline_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_notebook` fixture
Returns a function to create Databricks Notebooks and clean them up after the test.
The function returns [`os.PathLike` object](https://github.com/databrickslabs/blueprint?tab=readme-ov-file#python-native-pathlibpath-like-interfaces).

Keyword arguments:
* `path` (str, optional): The path of the notebook. Defaults to `dummy-*` notebook in current user's home folder.
* `content` (str | bytes | io.BinaryIO, optional): The content of the notebook. Defaults to `print(1)` for Python and `SELECT 1` for SQL.
* `language` ([`Language`](https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/workspace.html#databricks.sdk.service.workspace.Language), optional): The language of the notebook. Defaults to `Language.PYTHON`.
* `encoding` (`str`, optional): The file encoding. Defaults to `sys.getdefaultencoding()`.
* `format` ([`ImportFormat`](https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/workspace.html#databricks.sdk.service.workspace.ImportFormat), optional): The format of the notebook. Defaults to `ImportFormat.SOURCE`.
* `overwrite` (bool, optional): Whether to overwrite the notebook if it already exists. Defaults to `False`.

This example creates a notebook and verifies that `print(1)` is in the content:
```python
def test_creates_some_notebook(make_notebook):
    notebook = make_notebook()
    assert "print(1)" in notebook.read_text()
```

See also [`make_job`](#make_job-fixture), [`make_pipeline`](#make_pipeline-fixture), [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_purge_suffix`](#watchdog_purge_suffix-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_notebook_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_workspace_file` fixture
Returns a function to create Databricks workspace file and clean up after the test.
The function returns [`os.PathLike` object](https://github.com/databrickslabs/blueprint?tab=readme-ov-file#python-native-pathlibpath-like-interfaces).

Keyword arguments:
* `path` (str, optional): The path of the file. Defaults to `dummy-*` notebook in current user's home folder.
* `content` (str | bytes, optional): The content of the file. Defaults to `print(1)` for Python and `SELECT 1` for SQL.
* `language` ([`Language`](https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/workspace.html#databricks.sdk.service.workspace.Language), optional): The language of the notebook. Defaults to `Language.PYTHON`.
* `encoding` (`str`, optional): The file encoding. Defaults to `sys.getdefaultencoding()`.

This example creates a notebook and verifies that the workspace path is an existing file with contents `print(1)`:
```python
def test_create_file(make_workspace_file):
    workspace_file = make_workspace_file()
    assert workspace_file.is_file()
    assert "print(1)" in workspace_file.read_text()
```

TODO:
Merge functionality with `make_notebook` if `WorkspacePath` supports creating notebooks.

See also [`make_job`](#make_job-fixture), [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_purge_suffix`](#watchdog_purge_suffix-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_directory` fixture
Returns a function to create Databricks Workspace Folders and clean them up after the test.
The function returns [`os.PathLike` object](https://github.com/databrickslabs/blueprint?tab=readme-ov-file#python-native-pathlibpath-like-interfaces).

Keyword arguments:
* `path` (str, optional): The path of the notebook. Defaults to `dummy-*` folder in current user's home folder.

This example creates a folder and verifies that it contains a notebook:
```python
def test_creates_some_folder_with_a_notebook(make_directory, make_notebook):
    folder = make_directory()
    notebook = make_notebook(path=folder / 'foo.py')
    files = [_.name for _ in folder.iterdir()]
    assert ['foo.py'] == files
    assert notebook.parent == folder
```

See also [`make_experiment`](#make_experiment-fixture), [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_purge_suffix`](#watchdog_purge_suffix-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_directory_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_repo` fixture
Returns a function to create Databricks Repos and clean them up after the test.
The function returns a [`RepoInfo`](https://databricks-sdk-py.readthedocs.io/en/latest/dbdataclasses/workspace.html#databricks.sdk.service.workspace.RepoInfo) object.

Keyword arguments:
* `url` (str, optional): The URL of the repository.
* `provider` (str, optional): The provider of the repository.
* `path` (str, optional): The path of the repository. Defaults to `/Repos/{current_user}/sdk-{random}-{purge_suffix}`.

Usage:
```python
def test_repo(make_repo):
    logger.info(f"created {make_repo()}")
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture), [`watchdog_purge_suffix`](#watchdog_purge_suffix-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_repo_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_workspace_file_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_workspace_file_path_permissions` fixture
_No description yet._

See also [`ws`](#ws-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_secret_scope` fixture
This fixture provides a function to create secret scopes. The created secret scope will be
deleted after the test is complete. Returns the name of the secret scope.

To create a secret scope and use it within a test function:

```python
def test_secret_scope_creation(make_secret_scope):
    secret_scope_name = make_secret_scope()
    assert secret_scope_name.startswith("dummy-")
```

See also [`ws`](#ws-fixture), [`make_random`](#make_random-fixture).


[[back to top](#python-testing-for-databricks)]

### `make_secret_scope_acl` fixture
This fixture provides a function to manage access control lists (ACLs) for secret scopes.
ACLs define permissions for principa