datacheck-cli 2.0.2
CLI-first data validation tool for data engineers. Catch bad data before it breaks pipelines.
License: Apache-2.0
Requires Python: <4.0,>=3.10
Keywords: data-validation, cli, data-engineering, pipeline, ci-cd, data-quality, yaml, testing, csv, parquet, postgres, data-testing

# DataCheck — Data Validation Engine

[![Python 3.10+](https://img.shields.io/badge/python-3.10%2B-blue.svg)](https://www.python.org/downloads/)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)

DataCheck is a data quality validation engine for data engineers. Define validation rules in a YAML config and data sources inline (files) or in a `sources.yaml` (databases, cloud), then automatically validate data across files, databases, and cloud warehouses.

DataCheck provides the `datacheck` Command-Line Interface (CLI) and a Python API, which you can use to validate data, profile quality, and detect schema changes. These operations can be executed locally during development, embedded programmatically within your data pipelines (Airflow, Dagster, Prefect, etc.), or integrated into CI/CD workflows.

### Highlights

- Define validation rules in YAML config and data sources inline (files) or in a `sources.yaml` (databases, cloud)
- Run checks on CSV, Parquet, Delta Lake, Avro, PostgreSQL, MySQL, Snowflake, BigQuery, Redshift, and more
- Use 27+ built-in data quality rules for null checks, numeric ranges, patterns, timestamps, email/phone/URL validation, and cross-column checks
- Profile data quality with automatic scoring, outlier detection, and rule suggestions
- Detect schema evolution with compatibility levels (COMPATIBLE, WARNING, BREAKING)
- Extend with custom rules using the `@custom_rule` plugin decorator

## Installation

```bash
pip install datacheck-cli
```

To install with support for a specific data source, use extras:

```bash
pip install datacheck-cli[postgresql]    # PostgreSQL
pip install datacheck-cli[mysql]         # MySQL
pip install datacheck-cli[snowflake]     # Snowflake
pip install datacheck-cli[bigquery]      # BigQuery
pip install datacheck-cli[redshift]      # Redshift
pip install datacheck-cli[cloud]         # S3, GCS, Azure Blob
pip install datacheck-cli[all]           # All data sources
```

## Quickstart

Use `datacheck config init` to generate a config from a template. Add `--with-sample-data` to also generate a sample CSV file so you can test validation immediately:

```bash
datacheck config init --with-sample-data
datacheck config init --template ecommerce --with-sample-data
```

Or create a `.datacheck.yaml` config file manually with your data source and validation rules:

```yaml
data_source:
  type: csv
  path: ./data/orders.csv

checks:
  - name: id_check
    column: id
    rules:
      not_null: true
      unique: true

  - name: amount_check
    column: amount
    rules:
      not_null: true
      min: 0
      max: 10000

  - name: email_check
    column: email
    rules:
      email_valid: true
```

Run validation:

```bash
datacheck validate
```

## Database and Cloud Sources

For databases and cloud storage, define named sources in a `sources.yaml` file:

```yaml
# sources.yaml
sources:
  production_db:
    type: postgresql
    host: ${DB_HOST}
    port: ${DB_PORT:-5432}
    database: ${DB_NAME}
    user: ${DB_USER}
    password: ${DB_PASSWORD}

  analytics_wh:
    type: snowflake
    account: ${SF_ACCOUNT}
    user: ${SF_USER}
    password: ${SF_PASSWORD}
    warehouse: ${SF_WAREHOUSE:-COMPUTE_WH}
    database: ${SF_DATABASE}
    schema: ${SF_SCHEMA:-PUBLIC}

  s3_data:
    type: s3
    bucket: ${S3_BUCKET}
    path: data/orders.csv
    region: ${AWS_REGION:-us-east-1}
    access_key: ${AWS_ACCESS_KEY_ID}
    secret_key: ${AWS_SECRET_ACCESS_KEY}
```

Reference in your config:

```yaml
# datacheck.yaml
sources_file: ./sources.yaml
source: production_db
table: orders
```

## Profile Data Quality

```bash
datacheck profile                                              # Auto-discover config
datacheck profile data.csv                                     # Direct file path
datacheck profile --source production_db --sources-file sources.yaml  # Named source
datacheck profile --format json -o profile.json                # Export as JSON
```

## Detect Schema Changes

```bash
datacheck schema capture                                        # Auto-discover config
datacheck schema capture data.csv                               # Direct file path
datacheck schema capture --source production_db --sources-file sources.yaml  # Named source
datacheck schema compare                                        # Compare against baseline
```

## Python API

```python
from datacheck import ValidationEngine

engine = ValidationEngine(config_path=".datacheck.yaml")
summary = engine.validate()

print(f"Passed: {summary.passed_rules}/{summary.total_rules}")

for result in summary.get_failed_results():
    print(f"  FAIL: {result.rule_name} on {result.column} ({result.failed_rows} rows)")
```

## Available Rules

| Category | Rules |
|----------|-------|
| Null & Uniqueness | `not_null`, `unique`, `unique_combination` |
| Numeric | `min`, `max`, `mean_between`, `std_dev_less_than`, `percentile_range`, `z_score_outliers`, `distribution_type` |
| String & Pattern | `regex`, `allowed_values`, `length`, `min_length`, `max_length`, `type` |
| Temporal | `max_age`, `timestamp_range` (or `date_range`), `no_future_timestamps`, `date_format_valid` (or `date_format`), `business_days_only` |
| Semantic | `email_valid`, `phone_valid`, `url_valid`, `json_valid` |
| Cross-Column | `unique_combination`, `foreign_key_exists` (Python API), `sum_equals` |
| Custom | `custom` — user-defined functions via `@custom_rule` decorator |

## Links

- [Website](https://datacheck.squrtech.com)
- [GitHub](https://github.com/squrtech/datacheck)
- [Issues](https://github.com/squrtech/datacheck/issues)
- [Changelog](https://github.com/squrtech/datacheck/blob/main/CHANGELOG.md)

## License

Apache License 2.0 — Copyright 2026 Squrtech