data-sentinel 1.0.1
Python framework for building data quality tests
Requires Python: >=3.6

# data-sentinel

## Description
data-sentinel is a toolbox of flexible database connectors and test methods used to measure data integrity of datasets 
and database tables. 
* Build data quality tests which can be inserted into an existing Python script or run as a stand-alone application / 
pipeline. 
* Send outcome notifications to messaging and logging applications. 
* Halt pipelines and raise custom exceptions when needed.

## Prerequisites
Python 3.6+

sqlalchemy>=1.3.19

psycopg2

## Installing
`pip install data-sentinel`

## Build a simple data quality test script
In this example we build a script to test the integrity of three columns in a Postgres table.
 
```
import os

from datasentinel.connections import PostgresConnector
from datasentinel.log import get_file_logger
from datasentinel.tests import proportion_null_sql_test, multi_proportion_duplicate_sql_test, \
    proportion_duplicate_sql_test

def main():
    # database connection credentials
    user = os.environ["user"]
    password = os.environ["password"]
    host = os.environ["host"]
    dbname = os.environ["dbname"]
    port = os.environ["port"]

    # postgres database connector
    pg = PostgresConnector(user, password, host, dbname, port)

    # convenience function for creating python logging function with file handler
    logfile = "dq_test_example_.log"
    logger = get_file_logger(logfile).warning

    # test thresholds
    threshold0 = 0
    threshold10 = 0.1

    ##### TEST ORDERS TABLE #####
    orders_from_clause = "test_schema.Orders"  # table to be tested
    orders_where_clause = ["date(order_ts) = '2020-09-08'"]  # optional where clause

    # test for duplicates
    orders_duplicates_test_column = "id"  # column to be tested
    # execute the test for data duplication
    proportion_duplicate_sql_test(pg, orders_from_clause, threshold0, orders_duplicates_test_column,
                                  where_clause=orders_where_clause, hard_fail=False, logger=logger)

    # test multiple columns using one threshold
    orders_null_test_columns = ("product_id", "price")  # execute test on mulitple columns
    # set different test thresholds for each column
    orders_null_test_threshold = {"product_id": threshold0, "price": threshold10}
    # execute the test for null values
    proportion_null_sql_test(pg, orders_from_clause, orders_null_test_threshold, *orders_null_test_columns,
                             where_clause=orders_where_clause, logger=logger)

    # results from all tests are written to logfile


if __name__ == "__main__":
    main()
```

# Documentation
## Database Support
* Postgres

## Data Quality Tests
Data quality tests are used to measure the integrity of specified columns within a table or document. Every data 
quality test will return `'test_pass'` or `'test_fail'` depending on the given measure and threshold.

### Prebuilt Data Quality Test Input Parameters
The prebuilt data quality tests use the following input parameters:
* `db_connector` - The datbase connection object.
* `from_clause` - Path to table being tested.
* `threshold` - The threshold to use when determining if the test returns pass or fail. This parameter can be 
passed as a literal or a dictionary mapping the column name to the specified threshold literal.
* `columns` - Comma separated list of table columns to be tested.
* `where_clause` - Optional filtering logic to be applied to table contained in `from_clause`.
* `test_result_msg` - Optional message template to be returned by data quality test.
* `hard_fail` - Optional parameter. If True then data quality test will raise a `TestFailure` exception upon 
test failure.
* `logger` - Optional logging function. If no logging function is given then log messages will be swallowed.
* `kwargs` - Optional key word arguments used to format the test description.

### Prebuilt Data Quality Tests
`proportion_null_sql_test` - Performs a test for each column contained in `columns` which counts the number of 
`NULL` values found. Each test will return `'test_fail'` if the proportion of null values exceeds the specified
threshold. 

`multi_proportion_duplicate_sql_test` - Performs one test which counts the number of duplicates which occur in 
selecting all columns in `columns` simultaneously. This is equivalent to counting the number of rows returned from
a `SELECT DISTINCT` on all columns simultaneously and comparing to the total number of rows. Test will return 
`'test_fail'` if the proportion of duplicates exceedes the specified threshold. Note that for this test, the 
`threshold` parameter should be a literal and not a dictionary, since only one test is being performed.

`proportion_duplicate_sql_test` - Performs a test for each column contained in `columns` which counts the number
of duplicate values found in that column. This is equivalent to counting the number of rows returned from a 
`SELECT DISTINCT` on the target column and comparing to the total number of rows. Test will return `'test_fail'`
if the proportion of duplicates exceeds the specified threshold for that column.

## Logging
The `logger` parameter given on the prebuilt data quality test functions is expected to be a function of one string
variable. As the developer you are expected to provide the logger function for execution in any production environment.
For development purposes there are three built in logging functions which can be can be used to capture test results 
and runtime information. These logging functions provide a Python `logging` module log function while handling most of 
the typical boiler plate code.

1. `get_file_logger`

     **Input Parameters**: 
     * `file_path` - Path to file which will be used to store log messages.
     * `level` - Optional string parameter. Specifies the log level of the underlying handler. The default log level is 
     WARNING.
     * `format_` - Optional string parameter. Specifies the format to be used for logged messages. The default format is 
     `"%(asctime)s | %(name)s | %(levelname)s | %(message)s"`.
     * `logger_name` - Optional string parameter. Specifies the name of the logging function. The logger name is displayed 
     in the log message. The default `logger_name` is root.
     
     **Description**: Uses the Python `logging.FileHandler` handler to write logged messages to `file_path`.
 
 2. `get_stream_logger`
 
     **Input Parameters**:
     * `stream` - Optional parameter. Specifies the output stream which will receive logged messages. An example of an 
     output stream is the `io.StringIO` class, but any stream or file-like object which supports `write()` and `flush()` 
     methods will suffice. If None is specified the default stream is  `sys.stderr`.
      * `level` - Optional string parameter. Specifies the log level of the underlying handler. The default log level is 
     WARNING.
     * `format_` - Optional string parameter. Specifies the format to be used for logged messages. The default format is 
     `"%(asctime)s | %(name)s | %(levelname)s | %(message)s"`.
     * `logger_name` - Optional string parameter. Specifies the name of the logging function. The logger name is displayed 
     in the log message. The default `logger_name` is root.
     
     **Description**: Uses the Python `logging.StreamHandler` handler to write logged messages to `stream`.
 
 3. `get_null_logger`
 
     **Input Parameters**: None
     
     **Description**: Uses the Python `logging.NullHandler` handler. All messages logged with this logging function will be 
     swallowed.
 
 All three of the logger methods return a `logging.RootLogger` object. You must invoke one of the following methods on
 the `logging.RootLogger` object in order to write log messages.
 
 ```
logger = get_stream_logger().debug  # log messages at debug level

logger = get_stream_logger().info  # log messages at info level

logger = get_stream_logger().warning  # log messages at warning level

logger = get_stream_logger().error  # log messages at error level

logger = get_stream_logger().critical  # log messages at critical level
```
 
  **Note that if no logging method is specified then** `get_null_logger` **is used.**