np-validator 0.10.0
Awesome `np-validator` is a Python cli/package created with https://github.com/TezRomacH/python-package-template
License: MIT
Requires Python: >=3.9,<4.0

# np-validator

<div align="center">

[![Build status](https://github.com/np_validator/np-validator/workflows/build/badge.svg?branch=master&event=push)](https://github.com/np_validator/np-validator/actions?query=workflow%3Abuild)
[![Python Version](https://img.shields.io/pypi/pyversions/np-validator.svg)](https://pypi.org/project/np-validator/)
[![Dependencies Status](https://img.shields.io/badge/dependencies-up%20to%20date-brightgreen.svg)](https://github.com/np_validator/np-validator/pulls?utf8=%E2%9C%93&q=is%3Apr%20author%3Aapp%2Fdependabot)

[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![Security: bandit](https://img.shields.io/badge/security-bandit-green.svg)](https://github.com/PyCQA/bandit)
[![Pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/np_validator/np-validator/blob/master/.pre-commit-config.yaml)
[![Semantic Versions](https://img.shields.io/badge/%20%20%F0%9F%93%A6%F0%9F%9A%80-semantic--versions-e10079.svg)](https://github.com/np_validator/np-validator/releases)
[![License](https://img.shields.io/github/license/np_validator/np-validator)](https://github.com/np_validator/np-validator/blob/master/LICENSE)
![Coverage Report](assets/images/coverage.svg)

`np-validator` is just a simple Python cli/package that validates data sources for the neuropixel pipline using templated workflows.

</div>

## Quick overview

### API

#### Auto-generated steps

By default `run_validation` can be run on a list of
filepaths. A list of `ValidationStep` will be auto-generated if a valid project name is supplied.
_If no project name is supplied project name will be "default"._

```python
from np_validator import run_validation

filepaths = [
  "some/sort/of/path/prefix/uuid-maybe.mapping.pkl",
  "some/sort/of/path/prefix/uuid-maybe.behavior.pkl",
  "this/will/be/ignored/uuid-maybe.replay.pkl",
]

results = run_validation(filepaths, project="default")
```

Manually generating validation steps

```python
from np_validator import Processor, Validator, ValidationStep, run_validation

# make a basic filesize validator
fs_validator = Validator(
  name="meets_filesize_threshold",
  args={
    "threshold": 10,
  },
)

# add validator to a validation step
validation_step_0 = ValidationStep(
  path_suffix=".mapping.pkl",
  validators=[fs_validator, ],
)

# make a validation step with a processor
# processors convert the data source from it's basic state, a filepath, to an easier to use object

# unpickle, unpickles an arbitrary filepath into a python object
unpickler = Processor(
  name="unpickle",
)

# has_dict_key, checks if a dict-like interface has a key at path
session_uuid_validator = Validator(
  name="has_dict_key",
  args={
    "path": ["session_uuid", ],
  }
)

# assembling it all together as a validation step

validation_step_1 = ValidationStep(
  path_suffix=".behavior.pkl",
  processor=unpickler,
  validators=[session_uuid_validator, ],
)

# running a validation
filepaths = [
  "some/sort/of/path/prefix/uuid-maybe.mapping.pkl",
  "some/sort/of/path/prefix/uuid-maybe.behavior.pkl",
  "this/will/be/ignored/uuid-maybe.replay.pkl",
]
results = run_validation(
  filepaths,
  [validation_step_0, validation_step_1, ],
)
```

Updating autogenerated steps

```python
from np_validator import Processor, Validator, ValidationStep, autogenerate_validation_steps, update_project_validation_steps

# autogenerated steps are organized by projects
validation_step_0 = ValidationStep(
  path_suffix=".mapping.pkl",
  validators=[
    Validator(
      name="meets_filesize_threshold",
      args={
        "threshold": 10,
      },
    ),
  ],
)

validation_step_1 = ValidationStep(
  path_suffix=".behavior.pkl",
  processor=Processor(
    name="unpickle",
  ),
  validators=[
    Validator(
      name="has_dict_key",
      args={
        "path": ["session_uuid", ],
      }
    ),
  ],
)

# updating project validation steps overwrites the previous steps

update_project_validation_steps(
  "pretest",  # project name
  [validation_step_0, validation_step_1],
)

# to append, get the current steps with autogenerate_validation_steps and append to it
current = autogenerate_validation_steps("default")
updated = current + [validation_step_0, validation_step_1]
update_project_validation_steps(
  "default",  # project name
  updated, # append new steps
)
```

## Documentation

For more detailed documentation on using this package please refer to the [docs](https://alleninstitute.github.io/np_validator/).

## Contributing

### Processor

To add a new processor, add the function to `np_validator/processors.py`. Ideally create a test for it in `tests`. Run the _tests_\* to help ensure that no regressions have been introduced. Each processor is expected to have one required argument which is expected to be a string filepath.

### Validator

To add a new validator, add the function to `np_validator/validators.py`. Ideally create a test for it in `tests`. Run the _tests_\* to help ensure that no regressions have been introduced.

\*_Tests automatically run on pull_request\._

## Installation

```bash
pip install -U np-validator
```

or install with `Poetry`

```bash
poetry add np-validator
```

### Makefile usage

[`Makefile`](https://github.com/np_validator/np-validator/blob/master/Makefile) contains a lot of functions for faster development.

<details>
<summary>1. Download and remove Poetry</summary>
<p>

To download and install Poetry run:

```bash
make poetry-download
```

To uninstall

```bash
make poetry-remove
```

</p>
</details>

<details>
<summary>2. Install all dependencies and pre-commit hooks</summary>
<p>

Install requirements:

```bash
make install
```

Pre-commit hooks coulb be installed after `git init` via

```bash
make pre-commit-install
```

</p>
</details>

<details>
<summary>3. Codestyle</summary>
<p>

Automatic formatting uses `pyupgrade`, `isort` and `black`.

```bash
make codestyle

# or use synonym
make formatting
```

Codestyle checks only, without rewriting files:

```bash
make check-codestyle
```

> Note: `check-codestyle` uses `isort`, `black` and `darglint` library

Update all dev libraries to the latest version using one comand

```bash
make update-dev-deps
```

</p>
</details>

<details>
<summary>4. Code security</summary>
<p>

```bash
make check-safety
```

This command launches `Poetry` integrity checks as well as identifies security issues with `Safety` and `Bandit`.

```bash
make check-safety
```

</p>
</details>

<details>
<summary>5. Type checks</summary>
<p>

Run `mypy` static type checker

```bash
make mypy
```

</p>
</details>

<details>
<summary>6. Tests with coverage badges</summary>
<p>

Run `pytest`

```bash
make test
```

</p>
</details>

<details>
<summary>7. All linters</summary>
<p>

Of course there is a command to ~~rule~~ run all linters in one:

```bash
make lint
```

the same as:

```bash
make test && make check-codestyle && make mypy && make check-safety
```

</p>
</details>

<details>
<summary>8. Docker</summary>
<p>

```bash
make docker-build
```

which is equivalent to:

```bash
make docker-build VERSION=latest
```

Remove docker image with

```bash
make docker-remove
```

More information [about docker](https://github.com/np_validator/np-validator/tree/master/docker).

</p>
</details>

<details>
<summary>9. Cleanup</summary>
<p>
Delete pycache files

```bash
make pycache-remove
```

Remove package build

```bash
make build-remove
```

Delete .DS_STORE files

```bash
make dsstore-remove
```

Remove .mypycache

```bash
make mypycache-remove
```

Or to remove all above run:

```bash
make cleanup
```

</p>
</details>

## 📈 Releases

You can see the list of available releases on the [GitHub Releases](https://github.com/np_validator/np-validator/releases) page.

We follow [Semantic Versions](https://semver.org/) specification.

We use [`Release Drafter`](https://github.com/marketplace/actions/release-drafter). As pull requests are merged, a draft release is kept up-to-date listing the changes, ready to publish when you’re ready. With the categories option, you can categorize pull requests in release notes using labels.

### List of labels and corresponding titles

|               **Label**               |  **Title in Releases**  |
| :-----------------------------------: | :---------------------: |
|       `enhancement`, `feature`        |       🚀 Features       |
| `bug`, `refactoring`, `bugfix`, `fix` | 🔧 Fixes & Refactoring  |
|       `build`, `ci`, `testing`        | 📦 Build System & CI/CD |
|              `breaking`               |   💥 Breaking Changes   |
|            `documentation`            |    📝 Documentation     |
|            `dependencies`             | ⬆️ Dependencies updates |

You can update it in [`release-drafter.yml`](https://github.com/np_validator/np-validator/blob/master/.github/release-drafter.yml).

GitHub creates the `bug`, `enhancement`, and `documentation` labels for you. Dependabot creates the `dependencies` label. Create the remaining labels on the Issues tab of your GitHub repository, when you need them.

## 🛡 License

[![License](https://img.shields.io/github/license/np_validator/np-validator)](https://github.com/np_validator/np-validator/blob/master/LICENSE)

This project is licensed under the terms of the `MIT` license. See [LICENSE](https://github.com/np_validator/np-validator/blob/master/LICENSE) for more details.

## Additonal information

<details>
<summary>Initializing a new repo</summary>

### Initialize your code

1. Initialize `git` inside your repo:

```bash
cd np-validator && git init
```

2. If you don't have `Poetry` installed run:

```bash
make poetry-download
```

3. Initialize poetry and install `pre-commit` hooks:

```bash
make install
make pre-commit-install
```

4. Run the codestyle:

```bash
make codestyle
```

5. Upload initial code to GitHub:

```bash
git add .
git commit -m ":tada: Initial commit"
git branch -M main
git remote add origin https://github.com/np_validator/np-validator.git
git push -u origin main
```

### Set up bots

- Set up [Dependabot](https://docs.github.com/en/github/administering-a-repository/enabling-and-disabling-version-updates#enabling-github-dependabot-version-updates) to ensure you have the latest dependencies.
- Set up [Stale bot](https://github.com/apps/stale) for automatic issue closing.

### Poetry

Want to know more about Poetry? Check [its documentation](https://python-poetry.org/docs/).

<details>
<summary>Details about Poetry</summary>
<p>

Poetry's [commands](https://python-poetry.org/docs/cli/#commands) are very intuitive and easy to learn, like:

- `poetry add numpy@latest`
- `poetry run pytest`
- `poetry publish --build`

etc

</p>
</details>

### Building and releasing your package

Building a new version of the application contains steps:

- Bump the version of your package `poetry version <version>`. You can pass the new version explicitly, or a rule such as `major`, `minor`, or `patch`. For more details, refer to the [Semantic Versions](https://semver.org/) standard.
- Make a commit to `GitHub`.
- Create a `GitHub release`.
- And... publish 🙂 `poetry publish --build`

## 🎯 What's next

Well, that's up to you 💪🏻. I can only recommend the packages and articles that helped me.

- [`Typer`](https://github.com/tiangolo/typer) is great for creating CLI applications.
- [`Rich`](https://github.com/willmcgugan/rich) makes it easy to add beautiful formatting in the terminal.
- [`Pydantic`](https://github.com/samuelcolvin/pydantic/) – data validation and settings management using Python type hinting.
- [`Loguru`](https://github.com/Delgan/loguru) makes logging (stupidly) simple.
- [`tqdm`](https://github.com/tqdm/tqdm) – fast, extensible progress bar for Python and CLI.
- [`IceCream`](https://github.com/gruns/icecream) is a little library for sweet and creamy debugging.
- [`orjson`](https://github.com/ijl/orjson) – ultra fast JSON parsing library.
- [`Returns`](https://github.com/dry-python/returns) makes you function's output meaningful, typed, and safe!
- [`Hydra`](https://github.com/facebookresearch/hydra) is a framework for elegantly configuring complex applications.
- [`FastAPI`](https://github.com/tiangolo/fastapi) is a type-driven asynchronous web framework.

Articles:

- [Open Source Guides](https://opensource.guide/).
- [A handy guide to financial support for open source](https://github.com/nayafia/lemonade-stand)
- [GitHub Actions Documentation](https://help.github.com/en/actions).
- Maybe you would like to add [gitmoji](https://gitmoji.carloscuesta.me/) to commit names. This is really funny. 😄

## 🚀 Features

### Development features

- Supports for `Python 3.9` and higher.
- [`Poetry`](https://python-poetry.org/) as the dependencies manager. See configuration in [`pyproject.toml`](https://github.com/np_validator/np-validator/blob/master/pyproject.toml) and [`setup.cfg`](https://github.com/np_validator/np-validator/blob/master/setup.cfg).
- Automatic codestyle with [`black`](https://github.com/psf/black), [`isort`](https://github.com/timothycrosley/isort) and [`pyupgrade`](https://github.com/asottile/pyupgrade).
- Ready-to-use [`pre-commit`](https://pre-commit.com/) hooks with code-formatting.
- Type checks with [`mypy`](https://mypy.readthedocs.io); docstring checks with [`darglint`](https://github.com/terrencepreilly/darglint); security checks with [`safety`](https://github.com/pyupio/safety) and [`bandit`](https://github.com/PyCQA/bandit)
- Testing with [`pytest`](https://docs.pytest.org/en/latest/).
- Ready-to-use [`.editorconfig`](https://github.com/np_validator/np-validator/blob/master/.editorconfig), [`.dockerignore`](https://github.com/np_validator/np-validator/blob/master/.dockerignore), and [`.gitignore`](https://github.com/np_validator/np-validator/blob/master/.gitignore). You don't have to worry about those things.

### Deployment features

- `GitHub` integration: issue and pr templates.
- `Github Actions` with predefined [build workflow](https://github.com/np_validator/np-validator/blob/master/.github/workflows/build.yml) as the default CI/CD.
- Everything is already set up for security checks, codestyle checks, code formatting, testing, linting, docker builds, etc with [`Makefile`](https://github.com/np_validator/np-validator/blob/master/Makefile#L89). More details in [makefile-usage](#makefile-usage).
- [Dockerfile](https://github.com/np_validator/np-validator/blob/master/docker/Dockerfile) for your package.
- Always up-to-date dependencies with [`@dependabot`](https://dependabot.com/). You will only [enable it](https://docs.github.com/en/github/administering-a-repository/enabling-and-disabling-version-updates#enabling-github-dependabot-version-updates).
- Automatic drafts of new releases with [`Release Drafter`](https://github.com/marketplace/actions/release-drafter). You may see the list of labels in [`release-drafter.yml`](https://github.com/np_validator/np-validator/blob/master/.github/release-drafter.yml). Works perfectly with [Semantic Versions](https://semver.org/) specification.

### Open source community features

- Ready-to-use [Pull Requests templates](https://github.com/np_validator/np-validator/blob/master/.github/PULL_REQUEST_TEMPLATE.md) and several [Issue templates](https://github.com/np_validator/np-validator/tree/master/.github/ISSUE_TEMPLATE).
- Files such as: `LICENSE`, `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`, and `SECURITY.md` are generated automatically.
- [`Stale bot`](https://github.com/apps/stale) that closes abandoned issues after a period of inactivity. (You will only [need to setup free plan](https://github.com/marketplace/stale)). Configuration is [here](https://github.com/np_validator/np-validator/blob/master/.github/.stale.yml).
- [Semantic Versions](https://semver.org/) specification with [`Release Drafter`](https://github.com/marketplace/actions/release-drafter).
</details>