semvec/pypi-packages huggingface dataset:
data-prep-lab-kfp 0.1.8.dev1
Data Preparation Laboratory Library. KFP support
License: Apache-2.0
Requires Python: >=3.10

# KFP support library

This provides support for implementing KFP pipelines automating transform's execution.
It comprises 2 main modules
* [api server client](src/kfp_support/api_server_client/README.md) 
* [workflow support](src/kfp_support/workflow_support/README.md)

## Development

### Requirements
1. python 3.10 or later
2. git command line tools
3. [pre-commit](https://pre-commit.com/)
4. twine (pip install twine)
    * but on Mac you may have to include a dir in your PATH, such as `export PATH=$PATH:/Library/Frameworks/Python.framework/Versions/3.10/bin`

### Git
Simple clone the repo and set up the pre-commit hooks.
```shell
git clone git@github.com:IBM/data-prep-lab.git
cd kfp/kfp_support_lib
pre-commit install
```
If you don't have pre-commit, you can install from [here](https://pre-commit.com/)

## Library Artifact Build and Publish

The process of creating a release for `fm_data_processing_kfp` package  involves the following steps:

cd to the package directory.

update the version in [requirements.env](../requirements.env) file.

run `make build` and `make publish`.

## Testing

To run the package tests perform the following:

To begin with, establish a Kind cluster and deploy all required components by executing the makfefile command in the main directory of this repository. As an alternative, you can manually execute the instructions provided in the [README.md](../../kind/README.md) file.

```bash
make setup
```

The next step is to deploy the `data-prep-lab-kfp` package locally within a Python virtual environment.

```bash
make  build
```

lastly, execute the tests:

```bash
make test
```

### Cleanup

It is advisable to execute the following command prior to running `make test` once more. This will ensure that any 
previous test runs resources are removed before starting new tests.

```bash
kubectl delete workflows -n kubeflow --all
```
data-prep-lab 0.1.6.dev1
Data Preparation Laboratory Library
License: Apache-2.0
Requires Python: >=3.10

# Data Processing Library
This provides a python framework for developing _transforms_
on data stored in files - currently parquet files are supported -
and running them in a [ray](https://ray.com) cluster.
Data files may be stored in the local file system or  COS/S3.
For more details see the [documentation](doc/overview.md).

### Virtual Environment
The project uses `pyproject.toml` and a Makefile for operations.
To do development you should establish the virtual environment
```shell
make venv
```
and then either activate
```shell
source venv/bin/activate
```
or set up your IDE to use the venv directory when developing in this project

## Library Artifact Build and Publish
To test, build and publish the library to artifactory
```shell
make test build publish
```
To up the version number, edit the Makefile to change VERSION and rerun
the above.  This will require committing both the `Makefile` and the
autotmatically updated `pyproject.toml` file.